www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Unquoted regular expressions

reply Georg Wrede <georg.wrede nospam.org> writes:
IIRC, we left this issue at "they'd be followed by a left parenthesis."

I think that the lexer should try everyting else, and if nothing else is 
ok, then try if it is an unquoted regular expression. Without regard to 
parenthesis. Then we could write:

foo = /laksjdf/;
bar = /lskdf/sdf/;

Foo and bar would be functions taking a string, an int, and outputting 
an array, and returning bool.

This may not have to be slow. After all, we do already have the regex 
library, and the lexer could use only the "recognizing" part of it, i.e. 
do the parsing without generating any binary code.

If it does not parse as a regular expression, then flag error.

This would make parser writing much cleaner.

intR       = /lksjdf/;
floatR     = /fjfjf/;
stringR    = /yee\"lksjdlfks\"haw/;
hairyClass = /lksjdlfkjsdlfkjsldfklskdfjlskdjlfk/;

and also,

ok = /lksjdflks/ (string, pos, resultArray);

or

if (/lksjd/ (theArray, curPos, res))
{
     // do the doobydoo;
}

Since regular expressions are a natural part of parsing, that area 
should be the main target.

...
     if      (intR      (input, where, res) wehaveInt();
     else if (floatR    (input, where, res) wehaveFloat();
     else if (stringR   (input, where, res) wehaveString();
     else if (hairyClass(input, where, res) shootTheSheriff();
...
     else giveUp();

FTR, the whole idea here being that these regular expressions are 
compiled to executable at compile time.

And, that regexes encountered at runtime would be compiled at encounter, 
as in:

char[] getInput = askRegexDialog();  // string assignment
regexTypeFunction gi = getInput;     // compile regex

if (gi(ins, p, oa))
{
     // we found user's wishes in the input stream
}

------------

This is outline, so details are not accurate. (For example, how to 
mediate the end position of regex search, was not considered here.)
Mar 28 2005
parent reply "Walter" <newshound digitalmars.com> writes:
"Georg Wrede" <georg.wrede nospam.org> wrote in message
news:4248919A.1010208 nospam.org...

 IIRC, we left this issue at "they'd be followed by a left parenthesis."

 I think that the lexer should try everyting else, and if nothing else is
 ok, then try if it is an unquoted regular expression. Without regard to
 parenthesis. Then we could write:


This is done in the DMDScript lexer. However, it's an ugly hack and requires
the lexer and parser to cooperate with each other. One of the design goals
of D is to have a lexer that is independent of parsing. This pretty much
means that regular expression literals using / is out. But check out some of
the new functions I added to www.digitalmars.com/d/std_regexp.html, I think
the syntax is reasonably clean and useful.
Mar 29 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
Walter wrote:

 "Georg Wrede" <georg.wrede nospam.org> wrote in message
 news:4248919A.1010208 nospam.org...
 

IIRC, we left this issue at "they'd be followed by a left parenthesis."

I think that the lexer should try everyting else, and if nothing else is
ok, then try if it is an unquoted regular expression. Without regard to
parenthesis. Then we could write:


 
 This is done in the DMDScript lexer. However, it's an ugly hack and requires
 the lexer and parser to cooperate with each other. One of the design goals
 of D is to have a lexer that is independent of parsing. This pretty much
 means that regular expression literals using / is out. But check out some of
 the new functions I added to www.digitalmars.com/d/std_regexp.html, I think
 the syntax is reasonably clean and useful.


It is! And I'm not forcing this issue here, it's just that "it would be 
nice", and "not difficult". (IMHO)

So, with the unquoted regexes + separate lexing, we have the following 
problem:

height = volume /width/ length;
halfHeight = volume /width/length/ 2;

(I've emphasized the problem above with stupid spacing.)

/width/length/ can either be a regex that substitutes every width with 
length, or it can be part of a math expression.

 From the context it is painfully obvious which, but the lexer (who has 
aphasia because it is context independent), can't know if we here have 
unquoted regular expressions, or not.

Case closed. For good.

What else can we do? Oh, and why would we?

The existing regex library is nice! But as we see from other languages, 
having the regexes "in the code" and "obviously compiled at compile 
time, rather than runtime", does seem to have its merits.

"Obviously"? Well, reading source code of a compiled language, if a 
literal regex appears as such in the source code, then it is taken for 
granted that it is compiled at compile time. It's just something you do. 
OTOH, having regexes in quoted strings and used by library functions 
"feels" like they are runtime compiled.

Another thing, this discussion of Unquoted regexes has been a kind of 
misnomer (and I take most of the blame). A regex that looks like

/blkajsld/

is actually quoted already. (Look at substitute in ed, vi, vim, and the 
like, and you see that other characters can be used for quoting. That is 
usually convenient when the regex itself contains forward slashes.)

So, how would we have the lexer recognize a regex type? And the 
programmer immediately spot it too?

"/lkajsdl/" would seem natural. So "/ and /" would be two new tokens.

Very nice. Except that this precludes having /asdfadsf/ as the contents 
of a normal string. This is getting depressing.

Would reversing the tokens do it? Then we'd have /"lksjdlks"/ and 
/"lkasjdf/lkasjdf"/ for searching and substituting, respectively.

The lexer would not even have to know of the middle slash, since the 
function signatures for both are the same (at least as per the earlier 
thread).

Requiring backslash quoting of double quotes would take care of the 
situation where the regex contains a double quote immediately followed 
by a forward slash, as in /"lksjdlfkasd\"/lkasjdf"/.
Mar 30 2005
parent reply xs0 <xs0 xs0.com> writes:
/" isn't that good, it prevents you from having a normal opDiv(char[])...

how about RE"..." and RE`...`? we already have wysiwyg strings, so this 
wouldn't be something completely new (at least syntax-wise).. It doesn't 
use a slash, but who cares, I think it's still obvious that it's 
compile-time?


xs0


Georg Wrede wrote:

 Walter wrote:
 

 "Georg Wrede" <georg.wrede nospam.org> wrote in message
 news:4248919A.1010208 nospam.org...


 IIRC, we left this issue at "they'd be followed by a left parenthesis."

 I think that the lexer should try everyting else, and if nothing else is
 ok, then try if it is an unquoted regular expression. Without regard to
 parenthesis. Then we could write:



 This is done in the DMDScript lexer. However, it's an ugly hack and 
 requires
 the lexer and parser to cooperate with each other. One of the design 
 goals
 of D is to have a lexer that is independent of parsing. This pretty much
 means that regular expression literals using / is out. But check out 
 some of
 the new functions I added to www.digitalmars.com/d/std_regexp.html, I 
 think
 the syntax is reasonably clean and useful.


 
 
 It is! And I'm not forcing this issue here, it's just that "it would be 
 nice", and "not difficult". (IMHO)
 
 So, with the unquoted regexes + separate lexing, we have the following 
 problem:
 
 height = volume /width/ length;
 halfHeight = volume /width/length/ 2;
 
 (I've emphasized the problem above with stupid spacing.)
 
 /width/length/ can either be a regex that substitutes every width with 
 length, or it can be part of a math expression.
 
  From the context it is painfully obvious which, but the lexer (who has 
 aphasia because it is context independent), can't know if we here have 
 unquoted regular expressions, or not.
 
 Case closed. For good.
 
 What else can we do? Oh, and why would we?
 
 The existing regex library is nice! But as we see from other languages, 
 having the regexes "in the code" and "obviously compiled at compile 
 time, rather than runtime", does seem to have its merits.
 
 "Obviously"? Well, reading source code of a compiled language, if a 
 literal regex appears as such in the source code, then it is taken for 
 granted that it is compiled at compile time. It's just something you do. 
 OTOH, having regexes in quoted strings and used by library functions 
 "feels" like they are runtime compiled.
 
 Another thing, this discussion of Unquoted regexes has been a kind of 
 misnomer (and I take most of the blame). A regex that looks like
 
 /blkajsld/
 
 is actually quoted already. (Look at substitute in ed, vi, vim, and the 
 like, and you see that other characters can be used for quoting. That is 
 usually convenient when the regex itself contains forward slashes.)
 
 So, how would we have the lexer recognize a regex type? And the 
 programmer immediately spot it too?
 
 "/lkajsdl/" would seem natural. So "/ and /" would be two new tokens.
 
 Very nice. Except that this precludes having /asdfadsf/ as the contents 
 of a normal string. This is getting depressing.
 
 Would reversing the tokens do it? Then we'd have /"lksjdlks"/ and 
 /"lkasjdf/lkasjdf"/ for searching and substituting, respectively.
 
 The lexer would not even have to know of the middle slash, since the 
 function signatures for both are the same (at least as per the earlier 
 thread).
 
 Requiring backslash quoting of double quotes would take care of the 
 situation where the regex contains a double quote immediately followed 
 by a forward slash, as in /"lksjdlfkasd\"/lkasjdf"/.
Mar 31 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
xs0 wrote:

 
 /" isn't that good, it prevents you from having a normal opDiv(char[])...


To what is a "normal opDiv(char[])" used?


 how about RE"..." and RE`...`? we already have wysiwyg strings, so this 
 wouldn't be something completely new (at least syntax-wise).. It doesn't 
 use a slash, but who cares, I think it's still obvious that it's 
 compile-time?


I could live with that.
Mar 31 2005
parent reply xs0 <xs0 xs0.com> writes:
Georg Wrede wrote:

 xs0 wrote:
 

 /" isn't that good, it prevents you from having a normal opDiv(char[])...


 
 
 To what is a "normal opDiv(char[])" used?


I don't know (to split some string buffer with the parameter, perhaps?), 
  but the point was that /"RE"/ would clash with division by char[], 
which is not good..


xs0
Apr 01 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
xs0 wrote:

 Georg Wrede wrote:

 xs0 wrote:

 /" isn't that good, it prevents you from having a normal 
 opDiv(char[])...


 To what is a "normal opDiv(char[])" used?


 
 I don't know (to split some string buffer with the parameter, perhaps?), 
  but the point was that /"RE"/ would clash with division by char[], 
 which is not good..


Ehh, which might be more important? ;-)
Apr 01 2005
parent reply xs0 <xs0 xs0.com> writes:
Georg Wrede wrote:

 xs0 wrote:
 

 Georg Wrede wrote:


 xs0 wrote:


 /" isn't that good, it prevents you from having a normal 
 opDiv(char[])...


 To what is a "normal opDiv(char[])" used?



 I don't know (to split some string buffer with the parameter, 
 perhaps?),  but the point was that /"RE"/ would clash with division by 
 char[], which is not good..


 
 
 Ehh, which might be more important? ;-)


Well, one can argue both ways, but I think that the language should be 
without random exceptions like this.. It would be about the same s**t as 
requiring spaces between >> in template parameters in C++ ..


xs0
Apr 01 2005
parent Georg Wrede <georg.wrede nospam.org> writes:
xs0 wrote:

 Georg Wrede wrote:
 

 xs0 wrote:


 Georg Wrede wrote:


 xs0 wrote:


 /" isn't that good, it prevents you from having a normal 
 opDiv(char[])...



 To what is a "normal opDiv(char[])" used?




 I don't know (to split some string buffer with the parameter, 
 perhaps?),  but the point was that /"RE"/ would clash with division 
 by char[], which is not good..




 Ehh, which might be more important? ;-)


 
 
 Well, one can argue both ways, but I think that the language should be 
 without random exceptions like this.. It would be about the same s**t as 
 requiring spaces between >> in template parameters in C++ ..


Good point. And I've personally argued for the same logic on several 
occasions here.

On this particular occasion, however, we'd either have to come up with 
something else as the token, or weigh principle against practice.

Suggestions, ideas?
Apr 01 2005