www.digitalmars.com         C & C++   DMDScript  

D - Preprocessor.

reply Ilya Minkov <midiclub 8ung.at> writes:
Threre exist kinds of preprocessors, which are not a bad idea at all.
For example, OCampl-p4

http://caml.inria.fr/camlp4/tutorial/index.html

Please read through this. It doesn't make text-level preprocessing like 
C, but gives an alternative, script-steered parsing possibility. It is 
used to implement syntactic extensions, which can be 100% safe to use.

Examples of useful uses for such a preprocesor in D:
  - implement some syntactic features which don't make it into the 
language, either because Walter doesn'tlike them, or because he lacks of 
  time. Maybe they just shouldn't pollute the language itself. Like: 
patternmatching, base#value digital constants and so on...
  - implement other languages. Like to compile Delphi or Ada programs :)

BTW, to the idea of the patternmatcher. It could have switch sytax:

bit valid = match (pet; food) {
	case (CAT; FISH):
		true;
	case (CAT; RICE):
		false;
	default:
		false;
}

OK, silly example. The idea behind it is to pre-evaluate expressions 
which show up in "case()"-s and save them in the bit-mask, and then 
select the case after bit-masks. Thus it is converted into a switch 
statement, which evaluates some expression in the beginning and 
"switch"es after some constants. After each case a "break" is included, 
with some exceptions.

Such extensions to the language could be done in a pre-processor of this 
kind.

-i.
Jan 07 2003
next sibling parent reply Evan McClanahan <evan dontSPAMaltarinteractive.com> writes:
Ilya Minkov wrote:

 Threre exist kinds of preprocessors, which are not a bad idea at all.
 For example, OCampl-p4
 
 http://caml.inria.fr/camlp4/tutorial/index.html
 
 Please read through this. It doesn't make text-level preprocessing like 
 C, but gives an alternative, script-steered parsing possibility. It is 
 used to implement syntactic extensions, which can be 100% safe to use.


I can't think of any place where adding a separate step to compilaion 
makes sense anymore.  I would support something like lisp macros (which 
aren't preprocessed) for sytax expansion, if it could be safely worked 
into D, but I, for one, think that a preprocessor is a needless 
complication.  If you could come up with a way to do this type of stuff 
in the course of standard compilation, I think that it would be better.

Evan
Jan 08 2003
next sibling parent reply Evan McClanahan <evan dontSPAMaltarinteractive.com> writes:
Evan McClanahan wrote:

 Ilya Minkov wrote:
 

 Threre exist kinds of preprocessors, which are not a bad idea at all.
 For example, OCampl-p4

 http://caml.inria.fr/camlp4/tutorial/index.html

 Please read through this. It doesn't make text-level preprocessing 
 like C, but gives an alternative, script-steered parsing possibility. 
 It is used to implement syntactic extensions, which can be 100% safe 
 to use.


 
 
 I can't think of any place where adding a separate step to compilaion 
 makes sense anymore.  I would support something like lisp macros (which 
 aren't preprocessed) for sytax expansion, if it could be safely worked 
 into D, but I, for one, think that a preprocessor is a needless 
 complication.  If you could come up with a way to do this type of stuff 
 in the course of standard compilation, I think that it would be better.


Come to think of it, this brings up an even deeper issue than just 
preprocessors.  It seems to me that the preprocessor in C/C++, at least 
part of the time is merely a way around its terribly rigid syntax. 
There will always be times when people are going to need a more 
expressive way of doing what they're doing than the syntax will allow. 
However, this brings up the issue of readability.  When you have a 
language like lisp, where there's a relatively small core with huge 
libraries and additional libraries of standard macros, it seems to me 
that, while it might allow additional epxressiveness, creates a C++-like 
problem where no two people are programming in the same language.  While 
it would be nice for the language to be as small as possible, with all 
of its shortcomings taken care of my a nice, clean expansion mechanism, 
I feel like it would be a loss for a lot of beginner programmers, 
because it just isn't all that common a way of doing things, and could 
encourage the problem mentioned above.  It would be interesting, though 
to see an imperative language try the above path, though I don't really 
think that it's the right thing for D to do.

Anyone have any links to a language like that?  I'm not even sure what 
it would be called, in terms of type.

Evan
Jan 08 2003
parent reply Ilya Minkov <midiclub 8ung.at> writes:
Evan McClanahan wrote:

 Evan McClanahan wrote:
 

 Ilya Minkov wrote:
 

 Threre exist kinds of preprocessors, which are not a bad idea at
 all. For example, OCampl-p4
 
 http://caml.inria.fr/camlp4/tutorial/index.html
 
 Please read through this. It doesn't make text-level
 preprocessing like C, but gives an alternative, script-steered
 parsing possibility. It is used to implement syntactic
 extensions, which can be 100% safe to use.


 
 I can't think of any place where adding a separate step to
 compilaion makes sense anymore.  I would support something like
 lisp macros (which aren't preprocessed) for sytax expansion, if it
 could be safely worked into D, but I, for one, think that a
 preprocessor is a needless complication.  If you could come up with
 a way to do this type of stuff in the course of standard
 compilation, I think that it would be better.


 
 
 Come to think of it, this brings up an even deeper issue than just 
 preprocessors.  It seems to me that the preprocessor in C/C++, at
 least part of the time is merely a way around its terribly rigid
 syntax. There will always be times when people are going to need a
 more expressive way of doing what they're doing than the syntax will
 allow. However, this brings up the issue of readability.  When you
 have a language like lisp, where there's a relatively small core with
 huge libraries and additional libraries of standard macros, it seems
 to me that, while it might allow additional epxressiveness, creates a
 C++-like problem where no two people are programming in the same
 language.  While it would be nice for the language to be as small as
 possible, with all of its shortcomings taken care of my a nice, clean
 expansion mechanism, I feel like it would be a loss for a lot of
 beginner programmers, because it just isn't all that common a way of
 doing things, and could encourage the problem mentioned above.  It
 would be interesting, though to see an imperative language try the
 above path, though I don't really think that it's the right thing for
 D to do.


As far as I see the situation, the use of OCamlP4 is not very
widespread. But the extensions are very powerful. I guess there exists
even a kind if YACC-like extention written in it.

The programmers don't really seem to use this system at any ocassion. It
is just not too easy to write these extentions. In C and C++, most
coders simply drop them in, writing them at about the same rate as
normal code. They have become a part of "normal code", though less
predictable. I guess they appeared as a replacement to "inline
functions" which a compiler lacked, but because of the
oversimplification they don't cope with this particularly well, but have
also been used for many different things. They also go around the lack
of constants in C of that time, and lack of separate compilation.

I've taken a look at LISP macros, and i can see little improvement over
C macros. It is still the same macro expansion with a couple of
"features" dropped in. OCaml P4 system is completely different - you
write a plugin, which processes a complete pre-parsed stream of tokens
in a free manner. You can implement a LISP-like macro expander as such a
plug-in. You can implement a completely different syntax over the same
set of keywords. You can issue warnings or syntax errors. And i guess it
is not only a pre-processor - it can send modified parsed streams
directly to the compiler. Or even generated syntax-trees, depending on
the type and amount of changes made. Which effectively avoid adding
another parsing step. Note: I'm not sure the current implementation does
that, but i guess it is the idea behind the system. In OCaml it is
facilitated by the fact, that compiled ocaml executables can call
byte-interpreted ocaml code. It can even carry a bytecode-compiler with
it. BTW, That's how ocaml interpreter actually works - it is a compiled
ocaml programme calling ocaml bytecode calling compiled-in ocaml
libraries. :>

The way CamlP4 works is highly complicated, but it allowes to change
parse trees and thus even allows to write extensions which write with
original *and* modified basic syntaxes in a homogeniuos manner. It
allows to load BNF grammars as addition or replacement. I doubt there
would be every day more and more syntax extentions written on any
occasion and for every project, like macros are in C and C++. They are
hard to write, but still easier than fiddling around in compiler core.

D doesn't have such a feature, as it wasn't initially developed as
interpreter, unlike caml. The plugins would have to be compiled into
dynamic-load libraries. DM also offers its implementation of JavaScript
for scripting, but i somehow think that D is more appropriate because of
its power.

Think again: a fairly complicated extention method would discourage
people from writing new extentions where not appropriate. However, it
can be used to make D syntax closer to C for known working legacy code,
or towards even safer system than curent D, without defining new syntax,
warning about many constructs typical in C and allowed in D but not
requiered for quality programmimg. Like octal constants, assignments in
"if", and so on.

I shall try to write such a preprocessor when i have time, and i'll try
to make sure it can be integrated into D compiler later, if desired.
I'll try to base it on D frontend code.

-i.
Jan 08 2003
parent Ilya Minkov <midiclub 8ung.at> writes:
I've got a bug in my post :)

This paragraph:

 D doesn't have such a feature ...


should come one earlier.

i.
Jan 08 2003
prev sibling parent Ilya Minkov <midiclub 8ung.at> writes:
Evan McClanahan wrote:

 I would support something like lisp macros (which aren't
 preprocessed) for sytax expansion, if it could be safely worked into
 D, but I, for one, think that a preprocessor is a needless 
 complication.  If you could come up with a way to do this type of
 stuff in the course of standard compilation, I think that it would be
 better.


I finally came to *actually* read about the way lisp works and lisp 
macros and such. It is 100% genious, but i don't think it's possible to 
design it into the language at such a late point... I'll think and read 
more about it, it's an interesting concept...
Jan 29 2003
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
One major design goal of D was to eliminate the text preprocessor. This
involved figuring out what people were using the preprocessor for, and then
finding a way to do that symbolically. I believe that D covers this well,
having symbolic capability for generics, asserts, imports, constants, debug
compiles, conditional compilation, etc.

What D doesn't support are things like:

    #define BEGIN {
    #define END }

which are used to make C look like another language.

Using D doesn't mean you can't preprocess the source text - any general
purpose macro text processor can be used to preprocess text before feeding
it to the D compiler. The C preprocessor is so wretchedly primitive anyway,
the only reason it is in wide use is because people are so familiar with it.


"Ilya Minkov" <midiclub 8ung.at> wrote in message
news:aveuj4$1uqt$1 digitaldaemon.com...

 Threre exist kinds of preprocessors, which are not a bad idea at all.
 For example, OCampl-p4

 http://caml.inria.fr/camlp4/tutorial/index.html

 Please read through this. It doesn't make text-level preprocessing like
 C, but gives an alternative, script-steered parsing possibility. It is
 used to implement syntactic extensions, which can be 100% safe to use.

 Examples of useful uses for such a preprocesor in D:
   - implement some syntactic features which don't make it into the
 language, either because Walter doesn'tlike them, or because he lacks of
   time. Maybe they just shouldn't pollute the language itself. Like:
 patternmatching, base#value digital constants and so on...
   - implement other languages. Like to compile Delphi or Ada programs :)

 BTW, to the idea of the patternmatcher. It could have switch sytax:

 bit valid = match (pet; food) {
 case (CAT; FISH):
 true;
 case (CAT; RICE):
 false;
 default:
 false;
 }

 OK, silly example. The idea behind it is to pre-evaluate expressions
 which show up in "case()"-s and save them in the bit-mask, and then
 select the case after bit-masks. Thus it is converted into a switch
 statement, which evaluates some expression in the beginning and
 "switch"es after some constants. After each case a "break" is included,
 with some exceptions.

 Such extensions to the language could be done in a pre-processor of this
 kind.

 -i.
Jan 11 2003
next sibling parent reply "Sean L. Palmer" <seanpalmer directvinternet.com> writes:
All good points.

So have you decided how to handle C's __LINE__, __FILE__, __DATE__, and
__TIME__ builtin macros?  Those are useful for source control systems,
automated builds, external preprocessors, and various other stuff.  None of
these have D equivalents to my knowledge.  Unfortunately it seems they'd
need some keywords.  Maybe there could be one keyword sort of like
__declspec that controls all this kind of stuff?

Sean

"Walter" <walter digitalmars.com> wrote in message
news:avqrpo$2lct$1 digitaldaemon.com...

 One major design goal of D was to eliminate the text preprocessor. This
 involved figuring out what people were using the preprocessor for, and


then

 finding a way to do that symbolically. I believe that D covers this well,
 having symbolic capability for generics, asserts, imports, constants,


debug

 compiles, conditional compilation, etc.

 What D doesn't support are things like:

     #define BEGIN {
     #define END }

 which are used to make C look like another language.

 Using D doesn't mean you can't preprocess the source text - any general
 purpose macro text processor can be used to preprocess text before feeding
 it to the D compiler. The C preprocessor is so wretchedly primitive


anyway,

 the only reason it is in wide use is because people are so familiar with


it.
Jan 12 2003
parent reply Patrick Down <pat codemoon.com> writes:
"Sean L. Palmer" <seanpalmer directvinternet.com> wrote in
news:avrnni$uhd$1 digitaldaemon.com: 


 So have you decided how to handle C's __LINE__, __FILE__, __DATE__,
 and __TIME__ builtin macros?  Those are useful for source control
 systems, automated builds, external preprocessors, and various other
 stuff.  None of these have D equivalents to my knowledge. 
 Unfortunately it seems they'd need some keywords.  Maybe there could
 be one keyword sort of like __declspec that controls all this kind of
 stuff? 


How about:

build.file
build.date
build.line
Jan 12 2003
parent reply Burton Radons <loth users.sourceforge.net> writes:
Patrick Down wrote:

 "Sean L. Palmer" <seanpalmer directvinternet.com> wrote in
 news:avrnni$uhd$1 digitaldaemon.com: 
 
 

So have you decided how to handle C's __LINE__, __FILE__, __DATE__,
and __TIME__ builtin macros?  Those are useful for source control
systems, automated builds, external preprocessors, and various other
stuff.  None of these have D equivalents to my knowledge. 
Unfortunately it seems they'd need some keywords.  Maybe there could
be one keyword sort of like __declspec that controls all this kind of
stuff? 


 
 
 How about:
 
 build.file
 build.date
 build.line


debug.file/date/line.  We already have the keyword.
Jan 12 2003
parent reply "Daniel Yokomiso" <daniel_yokomiso yahoo.com.br> writes:
"Burton Radons" <loth users.sourceforge.net> escreveu na mensagem
news:avsa02$2vt0$1 digitaldaemon.com...

 Patrick Down wrote:

 "Sean L. Palmer" <seanpalmer directvinternet.com> wrote in
 news:avrnni$uhd$1 digitaldaemon.com:



So have you decided how to handle C's __LINE__, __FILE__, __DATE__,
and __TIME__ builtin macros?  Those are useful for source control
systems, automated builds, external preprocessors, and various other
stuff.  None of these have D equivalents to my knowledge.
Unfortunately it seems they'd need some keywords.  Maybe there could
be one keyword sort of like __declspec that controls all this kind of
stuff?



 How about:

 build.file
 build.date
 build.line


 debug.file/date/line.  We already have the keyword.


Hmmm more builtins and keywords. Somehow I think this can get out of
control. Which of these are inherent to source code (IMHO line is, the other
I don't think so).


---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.435 / Virus Database: 244 - Release Date: 30/12/2002
Jan 12 2003
parent reply "Walter" <walter digitalmars.com> writes:
"Daniel Yokomiso" <daniel_yokomiso yahoo.com.br> wrote in message
news:avt5fn$138h$1 digitaldaemon.com...

 Hmmm more builtins and keywords. Somehow I think this can get out of
 control. Which of these are inherent to source code (IMHO line is, the


other

 I don't think so).


Most of the use of those is for the assert macro. In D, assert is built in
to the language. For build versioning, use things like:

BUILDVERSION := $(shell date +%Y%m%d%H)

-DBUILDVERSION=$(BUILDVERSION)

or similar things I've seen, such as a program that all it did was write
out:

    char version[] = "date string";

and then that file was part of the make build.
Jan 13 2003
parent Ilya Minkov <midiclub 8ung.at> writes:
I'm sorry, i'm afraid i've started a redundant off-topic discussion. But 
what i meant, is not at all a "preprocessor" in the sense understood by 
most of the C people! I should have chosen other topic. I personally 
always favor syntax-based solutions over text-based, don't misundersand me.

Walter wrote:

 "Daniel Yokomiso" <daniel_yokomiso yahoo.com.br> wrote in message
 news:avt5fn$138h$1 digitaldaemon.com...
 

Hmmm more builtins and keywords. Somehow I think this can get out of
control. Which of these are inherent to source code (IMHO line is, the


 
 other
 

I don't think so).


 
 
 Most of the use of those is for the assert macro. In D, assert is built in
 to the language. For build versioning, use things like:
 
 BUILDVERSION := $(shell date +%Y%m%d%H)
 
 -DBUILDVERSION=$(BUILDVERSION)
 
 or similar things I've seen, such as a program that all it did was write
 out:
 
     char version[] = "date string";
 
 and then that file was part of the make build.
Jan 13 2003
prev sibling parent reply Ilya Minkov <midiclub 8ung.at> writes:
Walter wrote:

 One major design goal of D was to eliminate the text preprocessor.
 This involved figuring out what people were using the preprocessor
 for, and then finding a way to do that symbolically. I believe that D
 covers this well, having symbolic capability for generics, asserts,
 imports, constants, debug compiles, conditional compilation, etc.


I know. What i am going to write, is to be a tool to replace more
complex processing, like lex, yacc, adding *new* constructs, and so on,
without making the language into garbage. It will not replace C
preprocessor in any way. Well, except if some maliscious operator writes
that kind of script for it. ;) I bet the use of such a complex,
sophisticated instrument will remain very limited and will not take such
scale as C preprocessor which is crude and easy to (mis)use. I don't 
need a tool for the features of the C preprocessor i was using it for in 
C, since D is (or is to be) powerful enough.


 Examples of useful uses for such a preprocesor in D: - implement
 some syntactic features which don't make it into the language,
 either because Walter doesn'tlike them, or because he lacks of
 time. Maybe they just shouldn't pollute the language itself. Like:
 patternmatching, base#value digital constants and so on... -
 implement other languages. Like to compile Delphi or Ada programs
 :)


 What D doesn't support are things like:
 
 #define BEGIN { #define END }
 
 which are used to make C look like another language


implement, not mimic :)


 
 Using D doesn't mean you can't preprocess the source text - any
 general purpose macro text processor can be used to preprocess text
 before feeding it to the D compiler. The C preprocessor is so


I know.


 wretchedly primitive anyway, the only reason it is in wide use is
 because people are so familiar with it.


I'm afraid I was somehow short of a time and didn't explain it to the
extent. It is not going to be a macro processor, not at all. It is gonna
become a plugin- (or script-) steered *partial parse tree generator*,
and originally the tree would be output back into a D file to be
compiled. If it finds use, you may later integrate it into the compiler,
but i doubt it would make sense, since it is not for widespread use,
since D doesn't seem to have real gaps or misfeatures. And the features 
of functional programming which I would like to implement using such 
tool, can not be implemented with a text-based tool. I don't even want 
the tool to become too widespread, because:
  - foreign utilities would have to integrate it to make any kind of 
parsing of the code, using its features;
  - all other facts you pointed put already here or in the manual, which 
i *have* read.

thanks for your efforts,
and good luck.
-i
Jan 13 2003
parent reply Evan McClanahan <evan dontSPAMaltarinteractive.com> writes:
Ilya Minkov wrote:

 I'm afraid I was somehow short of a time and didn't explain it to the
 extent. It is not going to be a macro processor, not at all. It is gonna
 become a plugin- (or script-) steered *partial parse tree generator*,
 and originally the tree would be output back into a D file to be
 compiled. If it finds use, you may later integrate it into the compiler,
 but i doubt it would make sense, since it is not for widespread use,
 since D doesn't seem to have real gaps or misfeatures. And the features 
 of functional programming which I would like to implement using such 
 tool, can not be implemented with a text-based tool. I don't even want 
 the tool to become too widespread, because:
  - foreign utilities would have to integrate it to make any kind of 
 parsing of the code, using its features;
  - all other facts you pointed put already here or in the manual, which 
 i *have* read.


still, rather than tihs I would like to have some sort of robust sytax 
expansion mechanism built into the compiler, rather than something that 
happens in a separate step.

Evan
Jan 14 2003
parent reply Ilya Minkov <midiclub tiscali.de> writes:
Evan McClanahan wrote:

 still, rather than tihs I would like to have some sort of robust sytax 
 expansion mechanism built into the compiler, rather than something that 
 happens in a separate step.


I somehow think that it doesn't matter for design whether it's a 
separate programme or a part of a compiler. I shall try to make sure 
later integration is not too painful. And besides, Walter is probably 
not interested in such a thing.

Please recall that C++ originally was a separate step which generated C 
code? It is obviuosly better to integrate such complex things into a 
compiler, but only after it is shown that they make sense and advantage.

Besides, even C Preprocessor is nowadays often an integral part of the 
compiler. Even such a simple-minded compiler as LCC integrates it into 
the lexer. The only exception i'm aware of is GCC, but that's quite 
another story.

I shall give my best to design it robust, but it might take some trial 
and error, so that it's likely to change completely. Until it's proven 
to work well and be safe, under no circumstances should it be stuffed 
into the compiler.


-i.

PS. I guess someone has figured out already how to write backends for D? 
Where can this information be found? I might be interested to write a 
portable fast VM backend based on GNU Lightning or VCode.
Jan 14 2003
parent reply "Walter" <walter digitalmars.com> writes:
"Ilya Minkov" <midiclub tiscali.de> wrote in message
news:b0127n$114f$1 digitaldaemon.com...

 Besides, even C Preprocessor is nowadays often an integral part of the
 compiler. Even such a simple-minded compiler as LCC integrates it into
 the lexer. The only exception i'm aware of is GCC, but that's quite
 another story.

 I shall give my best to design it robust, but it might take some trial
 and error, so that it's likely to change completely. Until it's proven
 to work well and be safe, under no circumstances should it be stuffed
 into the compiler.


I agree. I think you are taking the right approach. And there is no reason
at all why D would not be suitable as a "back end" for a higher level
language.
Jan 24 2003
parent Ilya Minkov <midiclub 8ung.at> writes:
Walter wrote:

 "Ilya Minkov" <midiclub tiscali.de> wrote in message
 news:b0127n$114f$1 digitaldaemon.com...
 

Besides, even C Preprocessor is nowadays often an integral part of the
compiler. Even such a simple-minded compiler as LCC integrates it into
the lexer. The only exception i'm aware of is GCC, but that's quite
another story.

I shall give my best to design it robust, but it might take some trial
and error, so that it's likely to change completely. Until it's proven
to work well and be safe, under no circumstances should it be stuffed
into the compiler.


 
 
 I agree. I think you are taking the right approach. And there is no reason
 at all why D would not be suitable as a "back end" for a higher level
 language.
 


Talking about that. It requeres language processing tools, such as 
generating ASTs and such work. I have found an interpreted C-like OO 
language "Dino" with perfect capabilities of that kind. I thought D 
deserved similar things implemented as a library.

The severe thing i dislike about it - it has taken your file extention - 
"*.d".

It appears that it is a "concept test" for a compiler construction toolset.

http://cocom.sourceforge.net/

Pre-init constants come to mind. But *may be* i could write some types/ 
op-overloading/ wizardry to write recursive-descend parsers directly in 
D in a rather BNF-like form...
Hm. If i really get that to work, i'm raising my level to a 
mega-wizard... err... of level (-4). :>
That is, a very powerful wizard. It would be not unusual trickery in C, 
but in D... Hm.

BUT, if i get it to make a grammar description, than it need not be 
limited by recursive-descend, it can be anything at all!


-i.
Jan 24 2003