digitalmars.D - DIP81: Writing files at compile time
- JDemler (5/5) Aug 12 2015 Triggered by the original forum thread I wrote a DIP to further
- "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= (3/8) Aug 12 2015 1. How do you order imports and exports?
- Adam D. Ruppe (9/9) Aug 12 2015 I've been looking at a company's build system recently and it
- "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= (6/10) Aug 12 2015 Yes, you cannot use the file system for this. You could make it
- JDemler (10/21) Aug 12 2015 I see your point. But not using the file system would imply that
- "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= (14/20) Aug 12 2015 I think you should forget that there is a filesystem. You may
- "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= (1/1) Aug 12 2015 And for this to work you also need a highly concurrent compiler.
- JDemler (9/29) Aug 12 2015 I am not sure if I understand your idea correctly:
- "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= (10/17) Aug 13 2015 The filesystem is a compiler issue and not a language issue, so
- JDemler (22/39) Aug 13 2015 To summarize:
- "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= (14/27) Aug 13 2015 You need a highly concurrent compiler. If you import in one
- JDemler (23/29) Aug 14 2015 While I basically agree that this would be a nice to have feature
- lobo (2/11) Aug 12 2015 +100
- Dicebot (1/1) Aug 12 2015 Please no.
- Tofu Ninja (10/15) Aug 12 2015 The benefits of this I see are debugging, actually having the
- Nicholas Wilson (3/16) Aug 12 2015 I think this is a much better idea.
- JDemler (8/27) Aug 12 2015 While this might work for very simple and basic mixins, with the
- Tofu Ninja (6/35) Aug 12 2015 I suppose such an expansion flag would need to expand templates
- Dmitry Olshansky (5/22) Aug 12 2015 I agree with Ola, something more restricted then filesystem is required.
- Jacob Carlborg (7/13) Aug 12 2015 I agree, this is much better. I remember the old Eclipse plugin,
- ixid (4/20) Aug 13 2015 This would be great but we would need some support for the
- Rikki Cattermole (34/38) Aug 12 2015 Problem:
- JDemler (13/14) Aug 13 2015 So in the vibe.d example we would generate a file diet.di which
- Rikki Cattermole (11/23) Aug 13 2015 If it is done via D interface files, you would be in control of where it...
- Stefan Koch (3/8) Mar 08 2016 I like the idea of being able to write files at compile-time.
- =?UTF-8?Q?S=c3=b6nke_Ludwig?= (8/18) Mar 08 2016 No progress on the DIP, I made an alternative implementation for
Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!
Aug 12 2015
"Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!1. How do you order imports and exports? 2. How do you support parallel compilation on a cluster?
Aug 12 2015
I've been looking at a company's build system recently and it makes me think this is a bad idea: we have enough problems tracking import dependencies and changes as it is without other files being written in the middle of the process too. I'd prefer it if these were always done as separate steps so it can be more easily integrated into other build scripts. (Build systems and scripts suck btw, but they suck less than the confusion caused when files can be read and written arbitrarily by the compiler, which already has hidden dependencies).
Aug 12 2015
On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe wrote:I've been looking at a company's build system recently and it makes me think this is a bad idea: we have enough problems tracking import dependencies and changes as it is without other files being written in the middle of the process too.Yes, you cannot use the file system for this. You could make it work by having a central write-once key-value database, then block all imports from non-existing keys until they become available. And error out if either keys never materialize or if a key is written twice.
Aug 12 2015
On Wednesday, 12 August 2015 at 19:12:53 UTC, Ola Fosheim Grøstad wrote:On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe wrote:I see your point. But not using the file system would imply that the debugging and transparency advantages would not work. Maybe a combination of both could work? A central database that tracks which files have been generated and which have not, but the imported code still lies on the file system. Although that seems overly complex and would imply a differantiation in the import syntax between generated and non generated files.I've been looking at a company's build system recently and it makes me think this is a bad idea: we have enough problems tracking import dependencies and changes as it is without other files being written in the middle of the process too.Yes, you cannot use the file system for this. You could make it work by having a central write-once key-value database, then block all imports from non-existing keys until they become available. And error out if either keys never materialize or if a key is written twice.
Aug 12 2015
On Wednesday, 12 August 2015 at 20:21:06 UTC, JDemler wrote:Maybe a combination of both could work? A central database that tracks which files have been generated and which have not, but the imported code still lies on the file system. Although that seems overly complex and would imply a differantiation in the import syntax between generated and non generated files.I think you should forget that there is a filesystem. You may have at least 4 storage areas: 1. a source bundle 2. generated source bundle 3. an output bundle 4. perhaps a temporary storage area So you need to differentiate between those. You don't have to differentiate between generated and non-generated if you only allow keys in 2 that do not exist in 1. When you reference a module you simply look at 1. first, if it does not exist you try 2. If you can write files to the output bundle (like .ini or .xml files etc) you might also need specify the mime-type.
Aug 12 2015
And for this to work you also need a highly concurrent compiler.
Aug 12 2015
On Wednesday, 12 August 2015 at 20:39:31 UTC, Ola Fosheim Grøstad wrote:On Wednesday, 12 August 2015 at 20:21:06 UTC, JDemler wrote:I am not sure if I understand your idea correctly: The compiler would in case of an export(name, content) write the content both to a internal database and the filesystem and then only use the internal one? Or could the content of the internal database be copied to the file system at the end of the compilation process? Or is only the output bundle written to the file system?Maybe a combination of both could work? A central database that tracks which files have been generated and which have not, but the imported code still lies on the file system. Although that seems overly complex and would imply a differantiation in the import syntax between generated and non generated files.I think you should forget that there is a filesystem. You may have at least 4 storage areas: 1. a source bundle 2. generated source bundle 3. an output bundle 4. perhaps a temporary storage area So you need to differentiate between those. You don't have to differentiate between generated and non-generated if you only allow keys in 2 that do not exist in 1. When you reference a module you simply look at 1. first, if it does not exist you try 2. If you can write files to the output bundle (like .ini or .xml files etc) you might also need specify the mime-type.
Aug 12 2015
On Thursday, 13 August 2015 at 00:54:37 UTC, JDemler wrote:I am not sure if I understand your idea correctly: The compiler would in case of an export(name, content) write the content both to a internal database and the filesystem and then only use the internal one? Or could the content of the internal database be copied to the file system at the end of the compilation process? Or is only the output bundle written to the file system?The filesystem is a compiler issue and not a language issue, so sure, the compiler could do whatever it wants, flush everything to disk or into a SQL database or… Keep in mind that someone might want to compile on a diskless computer from an in-memory zip-file or similar. I think the important file generation that are relevant to the language would be for the output bundle, so that you e.g. can have a config file in D and generate platform specific files (.ini, .xml etc).
Aug 13 2015
On Thursday, 13 August 2015 at 08:19:15 UTC, Ola Fosheim Grøstad wrote:On Thursday, 13 August 2015 at 00:54:37 UTC, JDemler wrote:To summarize: We would have a central registry handling all the generated and non generated files. If an import is triggered, this registry is then searched for the imported file. If this file does not already exist, the import blocks. Writing these generated files to the file system would be optional, triggered by a compiler switch and happen after the compilation process is complete. This solves the concurrency problem. But what about non concurrent compiler runs? A block there would never be resolved. I do not know enough about compilers to judge this problem. Can a sane way of compiling the files be found easily? Can the compiler switch to compile something else when it is blocked on an import? And I do not understand why you differentiate between generated source files and generated non source files (.xml, .ini). As both types can be imported (or read at compile time) I do not think we should treat them differently. Also we would need a way to tell the one from the other (file extension? different parameter in the export syntax?). In my understanding storage area 2 and 3 should be merged.I am not sure if I understand your idea correctly: The compiler would in case of an export(name, content) write the content both to a internal database and the filesystem and then only use the internal one? Or could the content of the internal database be copied to the file system at the end of the compilation process? Or is only the output bundle written to the file system?The filesystem is a compiler issue and not a language issue, so sure, the compiler could do whatever it wants, flush everything to disk or into a SQL database or… Keep in mind that someone might want to compile on a diskless computer from an in-memory zip-file or similar. I think the important file generation that are relevant to the language would be for the output bundle, so that you e.g. can have a config file in D and generate platform specific files (.ini, .xml etc).
Aug 13 2015
On Thursday, 13 August 2015 at 09:52:02 UTC, JDemler wrote:This solves the concurrency problem. But what about non concurrent compiler runs? A block there would never be resolved. I do not know enough about compilers to judge this problem. Can a sane way of compiling the files be found easily? Can the compiler switch to compile something else when it is blocked on an import?You need a highly concurrent compiler. If you import in one definition and export in the next definition, then the first one would block and be queued, and the compiler would move on to the next definition. In the actor-model (co-routines/fibers) each definition will spawn a new actor.And I do not understand why you differentiate between generated source files and generated non source files (.xml, .ini). As both types can be imported (or read at compile time) I do not think we should treat them differently. Also we would need a way to tell the one from the other (file extension? different parameter in the export syntax?). In my understanding storage area 2 and 3 should be merged.Area 2 is pre-populated with area 1, so they are the same. You need to be able to tell the compiler what files should be included in the output bundle. Actually, I think the file-model is too simple. I think you need a more advanced key-value database so that you can update different fields in the same ".ini file" from different source files.
Aug 13 2015
On Thursday, 13 August 2015 at 10:06:32 UTC, Ola Fosheim Grøstad wrote:You need to be able to tell the compiler what files should be included in the output bundle.While I basically agree that this would be a nice to have feature I do not think it is necessary and it would not make up for the added complexity inside the export syntax. If a specific source file should not be written to the file system one can still use mixins.Actually, I think the file-model is too simple. I think you need a more advanced key-value database so that you can update different fields in the same ".ini file" from different source files.This would a huge amount of complexity to the feature and would not work in general. One can add keys to .ini and maybe .config files but not to a .sql or .css file. Also a append or prepend feature would make it impossible for the compiler to tell when a generated file is finished. Based on this discussion I updated the DIP. Included are now the idea of mixin and template expanding (in the section alternatives) and Ola's solution to the basic problems which I originally missed out on (in the section implementation considerations). All in all I have to admit that the original approach was not designed well enough. It was simple but not functional. Alas, this new approach brings with it a completly different implementation complexity. Nevertheless, the idea of writing files at compile time is still worth discussing about.
Aug 14 2015
On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe wrote:I've been looking at a company's build system recently and it makes me think this is a bad idea: we have enough problems tracking import dependencies and changes as it is without other files being written in the middle of the process too. I'd prefer it if these were always done as separate steps so it can be more easily integrated into other build scripts. (Build systems and scripts suck btw, but they suck less than the confusion caused when files can be read and written arbitrarily by the compiler, which already has hidden dependencies).+100
Aug 12 2015
On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins. A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
Aug 12 2015
On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:I think this is a much better idea. Nic[...]The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins. A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
Aug 12 2015
On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:While this might work for very simple and basic mixins, with the combination of TMP and compile time reflection this becomes not only impractical but also impossible. Think again about the vibe.d example: There is one mixin handeling all the template-type combinations. How would such a mixin be expanded? The resulting code differs from template to template and from type to type.Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins. A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
Aug 12 2015
On Thursday, 13 August 2015 at 00:58:14 UTC, JDemler wrote:On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:I suppose such an expansion flag would need to expand templates as well, which is still not a bad idea. Templates can be hard to follow sometimes and expanding them out in all their forms could be helpful for debugging and for tooling as well. Though there would be a lot of expansions.On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:While this might work for very simple and basic mixins, with the combination of TMP and compile time reflection this becomes not only impractical but also impossible. Think again about the vibe.d example: There is one mixin handeling all the template-type combinations. How would such a mixin be expanded? The resulting code differs from template to template and from type to type.Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins. A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
Aug 12 2015
On 13-Aug-2015 02:27, Tofu Ninja wrote:On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:pragma(msg, your_mixin);Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins.A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.I agree with Ola, something more restricted then filesystem is required. -- Dmitry Olshansky
Aug 12 2015
On 2015-08-13 01:27, Tofu Ninja wrote:A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.I agree, this is much better. I remember the old Eclipse plugin, Descent. It had a compile time view which showed how mixins where expanded and how some language features where lowered, i.e. "scope" to try-catch-finally. -- /Jacob Carlborg
Aug 12 2015
On Thursday, 13 August 2015 at 06:53:14 UTC, Jacob Carlborg wrote:On 2015-08-13 01:27, Tofu Ninja wrote:This would be great but we would need some support for the formatting I think, mixin code tends to be a mess, it would be nice to have some basic auto indent or general FMT style support.A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.I agree, this is much better. I remember the old Eclipse plugin, Descent. It had a compile time view which showed how mixins where expanded and how some language features where lowered, i.e. "scope" to try-catch-finally.
Aug 13 2015
On 13/08/2015 6:37 a.m., JDemler wrote:Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!Problem: debugging The debugger cannot attach to mixedin code; one has to print the resulting code with pragma(msg, ) to inspect it Fix: Make D interface file generation include the source code e.g. statements. Pros: - Already have most of the code to do this Cons: - None. Problem: compile-speed Even if the resulting code does not change, mixedin code has to be compiled anyway True. Why is this a problem exactly? Problem: scalability If the generated code is needed in two seperated places of the code, it has to be mixedin twice or has to be mixedin into a special module which introduces overhead Not a problem? The result should *theoretically* the return value should be cached. We could even go so far as to have ``mixin(string)`` as the return type. To cache the AST and force compile time only. Perhaps even make it return the assembly code itself for runtime? Problem: transparency As user of a library which relys on compile time code generation one has often no idea what code is generated See debugging problem for the solution. We do not need to add a new language feature based upon these problems and use cases. We can solve it by simply extending one we already have. D interface files. Making them generate the full source code instead of just the interface.
Aug 12 2015
On Thursday, 13 August 2015 at 04:58:06 UTC, Rikki Cattermole wrote:[...]So in the vibe.d example we would generate a file diet.di which would include all the generated diet templates of the whole project? Maybe 100 * 50 extra lines of code for a medium sized project? This is not debuggable, understandable or transparent. Especially because the generated code lies inside a file of a library which I should not need to know about. Also I do not understand how we could expand template instatiations. Maybe it would be possible for basic templates that only take types but imagine a template that takes a parse tree as a value. How would an expanded version of that look like?
Aug 13 2015
On 13/08/2015 7:57 p.m., JDemler wrote:On Thursday, 13 August 2015 at 04:58:06 UTC, Rikki Cattermole wrote:If it is done via D interface files, you would be in control of where it lands. It would not go into a dub package directory that you do not control for your project. You want it to be debuggable, this is. Is it nice and pretty? No. If you are interested in only one code path i.e. specific template arguments to outputted and not the others, then perhaps a pragma can be used to limit it.[...]So in the vibe.d example we would generate a file diet.di which would include all the generated diet templates of the whole project? Maybe 100 * 50 extra lines of code for a medium sized project? This is not debuggable, understandable or transparent. Especially because the generated code lies inside a file of a library which I should not need to know about.Also I do not understand how we could expand template instatiations. Maybe it would be possible for basic templates that only take types but imagine a template that takes a parse tree as a value. How would an expanded version of that look like?Template if statement. Also argument overloads. Also remember, the information must exist to expand it. If it didn't, we sure couldn't generate assembly.
Aug 13 2015
On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!I like the idea of being able to write files at compile-time. Is there any progress on this.
Mar 08 2016
Am 08.03.2016 um 11:28 schrieb Stefan Koch:On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:No progress on the DIP, I made an alternative implementation for vibe.d's Diet templates using multiple `shared static this()` and writing the files at run time: https://github.com/rejectedsoftware/vibe.d/pull/1385 Performance results are very promising. I don't know which way is nicer (or uglier), but at least for this use case, the DIP isn't strictly necessary.Triggered by the original forum thread I wrote a DIP to further explain the idea of the writing files at compile time feature and its implications. http://wiki.dlang.org/DIP81 Please discuss!I like the idea of being able to write files at compile-time. Is there any progress on this.
Mar 08 2016