www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A idea to improve diet-ng compile speed!

reply workman <workman gmail.com> writes:
diet-ng is great and I love to use it,  but it is too slow to 
compile.

One idea to speedup is use preprocessor like DMD compile C header 
file with GCC -E.

A diet-ng compiler write in D, translate template file into D 
source code, then mixin from projects.  A runtime diet-ng 
compiler will easaly speed up 1000 times compare to CTFE version, 
and you don't get any performance loss.

A runtime diet-ng compiler also possible compile template into 
dynamic library file,   you edit view files and a file monitor 
will recompile the dynamic library then http server reload it, no 
need recompile full projects.

Add new feature into diet will be more easy, since developer 
don't limit to use CTFE code.

D CTFE is great, but not suit for task like dieg-ng.
Aug 12 2021
next sibling parent reply workman <workman gmail.com> writes:
On Thursday, 12 August 2021 at 07:55:08 UTC, workman wrote:

 D CTFE is great, but not suit for task like dieg-ng.


simple code like this, finish in 0.004s for 20 files. ldc2 take 
hours to finish in product mode.


```c
import diet.defs;
import diet.dom;
import diet.input;
import diet.parser;
import diet.html;

import std.stdio;
import std.file;
import std.path;
int main(string[] args){
	foreach(filepath; args[1..$]){
		scope filename	= baseName(filepath);
		scope ext	= extension(filepath);
		if( ext != ".dt" ) {
			writefln("%s extension %s is not .dt", filepath, ext);
			break;
		}
		scope dfile	= filepath[0..$-1];
		try {
			scope data	= cast(string) read(filepath);
			scope doc	= parseDiet(data, filename);
			scope code	= getHTMLMixin(doc, dietOutputRangeName, 
HTMLOutputStyle.compact);
			if( std.file.exists(dfile) && isFile(dfile) ) {
				scope _code = cast(string) read(dfile);
				if(  _code == code ) continue;

			}
			writefln("%s => %s", filepath, dfile);
			if( is_cached ) {
				continue;
			}
			std.file.write(dfile, code);

		} catch(Exception e) {
			writefln("%s throw %s", filepath, e);
			break;
		}
	}
	return 0;
}

```

dieg not support inherit and layout like 
https://mozilla.github.io/nunjucks/ or 
https://jinja.palletsprojects.com/en/3.0.x/, to add more advance 
function CTFE will  crashed to compile.
Aug 12 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/12/21 9:45 AM, workman wrote:

 On Thursday, 12 August 2021 at 07:55:08 UTC, workman wrote:

 D CTFE is great, but not suit for task like dieg-ng.


 
 simple code like this, finish in 0.004s for 20 files. ldc2 take hours to 
 finish in product mode.


 From experience, I think the 2 hours is probably not because of diet 
CTFE interpretation. But I could be wrong, I don't know what your 
application is like.


 
 dieg not support inherit and layout like 
 https://mozilla.github.io/nunjucks/ or 
 https://jinja.palletsprojects.com/en/3.0.x/, to add more advance 
 function CTFE will  crashed to compile.


diet supports layout inheritance via the `extends` directive.

e.g. all my files look like:

```pug
extends layout
block extraCss
   // extra css includes
block extraJs
   // extra js includes
block content
   // page content
```

Where `layout.dt` provides the overall layout.

-Steve
Aug 12 2021
parent reply workman <workman gmail.com> writes:
On Thursday, 12 August 2021 at 13:49:32 UTC, Steven Schveighoffer 
wrote:


 But changing the code means recompiling. And that is not always 
 easy, because diet templates are compiled with specific context 
 and aliases, which are not easy to replicate as a subpart of 
 the project. I have considered splitting my routes out into 
 individual subprojects to see if that helps with the build 
 times.



Mixed d code with views made vibe.d like old PHP code, it hard to 
maintain if business logic change fast, and not easy to testing.


I am consider use a javascript server side render template like



On Thursday, 12 August 2021 at 14:04:03 UTC, Steven Schveighoffer 
wrote:

 On 8/12/21 9:45 AM, workman wrote:
 From experience, I think the 2 hours is probably not because of 
 diet CTFE interpretation. But I could be wrong, I don't know 
 what your application is like.


Not really take hours, but very slow and require a lot memory.



 diet supports layout inheritance via the `extends` directive.

 e.g. all my files look like:

 ```pug
 extends layout
 block extraCss
   // extra css includes
 block extraJs
   // extra js includes
 block content
   // page content
 ```



Thanks for the tips, I will try extends. What will be really nice 
is some thing like:

```twig
{% extends "base.html" %}

     {% block sidebar %}
         <h3>Table Of Contents</h3>
         ...
         {{ parent() }}
     {% endblock %}
     ```
Aug 12 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/12/21 10:11 AM, workman wrote:

 On Thursday, 12 August 2021 at 13:49:32 UTC, Steven Schveighoffer wrote:
 

 But changing the code means recompiling. And that is not always easy, 
 because diet templates are compiled with specific context and aliases, 
 which are not easy to replicate as a subpart of the project. I have 
 considered splitting my routes out into individual subprojects to see 
 if that helps with the build times.


 
 
 Mixed d code with views made vibe.d like old PHP code, it hard to 
 maintain if business logic change fast, and not easy to testing.
 
 
 I am consider use a javascript server side render template like
 
 
 
 On Thursday, 12 August 2021 at 14:04:03 UTC, Steven Schveighoffer wrote:

 On 8/12/21 9:45 AM, workman wrote:
 From experience, I think the 2 hours is probably not because of diet 
 CTFE interpretation. But I could be wrong, I don't know what your 
 application is like.


 
 Not really take hours, but very slow and require a lot memory.


OK. My experience is that precompiling templates and using the cache 
feature reduced my build time from 38 seconds to 11, and also reduced 
memory consumption quite a bit. However, you still must build production 
without these modes. I would like to introduce a production 
precompilation mode, where the diet processing is done separately, and 
the code just imports the file without having to checksum it. But I 
haven't had the need yet.

If you have memory constraints that prevent you from building 
production, this might be a viable path.


 
 

 diet supports layout inheritance via the `extends` directive.

 e.g. all my files look like:

 ```pug
 extends layout
 block extraCss
   // extra css includes
 block extraJs
   // extra js includes
 block content
   // page content
 ```


 
 
 Thanks for the tips, I will try extends. What will be really nice is 
 some thing like:
 
 ```twig
 {% extends "base.html" %}
 
      {% block sidebar %}
          <h3>Table Of Contents</h3>
          ...
          {{ parent() }}
      {% endblock %}
      ```


Yes, that is exactly what `extends` does. Same keywords even. See 
[docs](https://github.com/rejectedsoftware/diet-ng/blob/master/SPEC.md
extension-includes) 
(they are a bit sparse, but there's an example there to help you 
understand how to do it).

-Steve
Aug 12 2021
parent reply workman <workman gmail.com> writes:
On Thursday, 12 August 2021 at 14:51:28 UTC, Steven Schveighoffer 
wrote:

 Yes, that is exactly what `extends` does. Same keywords even. 
 See 
 [docs](https://github.com/rejectedsoftware/diet-ng/blob/master/SPEC.md
extension-includes) (they are a bit sparse, but there's an example there to
help you understand how to do it).

 -Steve


Thanks again for the tips.

The `extends file(.ext)` is not implement yet, and `include 
section.dt` is not work like other popular template system ( need 
super() or parent() ).


I am think of use https://mozilla.github.io/nunjucks/ as a client 
side solution, it is easy to add a server side render solution 
with quickjs or a nodejs.
Aug 12 2021
parent workman <workman gmail.com> writes:
On Thursday, 12 August 2021 at 15:48:29 UTC, workman wrote:

 On Thursday, 12 August 2021 at 14:51:28 UTC, Steven 
 Schveighoffer wrote:

 Yes, that is exactly what `extends` does. Same keywords even. 
 See 
 [docs](https://github.com/rejectedsoftware/diet-ng/blob/master/SPEC.md
extension-includes) (they are a bit sparse, but there's an example there to
help you understand how to do it).

 -Steve


 Thanks again for the tips.

 The `extends file(.ext)` is not implement yet, and `include 
 section.dt` is not work like other popular template system ( 
 need super() or parent() ).


 I am think of use https://mozilla.github.io/nunjucks/ as a 
 client side solution, it is easy to add a server side render 
 solution with quickjs or a nodejs.


`extends file(.ext)` work for me.
Aug 12 2021
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/12/21 3:55 AM, workman wrote:

 diet-ng is great and I love to use it,  but it is too slow to compile.
 
 One idea to speedup is use preprocessor like DMD compile C header file 
 with GCC -E.
 
 A diet-ng compiler write in D, translate template file into D source 
 code, then mixin from projects.  A runtime diet-ng compiler will easaly 
 speed up 1000 times compare to CTFE version, and you don't get any 
 performance loss.
 
 A runtime diet-ng compiler also possible compile template into dynamic 
 library file,   you edit view files and a file monitor will recompile 
 the dynamic library then http server reload it, no need recompile full 
 projects.
 
 Add new feature into diet will be more easy, since developer don't limit 
 to use CTFE code.
 
 D CTFE is great, but not suit for task like dieg-ng.


diet-ng is really hard to deal with at runtime.

I added a feature that allows any *non-code* portions of the template to 
be reinterpreted at runtime, so for instance, if you want to add or 
remove an html class from an element, that is doable without rebuilding. 
I also created a [diet 
precompiler](https://code.dlang.org/packages/dietpc) to move most of the 
CTFE heavy-lifting to a pre-compilation step (it still runs a checksum 
on the files at compile-time, which is annoying to me, but I haven't 
bothered yet to strip that part out).

But changing the code means recompiling. And that is not always easy, 
because diet templates are compiled with specific context and aliases, 
which are not easy to replicate as a subpart of the project. I have 
considered splitting my routes out into individual subprojects to see if 
that helps with the build times.

With precompiling templates and live mode, diet-ng is at least tolerable 
for development.

If I were to do my own template system, with my experience in diet-ng, 
it would be focused on runtime interpretation (only a change to the 
input data structure or backend code would require a recompile) with the 
ability to compile everything for performance.

-Steve
Aug 12 2021