www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [GSoC] 'Replace Runtime Hooks with Templates' progress and update

reply Dan Printzell <xwildn00bx gmail.com> writes:
Hey everyone,

I have now started with my Google Summer of Code project, Replace
Runtime Hooks with Templates. A small recap on what my project is 
about:

       A lot of the language features depend on runtime hooks,
       which in turn requires the TypeInfo class. This is a problem
       for betterC code, as classes are not supported. Another
       problem is that the current hooks can lie about their
       safety. Both of these problems can be solved by moving
       runtime hooks to use templates instead. This solves the
       betterC issue by removing the dependency of classes (the
       TypeInfo class), and it solves the safety issue because now
       the compiler will have all the information about the hook
       and the type it works on.

       This project will work on translating all the array hooks
       from using the TypeInfo class to using templates.

The full proposal pdf and LaTeX files can be found in the 
following
Github repo: [https://github.com/Vild/GSOC2019-Proposal]


This month
==========

As a warm-up project I have been working on learning about the 
dmd and
druntime codebases by implementing more descriptive error 
messages for
when a template hook is missing.
[https://github.com/dlang/dmd/pull/9839]

I have also read though, more in-depth, all of the previous hooks
conversion pull-requests to try and build up a mental picture of 
all the
code their conversion touched, and their problems and solutions.


Next month
==========

Over the next four weeks I will translate following hooks:

Week 22-23:
- _d_arrayappend*
Week 24-25:
- _d_arraysetctor
- _d_arrayctor
- _d_arraycat*

The first two weeks will be allocated to only the _d_arrayappend*
functions as these are the first hooks I will be translating and 
I will
also need to study and do two exams.


Blockers
========

So far I have not encountered any problems with the project.
May 28
next sibling parent reply Yatheendra <3df4 gmail.ru> writes:
Just a lurker here.

What would be the impact on compilation time of programs (not 
that of the runtime)?
May 28
parent reply Dan Printzell <xwildn00bx gmail.com> writes:
On Wednesday, 29 May 2019 at 03:06:23 UTC, Yatheendra wrote:
 Just a lurker here.

 What would be the impact on compilation time of programs (not 
 that of the runtime)?
I would guess that if the program uses a lot of array functionality with a lot of different types, that it would slow down the compilation a bit. But it should probably not be noticeable.
May 29
parent reply Yatheendra <3df4 gmail.ru> writes:
Thanks.

The reason I asked is, I remember the opposite of this approach 
being claimed as being a reason for fast compilation, i.e. having 
heavily-used features in the language instead of as templates in 
the standard library. Now, even as an implementation detail, it 
is a reversal in approach.
May 29
parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 29 May 2019 19:08:56 +0000 schrieb Yatheendra:

 Thanks.
 
 The reason I asked is, I remember the opposite of this approach being
 claimed as being a reason for fast compilation, i.e. having heavily-used
 features in the language instead of as templates in the standard
 library. Now, even as an implementation detail, it is a reversal in
 approach.
My observations here have been quite different, do you have a link / source to reinforce that statement? I've been following these newsgroups for quite some time now and Andrei's and Walters stance on this was always to move as much functions out of the compiler into the library as possible. In addition, the compiler should not do any magic, instead it lowers to code which could also have been written by the user. Now regarding this specific GSoC project: I don't think we have to worry that this will affect compilation speed a lot. The compiler code will be mostly identical, only instead of calling an extern(C) function, it calls a template. Simple template instantiations are not particularily slow though, so this should not cause much of a slow down. There is however one side-effect: Unlike the extern(C) functions used before, template instances will be visible to the optimizer. This means that parts of these hooks can be inlined and optimized to improve performance. This however might affect compile time a bit. -- Johannes
May 30
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, May 30, 2019 at 12:15:55PM +0000, Johannes Pfau via Digitalmars-d wrote:
[...]
 Now regarding this specific GSoC project: I don't think we have to
 worry that this will affect compilation speed a lot. The compiler code
 will be mostly identical, only instead of calling an extern(C)
 function, it calls a template. Simple template instantiations are not
 particularily slow though, so this should not cause much of a slow
 down.
That depends on how the template is implemented, doesn't it? If there are too many recursive templates, for example, the slowdown can be quite dramatic.
 There is however one side-effect: Unlike the extern(C) functions used
 before, template instances will be visible to the optimizer. This
 means that parts of these hooks can be inlined and optimized to
 improve performance. This however might affect compile time a bit.
[...] There's also the issue of code bloat, which should at least be considered, even if actually solving it might complicate things a bit too much for GSoC. Previously copying an array from A to B involved only a single runtime function; now we're looking at O(n) template instantiations, which potentially translates to O(n) functions that, ostensibly, do the exact same copying of some number of bytes from A to B. T -- People demand freedom of speech to make up for the freedom of thought which they avoid. -- Soren Aabye Kierkegaard (1813-1855)
May 30
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 31/05/2019 4:45 AM, H. S. Teoh wrote:
 On Thu, May 30, 2019 at 12:15:55PM +0000, Johannes Pfau via Digitalmars-d
wrote:
 [...]
 Now regarding this specific GSoC project: I don't think we have to
 worry that this will affect compilation speed a lot. The compiler code
 will be mostly identical, only instead of calling an extern(C)
 function, it calls a template. Simple template instantiations are not
 particularily slow though, so this should not cause much of a slow
 down.
That depends on how the template is implemented, doesn't it? If there are too many recursive templates, for example, the slowdown can be quite dramatic.
 There is however one side-effect: Unlike the extern(C) functions used
 before, template instances will be visible to the optimizer. This
 means that parts of these hooks can be inlined and optimized to
 improve performance. This however might affect compile time a bit.
[...] There's also the issue of code bloat, which should at least be considered, even if actually solving it might complicate things a bit too much for GSoC. Previously copying an array from A to B involved only a single runtime function; now we're looking at O(n) template instantiations, which potentially translates to O(n) functions that, ostensibly, do the exact same copying of some number of bytes from A to B.
Agreed, I'm worried about template bloat. But at the same time, I hope we can force inlining. If we can, that will solve that problem.
May 30
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 30 May 2019 at 19:20:36 UTC, rikki cattermole wrote:
 On 31/05/2019 4:45 AM, H. S. Teoh wrote:
 On Thu, May 30, 2019 at 12:15:55PM +0000, Johannes Pfau via 
 Digitalmars-d wrote:
 [...]
 Now regarding this specific GSoC project: I don't think we 
 have to
 worry that this will affect compilation speed a lot. The 
 compiler code
 will be mostly identical, only instead of calling an extern(C)
 function, it calls a template. Simple template instantiations 
 are not
 particularily slow though, so this should not cause much of a 
 slow
 down.
That depends on how the template is implemented, doesn't it? If there are too many recursive templates, for example, the slowdown can be quite dramatic.
 There is however one side-effect: Unlike the extern(C) 
 functions used
 before, template instances will be visible to the optimizer. 
 This
 means that parts of these hooks can be inlined and optimized 
 to
 improve performance. This however might affect compile time a 
 bit.
[...] There's also the issue of code bloat, which should at least be considered, even if actually solving it might complicate things a bit too much for GSoC. Previously copying an array from A to B involved only a single runtime function; now we're looking at O(n) template instantiations, which potentially translates to O(n) functions that, ostensibly, do the exact same copying of some number of bytes from A to B.
Agreed, I'm worried about template bloat. But at the same time, I hope we can force inlining. If we can, that will solve that problem.
FWIW we have been doing this for years now (have a look at the various templates in object.d, e.g. __cmp or __switch) and the fact that no one noticed nor complained, shows that this is no concern. tl;dr: the compiler already generates a special template everytime you do e.g. a switch, a cast, a comparison between classes etc. Examples: - https://run.dlang.io/is/gsCSO1 - https://run.dlang.io/is/ti7rlu There are many more.
May 31
next sibling parent Yatheendra <3df4 gmail.ru> writes:
On Friday, 31 May 2019 at 07:03:56 UTC, Seb wrote:
 FWIW we have been doing this for years now (have a look at the 
 various templates in object.d, e.g. __cmp or __switch) and the 
 fact that no one noticed nor complained, shows that this is no 
 concern.

 tl;dr: the compiler already generates a special template 
 everytime you do e.g. a switch, a cast, a comparison between 
 classes etc.

 Examples:
 - https://run.dlang.io/is/gsCSO1
 - https://run.dlang.io/is/ti7rlu

 There are many more.
Thanks. That clears things up.
May 31
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 31 May 2019 07:03:56 +0000 schrieb Seb:

 
 FWIW we have been doing this for years now (have a look at the various
 templates in object.d, e.g. __cmp or __switch) and the fact that no one
 noticed nor complained, shows that this is no concern.
 
 tl;dr: the compiler already generates a special template everytime you
 do e.g. a switch, a cast, a comparison between classes etc.
 
 Examples:
 - https://run.dlang.io/is/gsCSO1 - https://run.dlang.io/is/ti7rlu
 
 There are many more.
In addition, I don't think these hooks will be 'heavy', recursive templates or something similar. Most of them should be quite simple. Also, if we implement containers in phobos we also use templates. So there's no difference in implementing a templated array manually or making the compiler use templated hooks. In the end, it will have the same overhead (Though builtin slices are of course very commonly used). -- Johannes
Jun 02
prev sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 30 May 2019 at 16:45:12 UTC, H. S. Teoh wrote:
 On Thu, May 30, 2019 at 12:15:55PM +0000, Johannes Pfau via
 There's also the issue of code bloat, which should at least be 
 considered
Yes, it is something to be mindful of. There are a couple of things that can be done to mitigate that risk. One would be to factor out the parts of the implementation that are not type-dependent into their own functions. Those implementations would then be shared by the type-dependent implementations, reducing the potential for bloat. There is also a role for the compiler/linker to play. As a contrived example: T xor(T)(T a, T b) { return a ^ b; } int x; int y; int z = xor!int(x, y); uint a; uint b; uint c = xor!uint(a, b); The compiler/linker should be able to recognize that the `xor!int` and `xor!uint` are identical, and only one should end up the resulting binary. Mike
May 30
prev sibling next sibling parent Jonathan Marler <johnnymarler gmail.com> writes:
On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
 Hey everyone,

 I have now started with my Google Summer of Code project, 
 Replace
 Runtime Hooks with Templates. A small recap on what my project 
 is about:

 [...]
Yes PLEASE!!!!!!!
May 29
prev sibling next sibling parent Dan Printzell <xwildn00bx gmail.com> writes:
On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
 [...]
I forgot to post the updates after week one due to school and development, so here is my updates for week one and two. Week 1 ====== This week ~~~~~~~~~ - Turned `_d_arrayappendT` and `_d_arrayappendcTX`, with their `*Trace` function into templates. - `_d_arrayappendcd` and `_d_arrayappendcd` where left alone as they are wrapper around the `_d_arrayappendT` for when assigning a dchar to char[], and dchar to wchar[], respectively. - Started working on the dmd code. - Got the lowering code working. - Currently battling with the CTFE. - I started with the templates inside object.d but then moved them to rt/lifetime.d and did a `public import` inside object.d instead. I did this because I started getting template recursions problems when I use `externDFunc`, as it used string concatenations internally. I can probably later move it back to object.d if needed later when I get the CTFE code working again. Next week ~~~~~~~~~ - Continue with the dmd patches and get it be able to compile. - Submit both the druntime and the dmd PR for review. Blockers ~~~~~~~~ In the beginning I tried to translate the hook and all of the functions it called to templates. This just got me into a wormhole of needed changes, and it would have made the PRs harder to review. So I restarted from scratch and instead only translated the entrypoints of each hook, and then internally use `typeid(T)` to pass to the function underneath. Future work when all hooks are translated, could be to translate the functions underneath, when applicable. When I got the hook lowering working I realized how much the hooks have lied about their safety. So I had to add a ` trusted` wrapper that fakes `pure nothrow` for the hook to not break all code. [1] As mentioned above, I'm currently battling with the CTFE to understand my lowered hooks. I found examples of this is done for `__ArrayPostblit` and `__ArrayDtor` inside dinterpret.d, and I will try and understand how they do it and implement the same for my hooks. Week 2 ====== This week ~~~~~~~~~ - I continued to work on the code to a state were I felt I could start to get some feedback, even if the code was not in a working state. - I got the CTFE 'intercept' code working so the hooks are now using the dinterpret.d code instead. - Submitted work-in-progress PRs - druntime: [https://github.com/dlang/druntime/pull/2632] - dmd: [https://github.com/dlang/dmd/pull/9982] Next week ~~~~~~~~~ - I still need to get the PR into a working and mergeable state, I will try and get this done as soon as possible. - I need to implement unittests for the dmd patches to verify that the lowering code works as expected, both for the non-trace and trace versions. - When waiting for reviews I will work on the conversion of _d_array{setctor,ctor,cat*}, as according to my proposal document. - I will try to get a PR of this submitted earlier than what I did the previous ones. This should be possible as these functions are simpler than the last ones and now I know that I should not try and translate dependencies and instead just add `typeid(T)` whenever necessary. Blockers ~~~~~~~~ I've had compilation problem related to the translation of _d_arrayappendcTX. I fixed this by instead calling the TypeInfo version of that function. Due to the exam and school I was not able to finish what I allocated according to my proposal. Currently I just have small bug where I trigger the postblit one too many times with assigning a single element `arr ~= structVar`. This is because the old backend code assigned the element outside of the _d_arrayappend call, verses me now doing it internally. I probably just need to need to make sure that it always lowerest the `structVar` into a ref-able form, or make it assign the variable outside the hook again. Footnotes ====== [1] [https://github.com/Vild/druntime/blob/8f72a8b25f02653445e1b8288583788711b87595/src/rt/lifetime.d#L2871-L2876]
Jun 11
prev sibling next sibling parent reply Dan Printzell <xwildn00bx gmail.com> writes:
On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
 [...]
Week 3 ====== This week ~~~~~~~~~ - Debugged the phobos unittest problems for the _d_arrayappendcT PRs. - The problem is probably that the order the postblit and the assignments are swapped compared to the current hook implementation. The assignment and the postblit probably need to be moved to the outside of the to _d_arrayappendcT function. - Ported over _d_arrayctor, _d_arraysetctor, _d_arraycatT, and _d_arraycatnTX to templates. - Implemented lowering code for _d_arrayctor, _d_arraysetctor. - The whole functions for these could be moved to a template function, so these no longer depend on TypeInfo. - Worked on the lowering code for _d_arraycatT and _d_arraycatnTX. - Debating if _d_arraycatT should be dropped and only use _d_arraycatnTX, to help with the lowering code. Next week ~~~~~~~~~ - Finish the _d_arraycat{T,nTX} hook. - Probably merge _d_arraycatT into _d_arraycatnTX to help the lowering code. - Make and submit the PRs for _d_array*ctor and _d_arraycat{T,nTX}. - Move the assignment of _d_arrayappendcT back outside the hook to fix the problem. Blockers ~~~~~~~~ As mentioned in /This week/ about _d_arrayappendcT, the problem is that phobos unittests passes, even if the unittest works for druntime. The only thing I can think of is that the postblit is called before the object is sent to _d_arrayappendcT instead of after the assignment. Moving the assignment back outside of _d_arrayappendcTX should fix this, to restore what is currently done with the hook. I have not focused too much time on this issue because I wanted to get the _d_array*ctor and _d_arraycat* up and running first as I did not know what problem I would encounter with these. -- The lowering code for _d_arraycat{T,nTX} is currently pretty ugly because I need to handle many cases of the lowering. For example: ```d S[2] a; S[] a4 = a ~ a ~ a ~ a; // Step 1: S[] a4 = _d_arraycatT(a, a) ~ a ~ a; // Step 2: S[] a4 = (S[][3] __cat2 = [a, a, a];) , _d_arraycatnTX(cast(S[][])__cat2) ~ a; // Step 3: S[] a4 = (S[][4] __cat4 = [a, a, a, a];) , _d_arraycatnTX(cast(S[][])__cat4); ``` (This will '__cat*' is used to not allocate memory heap and break the 'scope' requirement code.) The rewrites steps are like this because it works from the inside outwards compared to the outside inwards, like what is done in e2ir.d. As mention in /Next week/ I will merge _d_arraycatT into nTX to help with all the different edge cases and to get more manageable and reviewable code. Week 4 ====== This week ~~~~~~~~~ - Worked on more _d_arrayappend{T,cTX}, _d_arraycatnTX and _d_array{set,}ctor. - I got _d_arrayappend{T,cTX} working after moving the assignment back outside of the _d_arrayappendcTX call. dmd PR: [https://github.com/dlang/dmd/pull/9982] druntime PR: [https://github.com/dlang/druntime/pull/2632] - I've submitted PR for _d_arraycatnTX, even if the dmd lowering code is not working right now. dmd PR: [https://github.com/dlang/dmd/pull/10064] druntime PR: [https://github.com/dlang/druntime/pull/2648] Next week ~~~~~~~~~ Rework my timetable to see if I will be able to finish the rest of the hook for the allocated timeslots. The rework is needed because _d_arraycat{T,nTX} and _d_array{set,}ctor are still not done due to bugs that are hard to understand and fix. The goals is to with my acquired knowledge of what I've done now be able to better classify the difficulty level of each hook. As I've really realized that the line-of-code metric was a really bad one that did not at-all show the difficulty level of the hooks. Continue the work on _d_arraycat{T,nTX} and _d_array{set,}ctor to make sure that they are finish before the second evaluation. I will probably have to put these temporary on the backlog while figuring out how hard the next hook will be to port, which will probably be _d_arraysetlength{,i}T. Blockers ~~~~~~~~ As mentioned before I had postblit problems with the previous _d_arrayappendcT function, but now when I moved the assignment outside it passes all the unittests. _d_arraycatnTX is currently not working because my lowering code somehow send in a temporary incorrectly into a hook, and it also break a assert inside the dmd backend code. More information about this can be found inside the dmd PR. The last hooks that isn't done is the _d_array{set,}ctor, here I still have left to finish the logic for when to write AssignExp expressions, as some of the expression should not be rewritten. I also need to add logic to the hook for when to export the interface as nothrow. If I force it to be nothrow this unittest will fail: [https://github.com/dlang/druntime/blob/9f44ee59ac784e28c19631dd0be2b868766f4348/src/object.d#L3538] But if I don't force it, it will always default to throw and it will break code that calls this hook from nothrow scopes. So I need to implement some logic that determine when to export as nothrow and when to export it as throw. Reflections on progress so far ============================== Now that the first evaluation is in progress I wanted to just write about what my thoughts are after working on three hooks during these four weeks. I added this section to add some transparency of what is happening behind the scenes of my GSoC work, and to document everything in case it could be good to know. I also enjoy being critical about what I have done to get a teachable moment out of it. Planning ~~~~~~~~ The metric I used in my proposal, lines of code, was a really bad choice. I've realized that it is not the amount of code that the hook is called that describes its difficulty but than in what context it is called and with that data it is called with. I have started to going through the rest of the hooks I have not started with to try and build up new metrics that I can use to better evaluate the difficulty of the hook and to make sure I allocate enough time to be able to success with the translation within the allocated timeslots. Because I did not fully realize the difficulty, i.e. possible blockers, of the hooks so far I have not been able to finish all of them within the timeslot I wanted. I got the array append hook working, but it was in the last minute before the week four ended. With the knowledge of everything I did not previously know, did not plan for, or even maybe overlooks, I want to be able to in the future of GSoC identify these problem and know what to do to fix them.
Jun 25
parent Piotrek <dummy dummy.gov> writes:
On Wednesday, 26 June 2019 at 01:49:37 UTC, Dan Printzell wrote:
 Reflections on progress so far
 ==============================

 Now that the first evaluation is in progress I wanted to just 
 write
 about what my thoughts are after working on three hooks during 
 these
 four weeks.
Thank you for sharing this. I find this project very interesting. IMO, all you wrote is valuable and gives some tips for the future. Cheers, Piotrek
Jun 26
prev sibling next sibling parent reply Dan Printzell <xwildn00bx gmail.com> writes:
On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
 [...]
I forgot to post week 5 & 6, so here is week 5-7 Week 5 ====== This week ~~~~~~~~~ - Translate _d_arraysetlengthT to template - _d_arraysetlengthT druntime patch got merged - [https://github.com/dlang/druntime/pull/2656] - Got _d_arraycatnTX to pass all the testcases. - Updated all the PRs based on feedback. Current PRs: _d_arraysetlengthT: - [https://github.com/dlang/dmd/pull/10106] _d_array{,set}ctor: - [https://github.com/dlang/druntime/pull/2655] - [https://github.com/dlang/dmd/pull/10102] _d_arraycatnTX: - [https://github.com/dlang/druntime/pull/2648] - [https://github.com/dlang/dmd/pull/10064] _d_arrayappend{T,cTx}: - [https://github.com/dlang/druntime/pull/2632] - [https://github.com/dlang/dmd/pull/9982] Next week ~~~~~~~~~ - Work on the feedback I get on the dmd patches, when the druntime patch have been merged. - Start working on the _d_arrayliteralTX hook, which was planned for next week. Blockers ~~~~~~~~ Catching `Throwable` and then rethrowing makes the function not be able to be deduced as `nothrow`. Realized that there is no way to easily check if something is `nothrow` using only druntime. Except for `__traits(getFunctionAttributes, x)`, but that requires a lot of code just to check for nothrow. Week 6 ====== This week ~~~~~~~~~ Made two fix PR for `_d_arraysetlengthT*` to disallow the hooks to be inlined, and to force it to generate a *Trace instant for each regular hook. Both of these PRs have been merged. Updated `_d_array{,set}ctor` based on feedback from thewilsonator. [https://github.com/dlang/druntime/pull/2655#issuecomment-507923713] `_d_arraycatnTX` is updated with the sane fixes as `_d_arraysetlengthT`. [https://github.com/dlang/druntime/pull/2648] Next week ~~~~~~~~~ - Porting over on `_d_arrayliteralTX`. - Apply fixes to `_d_arrayappendcTX`. - Work on getting the previous PRs merged. Blockers ~~~~~~~~ Learned that I need `pragma(inline, false);` for the hooks to not break my CTFE interception, and the bigger thing I had to fight with was getting the compiler to instantiate both `_d_arraysetlengthT` and `_d_arraysetlengthTTrace`. Here I tested multiple thing, like referencing the trace function from the regular hook, but this did not work and only caused template instantiating problems. What I used in the end, which was also the most simple and clean solution, was to wrap both hooks inside of a `template _d_arraysetlengthTImpl` block. This caused both functions to be instantiated, even if only one hook was referenced. All of this was needed because now if you, for example, use `std.array.Appender!(char[])` which is a template struct, in a project when you compile with the `-profile=gc`. Means that the template gets instantiated but because that template instantiating already exist inside of phobos, dmd chooses to use that one instance. But because that phobos instance didn't use `_d_arraysetlengthTTrace`, it cause a `unknown reference to`. I'm not sure if I explain the problem here correctly, but it was weird and hard to understand enough to be able to make a fix for it. _d_arraysetlength's dmd PR is currently failing the CI during the `profilegc`, but I cannot reproduce it locally. Week 7 ====== This week ~~~~~~~~~ Continued to work on my PR to make them into a mergeable state. 'Cleanup and fix implementation of _d_array{setlengthT,catnTX}Trace' - [https://github.com/dlang/druntime/pull/2673] Fixes some problem with the Trace entrypoint for hooks not calling `rt.profilegc.accumulate`, and a refactoring to templateize the Trace function to make sure that code isn't duplicated over all the Trace entrypoints. Finished the dmd PR for _d_arraysetlengthT using all the fixes I've been working on the last week. It currently waits for someone to do the last review and hopefully merge it. [https://github.com/dlang/dmd/pull/10106] Next week ~~~~~~~~~ Pretty much same as last week: - Finish port of `_d_arrayliteralTX` - Work on getting the previous PRs merged. - The pr for _d_arrayappend* needs to be rebased and integrated with the new way to implement Trace function. [https://github.com/dlang/druntime/pull/2632] Its dmd patches also need to be turned into a PR. - [https://github.com/dlang/dmd/pull/10102] - _d_array{,set}ctor and [https://github.com/dlang/dmd/pull/10064] - _d_arraycatnTX needs to be updated to follow how _d_arraysetlengthT was implemented. Blockers ~~~~~~~~ The previous week I've learned more about every edge-case that I've encountered, and how to fix them. Like the how I fixed and cleaned up the *Trace implementations, here I had to figured out how to do a `pure` bypass, while still not allowing the compiler to remove calls that are requested, but the optimizer thinks are useless. I solved this by `if (...) assert(0);`, with a condition that I know will never pass, and the optimizer will never remove a `assert(0)`, which forced it to allow the call to still be done. During the creation I found a way inside of the dmd's source-code on how to bypass pure, without using any `cast()`s. Sadly this was classified by github user aG0aep6G as a bug. Personally I'm a bit sad that this bypass method was removed in [https://github.com/dlang/dmd/pull/10172], as I really though it looked cleaner than using casts, but as this is a bug I cannot argue against removing it. On the note about bypassing pure I've mentioned that I will probably introduce `HookCallExp` or a bool inside of `CallExp` to allow purity to be ignore, but only for hook calls. If/when I do this I also want to add support to be able to detect hooks that have been inlined, maybe by adding a `HookCompoundStatement` and/or `HookCommaExp`. But currently I want to spend my time to get all the hooks into dmd and druntime, and probably after GSoC start introducing these improvements. Another problem I encountered after the `HookTraceImpl` PR was merged was that I broke the CTFE intercept code I had for *Trace hooks. This happened because the following code does not introduce a new symbol into the context and it is just alias: ,---- | alias _d_arraysetlengthTTrace = HookTraceImpl!(Tarr, _d_arraysetlengthT, errorMessage); `---- This is something I overlooked when doing the PR, but in the end this was not hard to big and I would say actually lead to better code. As now I have a `removeHookTraceImpl` function inside dinterpret.d: [https://github.com/dlang/dmd/pull/10106/files#diff-264b29261e215213d3026cd44003c2ebR7452] ,---- | //This function will replace: | HookTraceImpl!(T, Hook, errMsg).HookTraceImpl(..., parameters) | // with | Hook(parameters) `---- It does this by getting getting the `FuncDeclaration` which in this case is `HookTraceImpl` function, goes to its parent the `HookTraceImpl` template, takes the second template parameters, and uses to rewrite the `CallExp` and replace the `FuncDeclaration` that is used to do the CTFE interception.
Jul 17
parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Wednesday, 17 July 2019 at 17:04:45 UTC, Dan Printzell wrote:
 It does this by getting getting the `FuncDeclaration` which in 
 this case
 is `HookTraceImpl` function, goes to its parent the 
 `HookTraceImpl`
 template, takes the second template parameters, and uses to 
 rewrite the
 `CallExp` and replace the `FuncDeclaration` that is used to do 
 the CTFE
 interception.
Out of curiosity, what is CTFE interception?
Jul 17
parent Dan Printzell <xwildn00bx gmail.com> writes:
On Wednesday, 17 July 2019 at 20:47:02 UTC, Olivier FAURE wrote:
 On Wednesday, 17 July 2019 at 17:04:45 UTC, Dan Printzell wrote:
 It does this by getting getting the `FuncDeclaration` which in 
 this case
 is `HookTraceImpl` function, goes to its parent the 
 `HookTraceImpl`
 template, takes the second template parameters, and uses to 
 rewrite the
 `CallExp` and replace the `FuncDeclaration` that is used to do 
 the CTFE
 interception.
Out of curiosity, what is CTFE interception?
Basically I rewrite the expression `arr.length = len;` into a function call to the templated runtime function `_d_arraysetlengthT(arr, len);`. But because of this the CTFE can no longer recognize that the code is trying to change the size of `arr`, only that it is calling a function called `_d_arraysetlengthT`. The CTFE intercept is basically intercepting all the function call that the CTFE would run and checks if it is trying to call the `_d_arraysetlengthT`, and if it is it will rewrite the expression back to `arr.length = len` and then interpret that expression instead of the function call. This is needed as the CTFE way of resizing arrays is not the same as how regular code resizes arrays.
Jul 17
prev sibling next sibling parent Dan Printzell <xwildn00bx gmail.com> writes:
On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
 [...]
Week 30 ======= This week ~~~~~~~~~ The translation of _d_arraysetlengthT is fully merged! Worked on getting the _d_arrayappend{T,cTX} and _d_array{,set}ctor dmd PRs into a mergeable state, currently stuck at getting it to pass the testcases. More information in Blockers. I never finished the _d_arrayliteralTX PRs as I wanted to get the other PR working first, but I never thought they would take this long time to fix. Next week ~~~~~~~~~ Pretty much same as last week: - Finish port of `_d_arrayliteralTX` - Start working on the `_d_delarray_t*`, `_d_arrayassign*`, and `_d_arraysetassign` translations. - Work on getting the previous PRs merged. - The PR for _d_arrayappendcTX need to more logic on how to lower the appending, and more logic on how to interpret the lowered expressions. - _d_array{,set}ctor need to implements some fixes to get the CTFE to work correctly. - _d_arraycatnTX need fixes related to the CTFE interpretion. Also need to fix the code based on the received feedback. Blockers ~~~~~~~~ This passed week I've been fighting with the optimizer to always be able to detect the lowered code inside dinterpret. With _d_arrayappendcTX I've (so far?) identified four different expression groups that I need be able to detect and rewrite back so the CTFE interpretion works correctly. They are: ``` //Case 1: left is the _d_arrayappendcTX call, right is the assignment onto the array. trs ~= fork(t, next, t.counter); // => _d_arrayappendcTX(trs, 1LU) , trs[__dollar - 1LU] = fork(t, next, t.counter); //Case 2: left is the _d_arrayappendcTX call, right is the construction onto the array. arr ~= T(nanF, '\xff').this(0); // => _d_arrayappendcTX(arr, 1LU) , (arr[__dollar - 1LU] = T , arr[__dollar - 1LU].this(0)); //Case 3: left is the _d_arrayappendcTX call, right is a postblit and the assignment onto the array. this.data ~= ((T __copytmp = (__copytmp = val).__fieldPostblit();) , __copytmp); // => _d_arrayappendcTX(this.data, 1LU) , ((T __copytmp = (__copytmp = val).__fieldPostblit();) , this.data[__dollar - 1LU] = __copytmp); //Case 4: left is __appendtmp var, right is case 1 op ~= cast(immutable(char))this.x; // => (immutable ref immutable(char) __appendtmp115 = this.x;) , (_d_arrayappendcTX(op, 1LU) , op[__dollar - 1LU] = __appendtmp115); ``` Related to the _d_array{,set}ctor PR, it currently fails at object.d line 3636, 'assert(postblitRecurseOrder =' order);=. This is because order is equal to: ``` copy outer ``` Verses what is expected: ``` copy outer destroy outer ``` It has something to do with optimizations, and probably exceptions. I have not been able to find the cause of it yet. But what happens is that it never calls the destructor for 'arrCopy[0]'. Related to _d_arraycatnTX it currently broken two place so far which I need to find a real fix to before it is done. The first one is that implicit cast to 'const(char)*' does not work anymore of a string concatenations. The second problem I've found is that: `void badOp(int x, int y = 1 ~ "string") {}` prints the error: `Error: cannot implicitly convert expression ['\x01', 's', 't', 'r', 'i', 'n', 'g'] of type string to int` Instead of: `Error: cannot implicitly convert expression "\x01string" of type string to int` I'm not sure why it prints the string as a 'char[]', but it needs to be fixed before it can be merged. Thoughts after the second milestone =================================== From my point of view I would classify this milestone as less productive compared to the first one, but from the sense that I got a hook fully into dmd and druntime you could classify that this was more successful milestone. I need to figure out what is most important to get done for the final milestone and prioritize that. I don't think I will be able to finish everything 100% as the dmd PR are getting more difficult for optimization, and each edge-case. But I do think I will be able to finish all the druntime PRs.
Jul 23
prev sibling parent Dan Printzell <xwildn00bx gmail.com> writes:
On Tuesday, 28 May 2019 at 10:47:03 UTC, Dan Printzell wrote:
 [...]
Sorry, I've been busy with getting everything into a workable to state so I've been late to with posting these updates. So this time it is for the last three weeks. There is also some planning for the final submission at the bottom. Week 31 ======= This week ~~~~~~~~~ Last week I've been trying a lot to use the `__ctfe` variable to detect when to call the rewritten hook, verses when to call the original expression. As discussed, the old CTFE interception turned into a big edge-case friendly place where it needs to be able to detect all the different ways the AST could look when executing a hook. Sadly lowering `this.data ~= val` to ` __ctfe ? this.data ~= val : _d_arrayappendT(this.data, val)` didn't work as it threw the following assertion in the backend. `dmd: dmd/backend/symbol.d:1170: Assertion '(*s).Ssymnum == -1' failed.`. I have not really made any progress on the next hooks or the previous hooks as I wanted to get the DMD part done first. It is in my opinion pretty useless to add translated hooks into druntime if their dmd part won't work. As adding unused code is still adding maintenance cost to druntime. I've marked all the current dmd PR as WIP, as they should all move to a better way of detecting CTFE. I've also update [https://definewild.se/public/Hooks.html] to reflect the current state of all the hooks. Next week ~~~~~~~~~ - Try and figure how to use CondExp, if it is even possible. - Maybe rethink my time plan. Should I focus on getting more druntime hook merged, or should I focus on getting the current merged hooks working? (The later is probably the better choice.) Same as last week: - Finish port of `_d_arrayliteralTX` - Start working on the `_d_delarray_t*`, `_d_arrayassign*`, and `_d_arraysetassign` translations. - Work on getting the previous PRs merged. - The PR for _d_arrayappendcTX need to more logic on how to lower the appending, and more logic on how to interpret the lowered expressions. - _d_array{,set}ctor need to implements some fixes to get the CTFE to work correctly. - _d_arraycatnTX need fixes related to the CTFE interpretion. Also need to fix the code based on the received feedback. Blockers ~~~~~~~~ The assert problem seem to have something to do with the `val` part getting rewritten. Maybe dmd doesn't support DeclarationExp inside of CondExp? ``` e2ir.d, visit(CondExp de): __ctfe ? this.data ~= ((InversionList!(GcPolicy) __copytmp4144 = (__copytmp4144 = val).__fieldPostblit();) , __copytmp4144) : ((InversionList!(GcPolicy) __appendtmp4145 = (InversionList!(GcPolicy) __copytmp4144 = (__copytmp4144 = val).__fieldPostblit();) , __appendtmp4145 = __copytmp4144;) , (_d_arrayappendcTX(this.data, 1LU) , this.data[__dollar - 1LU] = __appendtmp4145) , this.data) e2ir.d, visit(CommaExp ce): (InversionList!(GcPolicy) __copytmp4106 = (__copytmp4106 = val).__fieldPostblit();) , this.data[__dollar - 1LU] = __copytmp4106 e2ir.d, visit(DeclarationExp de): (InversionList!(GcPolicy) __copytmp4106 = (__copytmp4106 = val).__fieldPostblit();) dmd: dmd/backend/symbol.d:1170: Assertion `(*s).Ssymnum == -1' failed. ``` Week 32 ======= This week ~~~~~~~~~ *_d_arrayappend{T,cTX}:* Required druntime fixes: [https://github.com/dlang/druntime/pull/2718] Update dmd patches: [https://github.com/dlang/dmd/pull/9982] This uses the new `__ctfe ? orgExp : loweredExp` technique instead of intercepting CTFE calls. I also added the fix to the assert [https://github.com/dlang/dmd/pull/9982/commits/2d81b673424175ae5304f2eee7dcb7bd87d43cc3] And I made the backend only compile the `!__ctfe` path to machine code, to be able to `assert(0)` the calls to the old hook functions: [https://github.com/dlang/dmd/pull/9982/commits/1f4d9bf17033cee8e2458c3d1c35a52632caa6f8] It took some fighting to get the hook working, because I triggered some asserts in the codegen. But I was able to fix / work around all of them. Next week ~~~~~~~~~ - Update `_d_arrayappend{T,cTX}` PRs based on feedback. - Fix `_d_array{,set}ctor`. - Fix `_d_arraycatnTX`. - Figure out what is needed for the final submission. Blockers ~~~~~~~~ The current problem with `_d_array{,set}ctor` is that it doesn't call all the dtors. It expects to find this in the output `order`: ``` copy outer destroy outer ``` But only this is found: ``` copy outer ``` Unittest: [https://github.com/dlang/druntime/blob/master/src/object.d#L3647] I think the problem is that DeclarationExp/VarDeclaration, doesn't handle `scope (failure)` correctly. Currently I'm only rewriting the AssignExp that is inside the `VarDeclaration`. My current theory is that if I rewrite the DeclarationExp instead the hook may start working against. I have not look into `_d_arraycatnTX` issues this week. But using the new `__ctfe ? :` way of restoring the CTFE functionality should hopefully allow me to remove some hacks I had to implement. Which in turn will allow the PR to not break any existing code. Week 33 ======= This week ~~~~~~~~~ *_d_arrayappend{T,cTX}:* Remove the debug `message()` calls. *_d_array{,set}ctor:* Fixed the hook by doing a weird hack I don't fully understand why it is needed. Not sure if it is a dmd bug that could happen in usercode or if it is just because it is called inside of a `DeclarationExp._init`. [https://github.com/Vild/druntime/blob/FixArrayCtorHook/src/core/internal/array/construction.d#L59] I have not PR this yet as I'm currently depending on the PR I submitted last week related to _d_arrayappend, [https://github.com/dlang/druntime/pull/2718]. I may merge these changes into that PR. *_d_arraycatnTX:* I've figured out the currently problem with why cannot compile this: [https://github.com/dlang/dmd/blob/b2522da8566783491648bc104a29b42dc2dc569e/src/dmd/mars.d#L2174] It is because the somewhere the `commited` member of StringExp get set to `1` even if the CTFE was able to run the call at compile and get a single StringExp out of it. Next week ~~~~~~~~~ - Fix the last changed to `_d_arraycatnTX`. - Get `_d_arrayappend{T,cTX}`, `_d_array{,set}ctor`, and `_d_arraycatnTX`. - Work on the writeup on what I've learned etc, that I lifted last week update. Blockers ~~~~~~~~ *_d_array{,set}ctor:* Here is just some of the differences I found in GHIDRA when I compared compile with and without the hack. The differences I could find of the hook is that the working one have two try-blocks ([https://i.definewild.se/3377]) and the loop looks a bit different ([https://i.definewild.se/165e] left is non-working, right is working). *_d_arraycatnTX:* The biggest blocker right now is how I should solve the `_d_arraycatnTX` problem. But I don't think it will be too bad, as now I've been able to identify where the problem is. Now I just need to compare what code paths the different concatenations goes though, and why they differ. ``` enum fourfivesix = "456"; enum world = () { string s; s ~= "world!"; return s; }(); void test(const(char)* fmt) {} void main() { test("derp" ~ " herp"); // Works test("123 " ~ fourfivesix); // Works test("hello " ~ world); // Doesn't /* Error: function cattesting.test(const(char)* fmt) is not callable using argument types (string) * cannot pass argument "hello world!" of type string to parameter const(char)* fmt */ } ``` Planning for final submission ============================= I will not be able to finish the hooks all the hooks I listed in my proposal. Instead I should probably just focus on finishing the current WIP hook: `_d_arrayappend{T,cTX}`, `_d_array{,set}ctor`, and `_d_arraycatnTX`. If I finish these I will have successed with translating four out of nine hooks (the forth hook is _d_arraysetlength which is already merged). Furthermore as a final submission it probably should include some documentation on how to do the translation process, to help aid others to translate hooks. This could include the following sections/topics: - Finding a hook to translate - Where to add the new code, how to update the buildsystem - How to manage the CTFE. - Navigating the codebase: Which dmd files are relevant? Where should I look for class X? - Examples of previous translations - with notes on how they were updated / fixed - Debugging technique - How to implement a -profile=gc entrypoint. I'm not sure where the best place for this would be as this is more for new compile developers than the average D user. I guess the best place would be the wiki.
Aug 14