www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Request for Review: DI Generation Improvements

reply "Adam Wilson" <flyboynw gmail.com> writes:
I am formally requesting review of the two pull requests that make up my
DI Generation Improvements patch.

The first pull is against the DRuntime and modifies the makefiles such
some files in the DRT are not generated as DI files but copied from the D
source files directly into the DI file. This solves a problem where
certain portions of Phobos the CTFE into the DRT cause a build failure
when using the new DI generation. The files that are copied instead of
generated are all of core and core.stdc, EXCEPT core.atomic, core.cpuid,
core.demangle, and core.thread.

Pull DRuntime/210:
https://github.com/D-Programming-Language/druntime/pull/210

The second pull is against DMD itself and consists of the actual changes
to the current DI generation. The improvements consist of removing all
function implementations, EXCEPT for those of template and auto-return
functions, and improved indenting. These changes represent the closest I
can come to agreement from the community as to the proper contents of a DI
file. This patch fixes the biggest problems of DI generation and provides
a baseline for future improvements to DI generation. Currently being
discussed are modifications to include a limited form of semantic analysis
of the source code during DI generation to improve the usefulness of the
output as well as the addition of an attribute to direct DI generation to
include a function implementation for CTFE or inlining purposes.

Pull DMD/945: https://github.com/D-Programming-Language/dmd/pull/945

I welcome all comments on these pulls and it is my goal to see them merged
into DMD 2.060.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 14 2012
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 15-05-2012 01:50, Adam Wilson wrote:

 I am formally requesting review of the two pull requests that make up my
 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the makefiles such
 some files in the DRT are not generated as DI files but copied from the D
 source files directly into the DI file. This solves a problem where
 certain portions of Phobos the CTFE into the DRT cause a build failure
 when using the new DI generation. The files that are copied instead of
 generated are all of core and core.stdc, EXCEPT core.atomic, core.cpuid,
 core.demangle, and core.thread.

 Pull DRuntime/210:
 https://github.com/D-Programming-Language/druntime/pull/210

 The second pull is against DMD itself and consists of the actual changes
 to the current DI generation. The improvements consist of removing all
 function implementations, EXCEPT for those of template and auto-return
 functions, and improved indenting. These changes represent the closest I
 can come to agreement from the community as to the proper contents of a DI
 file. This patch fixes the biggest problems of DI generation and provides
 a baseline for future improvements to DI generation. Currently being
 discussed are modifications to include a limited form of semantic analysis
 of the source code during DI generation to improve the usefulness of the
 output as well as the addition of an attribute to direct DI generation to
 include a function implementation for CTFE or inlining purposes.

 Pull DMD/945: https://github.com/D-Programming-Language/dmd/pull/945

 I welcome all comments on these pulls and it is my goal to see them merged
 into DMD 2.060.


I would suggest excluding core.[atomic,cpuid,demangle] from DI 
generation, too, and just copy them. I don't see any reason to do DI 
generation for them.

-- 
- Alex
May 14 2012
next sibling parent reply "Tyro[17]" <nospam home.com> writes:
On Tuesday, 15 May 2012 at 00:02:23 UTC, Alex Rønne Petersen 
wrote:

 On 15-05-2012 01:50, Adam Wilson wrote:

 I am formally requesting review of the two pull requests that 
 make up my
 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the 
 makefiles such
 some files in the DRT are not generated as DI files but copied 
 from the D
 source files directly into the DI file. This solves a problem 
 where
 certain portions of Phobos the CTFE into the DRT cause a build 
 failure
 when using the new DI generation. The files that are copied 
 instead of
 generated are all of core and core.stdc, EXCEPT core.atomic, 
 core.cpuid,
 core.demangle, and core.thread.




[snip]

 I would suggest excluding core.[atomic,cpuid,demangle] from DI 
 generation, too, and just copy them. I don't see any reason to 
 do DI generation for them.


Reread the last line of the paragraph starting with "The first 
pull". Your concern is already addressed.
May 14 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 15-05-2012 02:20, Tyro[17] wrote:

 On Tuesday, 15 May 2012 at 00:02:23 UTC, Alex Rønne Petersen wrote:

 On 15-05-2012 01:50, Adam Wilson wrote:

 I am formally requesting review of the two pull requests that make up my
 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the makefiles such
 some files in the DRT are not generated as DI files but copied from
 the D
 source files directly into the DI file. This solves a problem where
 certain portions of Phobos the CTFE into the DRT cause a build failure
 when using the new DI generation. The files that are copied instead of
 generated are all of core and core.stdc, EXCEPT core.atomic, core.cpuid,
 core.demangle, and core.thread.




 [snip]

 I would suggest excluding core.[atomic,cpuid,demangle] from DI
 generation, too, and just copy them. I don't see any reason to do DI
 generation for them.


 Reread the last line of the paragraph starting with "The first pull".
 Your concern is already addressed.


No, you got it backwards. Before my comment, his pull request still 
caused DI generation to happen for the modules I mentioned, and I 
suggested simply copying them like the rest.

-- 
- Alex
May 14 2012
parent "Tyro[17]" <nospam home.com> writes:
On Tuesday, 15 May 2012 at 00:25:15 UTC, Alex Rønne Petersen
wrote:

 On 15-05-2012 02:20, Tyro[17] wrote:

 On Tuesday, 15 May 2012 at 00:02:23 UTC, Alex Rønne Petersen




[snip]

 I would suggest excluding core.[atomic,cpuid,demangle] from DI
 generation, too, and just copy them. I don't see any reason 
 to do DI
 generation for them.


 Reread the last line of the paragraph starting with "The first 
 pull".
 Your concern is already addressed.


 No, you got it backwards. Before my comment, his pull request 
 still caused DI generation to happen for the modules I 
 mentioned, and I suggested simply copying them like the rest.


I see. Pardon the noise.
May 14 2012
prev sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 14 May 2012 17:02:23 -0700, Alex R=F8nne Petersen
<xtzgzorex gmail.com> wrote:


 On 15-05-2012 01:50, Adam Wilson wrote:

 I am formally requesting review of the two pull requests that make up=




 my

 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the makefiles suc=




h

 some files in the DRT are not generated as DI files but copied from t=




he  =


 D
 source files directly into the DI file. This solves a problem where
 certain portions of Phobos the CTFE into the DRT cause a build failur=




e

 when using the new DI generation. The files that are copied instead o=




f

 generated are all of core and core.stdc, EXCEPT core.atomic, core.cpu=




id,

 core.demangle, and core.thread.

 Pull DRuntime/210:
 https://github.com/D-Programming-Language/druntime/pull/210

 The second pull is against DMD itself and consists of the actual chan=




ges

 to the current DI generation. The improvements consist of removing al=




l

 function implementations, EXCEPT for those of template and auto-retur=




n

 functions, and improved indenting. These changes represent the closes=




t I

 can come to agreement from the community as to the proper contents of=




 a  =


 DI
 file. This patch fixes the biggest problems of DI generation and  =





 provides
 a baseline for future improvements to DI generation. Currently being
 discussed are modifications to include a limited form of semantic  =





 analysis
 of the source code during DI generation to improve the usefulness of =




the

 output as well as the addition of an attribute to direct DI generatio=




n  =


 to
 include a function implementation for CTFE or inlining purposes.

 Pull DMD/945: https://github.com/D-Programming-Language/dmd/pull/945

 I welcome all comments on these pulls and it is my goal to see them  =





 merged
 into DMD 2.060.


 I would suggest excluding core.[atomic,cpuid,demangle] from DI  =



 generation, too, and just copy them. I don't see any reason to do DI  =



 generation for them.


This can be easily changed. Does anyone see any potential problems with
making this change?

-- =

Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 14 2012
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, May 14, 2012 17:36:49 Adam Wilson wrote:

 This can be easily changed. Does anyone see any potential problems with
 making this change?


The only modules in druntime that I am aware of where there would be actual 
problems if their .d files were used instead of .di files would be those which 
already have hand-maintained .di files (core.thread and object are the only two 
that I'm aware of, though there might be others). Beyond that, core.rt 
definitely should be using .di files, but AFAIK, it wouldn't actually hurt 
anything beyond compilation times for them to be .d files, and AFAIK, all of 
the rest of druntime can be .d files without causing any problems. Most of them 
just hold C declarations anyway.

- Jonathan M Davis
May 14 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 15-05-2012 02:59, Jonathan M Davis wrote:

 On Monday, May 14, 2012 17:36:49 Adam Wilson wrote:

 This can be easily changed. Does anyone see any potential problems with
 making this change?


 The only modules in druntime that I am aware of where there would be actual
 problems if their .d files were used instead of .di files would be those which
 already have hand-maintained .di files (core.thread and object are the only two
 that I'm aware of, though there might be others). Beyond that, core.rt
 definitely should be using .di files, but AFAIK, it wouldn't actually hurt
 anything beyond compilation times for them to be .d files, and AFAIK, all of
 the rest of druntime can be .d files without causing any problems. Most of them
 just hold C declarations anyway.

 - Jonathan M Davis


core.rt? Do you mean rt.*? Those are completely and entirely private anyway.

-- 
- Alex
May 14 2012
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, May 15, 2012 03:20:24 Alex R=C3=B8nne Petersen wrote:

 On 15-05-2012 02:59, Jonathan M Davis wrote:

 On Monday, May 14, 2012 17:36:49 Adam Wilson wrote:

 This can be easily changed. Does anyone see any potential problems=






 with

 making this change?


=20
 The only modules in druntime that I am aware of where there would b=




e

 actual
 problems if their .d files were used instead of .di files would be =




those

 which already have hand-maintained .di files (core.thread and objec=




t are

 the only two that I'm aware of, though there might be others). Beyo=




nd

 that, core.rt definitely should be using .di files, but AFAIK, it
 wouldn't actually hurt anything beyond compilation times for them t=




o be

 .d files, and AFAIK, all of the rest of druntime can be .d files wi=




thout

 causing any problems. Most of them just hold C declarations anyway.=





=20
 - Jonathan M Davis


=20
 core.rt? Do you mean rt.*? Those are completely and entirely private =


anyway.

Probably. I didn't look at the source tree before posting. My point was=
 that=20
the rt stuff shouldn't be provided as .d files (however they may be org=
anized),=20
whereas pretty much everything other than core.thread and object should=
 be fine=20
as .d files AFAIK.

- Jonathan M Davis
May 14 2012
prev sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 14 May 2012 16:50:24 -0700, Adam Wilson <flyboynw gmail.com> wrote:


 I am formally requesting review of the two pull requests that make up my
 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the makefiles such
 some files in the DRT are not generated as DI files but copied from the D
 source files directly into the DI file. This solves a problem where
 certain portions of Phobos the CTFE into the DRT cause a build failure
 when using the new DI generation. The files that are copied instead of
 generated are all of core and core.stdc, EXCEPT core.atomic, core.cpuid,
 core.demangle, and core.thread.

 Pull DRuntime/210:
 https://github.com/D-Programming-Language/druntime/pull/210

 The second pull is against DMD itself and consists of the actual changes
 to the current DI generation. The improvements consist of removing all
 function implementations, EXCEPT for those of template and auto-return
 functions, and improved indenting. These changes represent the closest I
 can come to agreement from the community as to the proper contents of a  
 DI
 file. This patch fixes the biggest problems of DI generation and provides
 a baseline for future improvements to DI generation. Currently being
 discussed are modifications to include a limited form of semantic  
 analysis
 of the source code during DI generation to improve the usefulness of the
 output as well as the addition of an attribute to direct DI generation to
 include a function implementation for CTFE or inlining purposes.

 Pull DMD/945: https://github.com/D-Programming-Language/dmd/pull/945

 I welcome all comments on these pulls and it is my goal to see them  
 merged
 into DMD 2.060.


I have updated the make files so that only core.thread and core.sync.* are  
run through DI generation. ALL other core.* modules are copied into the  
import directory now.

Git wrecked my branch on a rebase so I had to recreate the pull.
Pull DRuntime/218:  
https://github.com/D-Programming-Language/druntime/pull/218

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 14 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:

 I have updated the make files so that only core.thread and core.sync.* are
 run through DI generation. ALL other core.* modules are copied into the
 import directory now.


I assume that object.di and core/thread.di are being used rather than their 
corresponding .d files being run through the .di generation? They already have 
hand-crafted .di files.

- Jonathan M Davis
May 14 2012
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 14 May 2012 23:11:50 -0700, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:


 On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:

 I have updated the make files so that only core.thread and core.sync.*  
 are
 run through DI generation. ALL other core.* modules are copied into the
 import directory now.


 I assume that object.di and core/thread.di are being used rather than  
 their
 corresponding .d files being run through the .di generation? They  
 already have
 hand-crafted .di files.

 - Jonathan M Davis


The funny thing is that core.thread.di was being run through the DI  
generator in the old make file. I had left it that way, I have posted a  
commit that moves it to the copy section.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 14 2012
next sibling parent reply kenji hara <k.hara.pg gmail.com> writes:
Old days import/core/thread.di was generated from src/core/thread.d .
Current import/core/thread.di is generated from src/core/thread.*di* .

Kenji Hara

2012/5/15 Jonathan M Davis <jmdavisProg gmx.com>:

 On Monday, May 14, 2012 23:46:08 Adam Wilson wrote:

 On Mon, 14 May 2012 23:11:50 -0700, Jonathan M Davis <jmdavisProg gmx.com>

 wrote:

 On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:

 I have updated the make files so that only core.thread and core.sync.*
 are
 run through DI generation. ALL other core.* modules are copied into the
 import directory now.


 I assume that object.di and core/thread.di are being used rather than
 their
 corresponding .d files being run through the .di generation? They
 already have
 hand-crafted .di files.

 - Jonathan M Davis


 The funny thing is that core.thread.di was being run through the DI
 generator in the old make file. I had left it that way, I have posted a
 commit that moves it to the copy section.


 Well, if that works... It's weird though, because there's no reason to have a
 thread.di if it's not used, and as I understand it, some work was being done
 that required some of Thread's internals _not_ be in the .di file, but I don't
 remember the details, and I don't know where that stands.

 - Jonathan M Davis
May 15 2012
next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 15 May 2012 02:51:44 -0700, kenji hara <k.hara.pg gmail.com> wrote:


 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .

 Kenji Hara


Well, I've updated the make file to just copy thread.di instead of running  
it through the generator.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 15 2012
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com> wrote:


 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .


Huh?  Why the copy?  Just move src/core/thread.di to import/core/thread.di

object.di lives in import/core, I think it should be the same for all the  
hand-maintained .di files.

FWIW, I thought thread.di was being generated because of this.

Also, I agree that thread and object are the only modules that need to be  
.di files.  Everything else is already opaque for the most part, and the  
pieces that aren't are just supporting code that can be visible.

What we need to protect is the runtime implementation, so projects cannot  
depend on private APIs that may change.

-Steve
May 16 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 16-05-2012 14:46, Steven Schveighoffer wrote:

 On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com> wrote:


 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .


 Huh? Why the copy? Just move src/core/thread.di to import/core/thread.di

 object.di lives in import/core, I think it should be the same for all
 the hand-maintained .di files.


Good point. I wonder why we didn't do that initially.


 FWIW, I thought thread.di was being generated because of this.

 Also, I agree that thread and object are the only modules that need to
 be .di files. Everything else is already opaque for the most part, and
 the pieces that aren't are just supporting code that can be visible.

 What we need to protect is the runtime implementation, so projects
 cannot depend on private APIs that may change.

 -Steve


-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org
May 16 2012
prev sibling next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 16 May 2012 05:46:44 -0700, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com>  
 wrote:


 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .


 Huh?  Why the copy?  Just move src/core/thread.di to  
 import/core/thread.di

 object.di lives in import/core, I think it should be the same for all  
 the hand-maintained .di files.


Actually, object.di lives /import. Moving thread.di to /import/core would  
require a significant change to the clean targets of both makefiles since  
they just rm the whole /import/core directory at the moment. It's not  
impossible and if you guys agree that the change should be made I'd be  
happy to make it happen.


 FWIW, I thought thread.di was being generated because of this.

 Also, I agree that thread and object are the only modules that need to  
 be .di files.  Everything else is already opaque for the most part, and  
 the pieces that aren't are just supporting code that can be visible.

 What we need to protect is the runtime implementation, so projects  
 cannot depend on private APIs that may change.

 -Steve


The biggest problem right now is that, while we all agree that these  
changes need to happen, getting them merged appears to be nigh impossible.  
There appears to be a bottleneck in the process caused by the lack of  
capable persons to verify and execute the merges, particularly for  
druntime. There are 11 passing merges for the druntime at last count, out  
of 28 total. Alexrp and dawgfoto particularly seem to have a lot of pulls  
that are just languishing.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 16 2012
next sibling parent =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 16-05-2012 19:00, Adam Wilson wrote:

 On Wed, 16 May 2012 05:46:44 -0700, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:


 On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com>
 wrote:


 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .


 Huh? Why the copy? Just move src/core/thread.di to import/core/thread.di

 object.di lives in import/core, I think it should be the same for all
 the hand-maintained .di files.


 Actually, object.di lives /import. Moving thread.di to /import/core
 would require a significant change to the clean targets of both
 makefiles since they just rm the whole /import/core directory at the
 moment. It's not impossible and if you guys agree that the change should
 be made I'd be happy to make it happen.


I think it's a good idea.


 FWIW, I thought thread.di was being generated because of this.

 Also, I agree that thread and object are the only modules that need to
 be .di files. Everything else is already opaque for the most part, and
 the pieces that aren't are just supporting code that can be visible.

 What we need to protect is the runtime implementation, so projects
 cannot depend on private APIs that may change.

 -Steve


 The biggest problem right now is that, while we all agree that these
 changes need to happen, getting them merged appears to be nigh
 impossible. There appears to be a bottleneck in the process caused by
 the lack of capable persons to verify and execute the merges,
 particularly for druntime. There are 11 passing merges for the druntime
 at last count, out of 28 total. Alexrp and dawgfoto particularly seem to
 have a lot of pulls that are just languishing.


-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org
May 16 2012
prev sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, May 16, 2012 10:00:24 Adam Wilson wrote:

 The biggest problem right now is that, while we all agree that these
 changes need to happen, getting them merged appears to be nigh impossible.
 There appears to be a bottleneck in the process caused by the lack of
 capable persons to verify and execute the merges, particularly for
 druntime. There are 11 passing merges for the druntime at last count, out
 of 28 total. Alexrp and dawgfoto particularly seem to have a lot of pulls
 that are just languishing.


That's just life with druntime, Phobos, and dmd. There are only so many people 
with commit access, and they're all busy. druntime is a bit worse than Phobos, 
since it often requires Sean to look at stuff, but they both have the problem. 
And in this particular case, since it impacts so much, it's one of the 
requests that Sean probabyl should look at, and I don't know how often he goes 
through druntime's pull requests.

- Jonathan M Davis
May 16 2012
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 16 May 2012 14:37:46 -0700, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:


 On Wednesday, May 16, 2012 10:00:24 Adam Wilson wrote:

 The biggest problem right now is that, while we all agree that these
 changes need to happen, getting them merged appears to be nigh  
 impossible.
 There appears to be a bottleneck in the process caused by the lack of
 capable persons to verify and execute the merges, particularly for
 druntime. There are 11 passing merges for the druntime at last count,  
 out
 of 28 total. Alexrp and dawgfoto particularly seem to have a lot of  
 pulls
 that are just languishing.


 That's just life with druntime, Phobos, and dmd. There are only so many  
 people
 with commit access, and they're all busy. druntime is a bit worse than  
 Phobos,
 since it often requires Sean to look at stuff, but they both have the  
 problem.
 And in this particular case, since it impacts so much, it's one of the
 requests that Sean probabyl should look at, and I don't know how often  
 he goes
 through druntime's pull requests.

 - Jonathan M Davis


I absolutely agree that he should look at it. I was just kvetching about  
the backlog in general. There are some really good fixes that aren't  
getting merged. Realistically, I no longer expect for the full DI patch to  
make it in to 2.060; but at the same time I don't want it to get  
forgotten, and there are A LOT of bitrotten pulls on GitHub.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 16 2012
prev sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 16 May 2012 05:46:44 -0700, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com>  
 wrote:


 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .


 Huh?  Why the copy?  Just move src/core/thread.di to  
 import/core/thread.di

 object.di lives in import/core, I think it should be the same for all  
 the hand-maintained .di files.

 FWIW, I thought thread.di was being generated because of this.

 Also, I agree that thread and object are the only modules that need to  
 be .di files.  Everything else is already opaque for the most part, and  
 the pieces that aren't are just supporting code that can be visible.

 What we need to protect is the runtime implementation, so projects  
 cannot depend on private APIs that may change.

 -Steve


I have a waiting commit that will move thread.di to import/core and  
updates the makefiles accordingly, however, I need Zor's threading pulls  
merged before this can happen.

https://github.com/D-Programming-Language/druntime/pull/214/
https://github.com/D-Programming-Language/druntime/pull/204/

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 16 2012
prev sibling parent reply "dawg" <dawg dawgfoto.de> writes:
On Tuesday, 15 May 2012 at 06:46:58 UTC, Adam Wilson wrote:

 On Mon, 14 May 2012 23:11:50 -0700, Jonathan M Davis 
 <jmdavisProg gmx.com> wrote:


 On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:

 I have updated the make files so that only core.thread and 
 core.sync.* are
 run through DI generation. ALL other core.* modules are 
 copied into the
 import directory now.


 I assume that object.di and core/thread.di are being used 
 rather than their
 corresponding .d files being run through the .di generation? 
 They already have
 hand-crafted .di files.

 - Jonathan M Davis


 The funny thing is that core.thread.di was being run through 
 the DI generator in the old make file. I had left it that way, 
 I have posted a commit that moves it to the copy section.


I did that so that the same di generation logic is applied to
the handwritten import files. Currently I don't really expect any
difference though.
May 21 2012
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 21 May 2012 05:12:32 -0700, dawg <dawg dawgfoto.de> wrote:


 On Tuesday, 15 May 2012 at 06:46:58 UTC, Adam Wilson wrote:

 On Mon, 14 May 2012 23:11:50 -0700, Jonathan M Davis  
 <jmdavisProg gmx.com> wrote:


 On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:

 I have updated the make files so that only core.thread and  
 core.sync.* are
 run through DI generation. ALL other core.* modules are copied into  
 the
 import directory now.


 I assume that object.di and core/thread.di are being used rather than  
 their
 corresponding .d files being run through the .di generation? They  
 already have
 hand-crafted .di files.

 - Jonathan M Davis


 The funny thing is that core.thread.di was being run through the DI  
 generator in the old make file. I had left it that way, I have posted a  
 commit that moves it to the copy section.


 I did that so that the same di generation logic is applied to
 the handwritten import files. Currently I don't really expect any
 difference though.


Currently, there isn't any difference, however, once DI generation is  
changed to strip out function implementations it could change  
significantly. There is even talk of building a limited amount of semantic  
analysis into DI generation. Because of this, relying on the current DI  
generation in the future would not be a good idea; these changes are in  
preparation for future DI changes.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 21 2012
parent reply "timotheecour" <thelastmammoth gmail.com> writes:
questions:

A) as I understand it, the new di generation will systematically 
strip out the implementation of non-auto-return, non-templated 
functions, is that correct?

B) since there are some important differences with the current di 
files (in terms of inlining optimization, etc), will there be a 
dmd command-line flag to output those stripped down di files (eg: 
-stripdi), so user still has choice of which to output ?

C) why can't auto-return functions be semantically analyzed and 
resolved?
(eg:auto fun(int x){return x;} ) => generated di should be: int 
fun(int x); )

D) can we have an option to strip out template functions as well? 
This could be more or less customizable, eg with a file that 
contains a list of template functions to strip, or simply strip 
all templates). The library writer would instantiate those 
templates to certain predefined values. Eg:

module fun;
T fun1(T)(T x){
	return 2*x;
}
void dummy_instantiate(){
//instantiate to double, and repeat for all desired types, eg 
with a mixin.
	alias double T;
	fun1!(T)(T.init);
}
Then the library writer generates a library (static or shared) 
and the end user uses the templates with one of the allowed types 
(otherwise link error happens). In many cases (eg 
matrix/numerical libraries), all that's needed is a finite set of 
predefined types (eg int,float etc). Bonus points would be if the 
generated di file automatically generates template constraints to 
reflect the allowed types, to have compile time errors instead of 
link time ones.
Having ability to strip templates greatly simplifies distribution 
of code, as it doesn't have to carry all dependencies recursively 
if all one wants is a few predefined types.

D) btw, is there a way to force the instantiations more elegantly 
rather than using dummy_instantiate? in C++ we can just write 
something like: template double fun<double>(double);, but the 
same doesnt in D.
Jun 11 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 06/11/2012 09:37 AM, timotheecour wrote:

 questions:

 A) as I understand it, the new di generation will systematically strip out the
implementation of
 non-auto-return, non-templated functions, is that correct?


This is my understanding as well.


B) since there are some important differences with the current di files (in
terms of inlining
 optimization, etc), will there be a dmd command-line flag to output those
stripped down di files
 (eg: -stripdi), so user still has choice of which to output ?


You could use cp instead of dmd -H.


 C) why can't auto-return functions be semantically analyzed and resolved?
 (eg:auto fun(int x){return x;} ) => generated di should be: int fun(int
 x); )


Conditional compilation.

version(A) int x;
else version(B) double x;
else static assert(0);

auto foo(){return x;}

would need to be stripped to

version(A){
     int x;
     int foo();
}else version(B){
     double x;
     double foo();
}else static assert(0);

which is a nontrivial transformation.

This is just a simple example. Resolving the return type conditionally
in the general case is undecidable, therefore, making it work
satisfactorily involves a potentially unbounded amount of work.



 D) can we have an option to strip out template functions as well? This
 could be more or less customizable, eg with a file that contains a list
 of template functions to strip, or simply strip all templates). The
 library writer would instantiate those templates to certain predefined
 values. Eg:

 module fun;
 T fun1(T)(T x){
      return 2*x;
 }
 void dummy_instantiate(){
 //instantiate to double, and repeat for all desired types, eg with a mixin.
      alias double T;
      fun1!(T)(T.init);
 }
 Then the library writer generates a library (static or shared) and the
 end user uses the templates with one of the allowed types (otherwise
 link error happens). In many cases (eg matrix/numerical libraries), all
 that's needed is a finite set of predefined types (eg int,float etc).
 Bonus points would be if the generated di file automatically generates
 template constraints to reflect the allowed types, to have compile time
 errors instead of link time ones.
 Having ability to strip templates greatly simplifies distribution of
 code, as it doesn't have to carry all dependencies recursively if all
 one wants is a few predefined types.


You could use overloads instead and use templates for implementing them. 
Templates are generally only exposed in a well-designed library 
interface if they work with an unbounded number of types.


 D) btw, is there a way to force the instantiations more elegantly rather
 than using dummy_instantiate? in C++ we can just write something like:
 template double fun<double>(double);, but the same doesnt in D.


For example:

T foo(T)(T arg){return arg; pragma(msg, T);}

mixin template Instantiate(alias t,T...){
     static assert({
         void _(){mixin t!T;}
         return true;
     }());
}

mixin Instantiate!(foo,int);
mixin Instantiate!(foo,double);

The nested function named '_' is unnecessary. It works around a DMD bug, 
mixin t!T is 'not yet implemented in CTFE'.
Jun 11 2012
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 11 Jun 2012 04:55:37 -0700, Timon Gehr <timon.gehr gmx.ch> wrote:


 On 06/11/2012 09:37 AM, timotheecour wrote:

 questions:

 A) as I understand it, the new di generation will systematically strip  
 out the implementation of
 non-auto-return, non-templated functions, is that correct?


 This is my understanding as well.


Correct. A lot of community consultation went into the improvements.


 B) since there are some important differences with the current di files  
 (in terms of inlining
 optimization, etc), will there be a dmd command-line flag to output  
 those stripped down di files
 (eg: -stripdi), so user still has choice of which to output ?


 You could use cp instead of dmd -H.


In fact I rewrote the DRuntime makefiles to do precisely this with the  
hand-crafted .DI files.
https://github.com/D-Programming-Language/druntime/pull/218


 C) why can't auto-return functions be semantically analyzed and  
 resolved?
 (eg:auto fun(int x){return x;} ) => generated di should be: int fun(int
 x); )


 Conditional compilation.

 version(A) int x;
 else version(B) double x;
 else static assert(0);

 auto foo(){return x;}

 would need to be stripped to

 version(A){
      int x;
      int foo();
 }else version(B){
      double x;
      double foo();
 }else static assert(0);

 which is a nontrivial transformation.

 This is just a simple example. Resolving the return type conditionally
 in the general case is undecidable, therefore, making it work
 satisfactorily involves a potentially unbounded amount of work.


The general explanation is that any time you rewrite the AST (such as the  
operation performed above) you have to duplicate that work in DI  
generation to maintain semantic cohesion (what I put in is what I get  
out). Another reason is that DI generation is *required* to be run prior  
to the semantic analysis stage due to fact that your command line could  
alter the analysis and subsequent AST. In fact this note is one of the  
*very* few multi-line comments Walter put into the DMD source. In essence  
DI generation is an AST pretty-printer, and as such is must be run prior  
to analysis and after parsing. All my patch does is insert checks into the  
printing process to stop it from printing certain parts of the AST.

Theoretically one could rebuild a semantic analyzer that didn't change  
it's behavior based on the the command-line into the DI generation, but  
that would literally require rewriting DI generation from the ground up.  
And you'd still have to verify that the primary semantic analysis didn't  
change anything from the DI analysis. Then you'd have to write a  
reconciliation process. Personally, I think that we have better places to  
focus our efforts.

That said, what you really want is the full source embedded into the  
library (similar, to .NET's CIL). That would get you want you are after.  
Such a thing could actually be done except for OMF/Optlink. Since OMF  
doesn't support custom sections there is a no special place to store the  
code that the compiler could easily access. This would enable the compiler  
to extract the source during compilation and analyze the both the user  
source and library source and perform all possible optimizations  
accordingly. I look at adding COFF to DMD and my brain melted. There are  
enough #IFDEFs in there to cause permanent insanity... *sigh*


 D) can we have an option to strip out template functions as well? This
 could be more or less customizable, eg with a file that contains a list
 of template functions to strip, or simply strip all templates). The
 library writer would instantiate those templates to certain predefined
 values. Eg:

 module fun;
 T fun1(T)(T x){
      return 2*x;
 }
 void dummy_instantiate(){
 //instantiate to double, and repeat for all desired types, eg with a  
 mixin.
      alias double T;
      fun1!(T)(T.init);
 }
 Then the library writer generates a library (static or shared) and the
 end user uses the templates with one of the allowed types (otherwise
 link error happens). In many cases (eg matrix/numerical libraries), all
 that's needed is a finite set of predefined types (eg int,float etc).
 Bonus points would be if the generated di file automatically generates
 template constraints to reflect the allowed types, to have compile time
 errors instead of link time ones.
 Having ability to strip templates greatly simplifies distribution of
 code, as it doesn't have to carry all dependencies recursively if all
 one wants is a few predefined types.


 You could use overloads instead and use templates for implementing them.  
 Templates are generally only exposed in a well-designed library  
 interface if they work with an unbounded number of types.


 D) btw, is there a way to force the instantiations more elegantly rather
 than using dummy_instantiate? in C++ we can just write something like:
 template double fun<double>(double);, but the same doesnt in D.


 For example:

 T foo(T)(T arg){return arg; pragma(msg, T);}

 mixin template Instantiate(alias t,T...){
      static assert({
          void _(){mixin t!T;}
          return true;
      }());
 }

 mixin Instantiate!(foo,int);
 mixin Instantiate!(foo,double);

 The nested function named '_' is unnecessary. It works around a DMD bug,  
 mixin t!T is 'not yet implemented in CTFE'.



-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Jun 11 2012
parent reply "timotheecour" <thelastmammoth gmail.com> writes:
 mixin Instantiate!(foo,int);


Thanks for the syntax tip!


 You could use cp instead of dmd -H.


That won't produce the same output (eg large functions tend to be 
stripped currently), but I guess the current behavior is 
relatively useless so it's fine.



 want you are after. Such a thing could actually be done except
 for OMF/Optlink. Since OMF doesn't support custom sections
 there is a no special place to store the code that the compiler
 could easily access.



If we want to embed the AST inside an "import" library file 
(which is optional, could be done with a directory), wouldn't it 
be possible to store the AST as a global / static variable? When 
dmd compiles myfun.d, it generates the AST and inserts a global 
variable (eg void*_ast_myfun=...) in the data segment of the 
library/object file (that could be done with a mixin and be 
portable).

just started a new thread for related ideas: 
http://forum.dlang.org/thread/lmepufogzaxlbxwgubvl forum.dlang.org
thanks!
Jun 12 2012
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 12 Jun 2012 02:16:19 -0700, timotheecour  
<thelastmammoth gmail.com> wrote:


 mixin Instantiate!(foo,int);


 Thanks for the syntax tip!


 You could use cp instead of dmd -H.


 That won't produce the same output (eg large functions tend to be  
 stripped currently), but I guess the current behavior is relatively  
 useless so it's fine.



 want you are after. Such a thing could actually be done except
 for OMF/Optlink. Since OMF doesn't support custom sections
 there is a no special place to store the code that the compiler
 could easily access.



 If we want to embed the AST inside an "import" library file (which is  
 optional, could be done with a directory), wouldn't it be possible to  
 store the AST as a global / static variable? When dmd compiles myfun.d,  
 it generates the AST and inserts a global variable (eg  
 void*_ast_myfun=...) in the data segment of the library/object file  
 (that could be done with a mixin and be portable).

 just started a new thread for related ideas:  
 http://forum.dlang.org/thread/lmepufogzaxlbxwgubvl forum.dlang.org
 thanks!


Theoretically yes, but it's extremely poor form for the compiler to be  
adding variables to programmer created structures especially when said  
variables are only ever going to be used by the compiler.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Jun 12 2012