www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DI Generation Needs your Help!

reply "Adam Wilson" <flyboynw gmail.com> writes:
As you may all be aware, I've been trying to improve the automated  
generation of .di files and I now have something that I feel is testable.  
Currently the new code only makes the following changes.

1.	Function Implementations are removed
2.	Private Function Declarations are removed.
3.	Variable Initializers, except for const, are removed.

Everything else is left alone. Templates and mixins are not addressed with  
this code and *should* not be modified. That's where I need your help, the  
test cases I have written cover some basic scenarios but I don't have the  
capability to test these changes with the diverse code base that the  
community has created.

drey_ from IRC was kind enough to test build Derelict with the changes and  
has discovered a potential issue around private imports. Derelict uses  
private imports that contain types which are used in function alias  
declarations. As one would expect, this caused many compiler errors.   
Currently, I feel that private imports should be stripped from the DI file  
as they are intended to be internal to the module. However, I want to put  
it to the community to decide, and I would especially appreciate Mr.  
Bright's opinion on private imports in DI files.

If anyone wishes to test my changes against their code, you can download  
them from my git account here:  
https://LightBender github.com/LightBender/dmd.git
I only ask that you inform me of anything broken and provide a test case  
that reproduces the error.

The following is my current test case, it comes from a module in my  
project that has been modified to include the various tests I and others  
have been able to think of. I know for a fact that it is missing a great  
number of features available in D, and I would really appreciate any tests  
you might wish to add to this case.

module Horizon.Collections.Basic;

export void TestFunc(int a) {}
private void TestFunc2(int b) {}

const gss = "__gshared";

export interface IEnumerator
{
	export  property Object Current();

	export bool MoveNext();
	export void Reset();
}

export interface ICollection
{
	export  property uint Count();

	export int opApply(int delegate(ref Object) dg);
	export IEnumerator GetEnumerator();
}

export interface IList : ICollection
{
	export  property bool IsFixedSize();
	export  property bool IsReadOnly();

	export void Add(Object Value);
	export void Clear();
	export bool Contains(Object Value);
	export Object opIndex(ulong Index);
	export void opIndexAssign(Object Value, ulong Index);
	export int IndexOf(Object Value);
	export void Insert(ulong Index, Object Value);
	export void Remove(Object Value);
	export void RemoveAt(ulong Index);
}

export struct StructTesting
{
	private int i = 0;
	protected int j = 1;

	export void ExportedStructFunc() {}
	private void PrivateStructFunc() {}
}

export class List : IList
{
	export this(uint Capacity)
	{
		vCapacity = Capacity;
		internal = new Object[4];
	}

	private Object[] internal;

	private uint vCapacity = 0;
	export  property uint Capacity()
	{
		return vCapacity;
	}

	private uint vCount = 0;
	export  property uint Count()
	{
		return vCount;
	}

	export void Add(Object Value)
	{
		if(vCount == vCapacity) internal.length *= 2;

		internal[++vCount] = Value;
	}

	export int opApply(int delegate(ref Object) dg) { return 0; }
	export IEnumerator GetEnumerator() {return null;}

	export  property bool IsFixedSize() {return false;}
	export  property bool IsReadOnly() {return false;}

	export void Clear() { return; }
	export bool Contains(Object Value) {return false;}
	export Object opIndex(ulong Index) {return null;}
	export void opIndexAssign(Object Value, ulong Index) {}
	export int IndexOf(Object Value) {return 0;}
	export void Insert(ulong Index, Object Value) {}
	export void Remove(Object Value) {}
	export void RemoveAt(ulong Index) {}
	private int FooTest(int c) { return c*2; }
	protected int ProtectedTest(int d) { return d*3; }
}



This is the DI file as exported with the generation changes:

// D import file generated from 'Basic.d'
module Horizon.Collections.Basic;
export void TestFunc(int a);
const gss = "__gshared";
export interface IEnumerator
{
     export  property Object Current();

     export bool MoveNext();
     export void Reset();
}

export interface ICollection
{
     export  property uint Count();

     export int opApply(int delegate(ref Object) dg);
     export IEnumerator GetEnumerator();
}

export interface IList : ICollection
{
     export  property bool IsFixedSize();

     export  property bool IsReadOnly();

     export void Add(Object Value);
     export void Clear();
     export bool Contains(Object Value);
     export Object opIndex(ulong Index);
     export void opIndexAssign(Object Value, ulong Index);
     export int IndexOf(Object Value);
     export void Insert(ulong Index, Object Value);
     export void Remove(Object Value);
     export void RemoveAt(ulong Index);
}

export struct StructTesting
{
     private int i;

     protected int j;

     export void ExportedStructFunc();
}

export class List : IList
{
     export this(uint Capacity);
     private Object[] internal;

     private uint vCapacity;

     export  property uint Capacity();

     private uint vCount;

     export  property uint Count();

     export void Add(Object Value);
     export int opApply(int delegate(ref Object) dg);
     export IEnumerator GetEnumerator();
     export  property bool IsFixedSize();

     export  property bool IsReadOnly();

     export void Clear();
     export bool Contains(Object Value);
     export Object opIndex(ulong Index);
     export void opIndexAssign(Object Value, ulong Index);
     export int IndexOf(Object Value);
     export void Insert(ulong Index, Object Value);
     export void Remove(Object Value);
     export void RemoveAt(ulong Index);
     protected int ProtectedTest(int d);
}

Let me know what you think!

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 19 2011
next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 19 December 2011 at 08:11:26 UTC, Adam Wilson wrote:

 Let me know what you think!


Nice work! I think this is going to be more and more relevant now 
that the situation with shared libraries is getting more 
interesting in D.

I would argue that anything that can't be moved to a different 
library, like templates, manifest constants etc. must be kept in 
as-is. D interface generation isn't just useful for hiding source 
code; some people just want to put as much as possible in a 
shared library, speed up compilation or hide as much as possible 
with the restriction of keeping the code working as usual.

If people want to hide their source code at all cost, they can 
make this do a pass to easily identify what needs to be moved to 
a different module, then do the rest manually (which of course, 
only needs to be done once).
Dec 19 2011
prev sibling next sibling parent reply so <so so.so> writes:
Good work Adam!

After class/struct/interface level "export" we shouldn't need it again.
It already implies that everything (if not stated otherwise via private)  
will be exported.

Removing them also would automagicaly solve the inconsistency below.


 export class List : IList
 {
      export void RemoveAt(ulong Index);
      protected int ProtectedTest(int d);
 }


On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw gmail.com> wrote:


 As you may all be aware, I've been trying to improve the automated  
 generation of .di files and I now have something that I feel is  
 testable. Currently the new code only makes the following changes.

 1.	Function Implementations are removed
 2.	Private Function Declarations are removed.
 3.	Variable Initializers, except for const, are removed.

 Everything else is left alone. Templates and mixins are not addressed  
 with this code and *should* not be modified. That's where I need your  
 help, the test cases I have written cover some basic scenarios but I  
 don't have the capability to test these changes with the diverse code  
 base that the community has created.

 drey_ from IRC was kind enough to test build Derelict with the changes  
 and has discovered a potential issue around private imports. Derelict  
 uses private imports that contain types which are used in function alias  
 declarations. As one would expect, this caused many compiler errors.   
 Currently, I feel that private imports should be stripped from the DI  
 file as they are intended to be internal to the module. However, I want  
 to put it to the community to decide, and I would especially appreciate  
 Mr. Bright's opinion on private imports in DI files.

 If anyone wishes to test my changes against their code, you can download  
 them from my git account here:  
 https://LightBender github.com/LightBender/dmd.git
 I only ask that you inform me of anything broken and provide a test case  
 that reproduces the error.

 The following is my current test case, it comes from a module in my  
 project that has been modified to include the various tests I and others  
 have been able to think of. I know for a fact that it is missing a great  
 number of features available in D, and I would really appreciate any  
 tests you might wish to add to this case.

 module Horizon.Collections.Basic;

 export void TestFunc(int a) {}
 private void TestFunc2(int b) {}

 const gss = "__gshared";

 export interface IEnumerator
 {
 	export  property Object Current();

 	export bool MoveNext();
 	export void Reset();
 }

 export interface ICollection
 {
 	export  property uint Count();

 	export int opApply(int delegate(ref Object) dg);
 	export IEnumerator GetEnumerator();
 }

 export interface IList : ICollection
 {
 	export  property bool IsFixedSize();
 	export  property bool IsReadOnly();

 	export void Add(Object Value);
 	export void Clear();
 	export bool Contains(Object Value);
 	export Object opIndex(ulong Index);
 	export void opIndexAssign(Object Value, ulong Index);
 	export int IndexOf(Object Value);
 	export void Insert(ulong Index, Object Value);
 	export void Remove(Object Value);
 	export void RemoveAt(ulong Index);
 }

 export struct StructTesting
 {
 	private int i = 0;
 	protected int j = 1;

 	export void ExportedStructFunc() {}
 	private void PrivateStructFunc() {}
 }

 export class List : IList
 {
 	export this(uint Capacity)
 	{
 		vCapacity = Capacity;
 		internal = new Object[4];
 	}

 	private Object[] internal;

 	private uint vCapacity = 0;
 	export  property uint Capacity()
 	{
 		return vCapacity;
 	}

 	private uint vCount = 0;
 	export  property uint Count()
 	{
 		return vCount;
 	}

 	export void Add(Object Value)
 	{
 		if(vCount == vCapacity) internal.length *= 2;

 		internal[++vCount] = Value;
 	}

 	export int opApply(int delegate(ref Object) dg) { return 0; }
 	export IEnumerator GetEnumerator() {return null;}

 	export  property bool IsFixedSize() {return false;}
 	export  property bool IsReadOnly() {return false;}

 	export void Clear() { return; }
 	export bool Contains(Object Value) {return false;}
 	export Object opIndex(ulong Index) {return null;}
 	export void opIndexAssign(Object Value, ulong Index) {}
 	export int IndexOf(Object Value) {return 0;}
 	export void Insert(ulong Index, Object Value) {}
 	export void Remove(Object Value) {}
 	export void RemoveAt(ulong Index) {}
 	private int FooTest(int c) { return c*2; }
 	protected int ProtectedTest(int d) { return d*3; }
 }



 This is the DI file as exported with the generation changes:

 // D import file generated from 'Basic.d'
 module Horizon.Collections.Basic;
 export void TestFunc(int a);
 const gss = "__gshared";
 export interface IEnumerator
 {
      export  property Object Current();

      export bool MoveNext();
      export void Reset();
 }

 export interface ICollection
 {
      export  property uint Count();

      export int opApply(int delegate(ref Object) dg);
      export IEnumerator GetEnumerator();
 }

 export interface IList : ICollection
 {
      export  property bool IsFixedSize();

      export  property bool IsReadOnly();

      export void Add(Object Value);
      export void Clear();
      export bool Contains(Object Value);
      export Object opIndex(ulong Index);
      export void opIndexAssign(Object Value, ulong Index);
      export int IndexOf(Object Value);
      export void Insert(ulong Index, Object Value);
      export void Remove(Object Value);
      export void RemoveAt(ulong Index);
 }

 export struct StructTesting
 {
      private int i;

      protected int j;

      export void ExportedStructFunc();
 }

 export class List : IList
 {
      export this(uint Capacity);
      private Object[] internal;

      private uint vCapacity;

      export  property uint Capacity();

      private uint vCount;

      export  property uint Count();

      export void Add(Object Value);
      export int opApply(int delegate(ref Object) dg);
      export IEnumerator GetEnumerator();
      export  property bool IsFixedSize();

      export  property bool IsReadOnly();

      export void Clear();
      export bool Contains(Object Value);
      export Object opIndex(ulong Index);
      export void opIndexAssign(Object Value, ulong Index);
      export int IndexOf(Object Value);
      export void Insert(ulong Index, Object Value);
      export void Remove(Object Value);
      export void RemoveAt(ulong Index);
      protected int ProtectedTest(int d);
 }

 Let me know what you think!
Dec 19 2011
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 19 Dec 2011 03:53:27 -0800, so <so so.so> wrote:


 Good work Adam!

 After class/struct/interface level "export" we shouldn't need it again.
 It already implies that everything (if not stated otherwise via private)  
 will be exported.


Well, I tried it that way and it didn't work on Windows. Without the  
export in front of the function declaration the function won't get  
exported to Windows in a shared library situation. I have verified this  
behavior with Walter and he assures me that, under Windows, this is the  
correct behavior. On other OS'es public works just fine and D treats  
export as public, but I work on Windows so I have to use export. I am  
working on an article for the site documenting my experiences with dynamic  
libraries and the pitfalls you can run into, including this one. Expect to  
see the initial draft after I finish DI generation.


 Removing them also would automagicaly solve the inconsistency below.


 export class List : IList
 {
      export void RemoveAt(ulong Index);
      protected int ProtectedTest(int d);
 }


 On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw gmail.com>  
 wrote:


 As you may all be aware, I've been trying to improve the automated  
 generation of .di files and I now have something that I feel is  
 testable. Currently the new code only makes the following changes.

 1.	Function Implementations are removed
 2.	Private Function Declarations are removed.
 3.	Variable Initializers, except for const, are removed.

 Everything else is left alone. Templates and mixins are not addressed  
 with this code and *should* not be modified. That's where I need your  
 help, the test cases I have written cover some basic scenarios but I  
 don't have the capability to test these changes with the diverse code  
 base that the community has created.

 drey_ from IRC was kind enough to test build Derelict with the changes  
 and has discovered a potential issue around private imports. Derelict  
 uses private imports that contain types which are used in function  
 alias declarations. As one would expect, this caused many compiler  
 errors.  Currently, I feel that private imports should be stripped from  
 the DI file as they are intended to be internal to the module. However,  
 I want to put it to the community to decide, and I would especially  
 appreciate Mr. Bright's opinion on private imports in DI files.

 If anyone wishes to test my changes against their code, you can  
 download them from my git account here:  
 https://LightBender github.com/LightBender/dmd.git
 I only ask that you inform me of anything broken and provide a test  
 case that reproduces the error.

 The following is my current test case, it comes from a module in my  
 project that has been modified to include the various tests I and  
 others have been able to think of. I know for a fact that it is missing  
 a great number of features available in D, and I would really  
 appreciate any tests you might wish to add to this case.

 module Horizon.Collections.Basic;

 export void TestFunc(int a) {}
 private void TestFunc2(int b) {}

 const gss = "__gshared";

 export interface IEnumerator
 {
 	export  property Object Current();

 	export bool MoveNext();
 	export void Reset();
 }

 export interface ICollection
 {
 	export  property uint Count();

 	export int opApply(int delegate(ref Object) dg);
 	export IEnumerator GetEnumerator();
 }

 export interface IList : ICollection
 {
 	export  property bool IsFixedSize();
 	export  property bool IsReadOnly();

 	export void Add(Object Value);
 	export void Clear();
 	export bool Contains(Object Value);
 	export Object opIndex(ulong Index);
 	export void opIndexAssign(Object Value, ulong Index);
 	export int IndexOf(Object Value);
 	export void Insert(ulong Index, Object Value);
 	export void Remove(Object Value);
 	export void RemoveAt(ulong Index);
 }

 export struct StructTesting
 {
 	private int i = 0;
 	protected int j = 1;

 	export void ExportedStructFunc() {}
 	private void PrivateStructFunc() {}
 }

 export class List : IList
 {
 	export this(uint Capacity)
 	{
 		vCapacity = Capacity;
 		internal = new Object[4];
 	}

 	private Object[] internal;

 	private uint vCapacity = 0;
 	export  property uint Capacity()
 	{
 		return vCapacity;
 	}

 	private uint vCount = 0;
 	export  property uint Count()
 	{
 		return vCount;
 	}

 	export void Add(Object Value)
 	{
 		if(vCount == vCapacity) internal.length *= 2;

 		internal[++vCount] = Value;
 	}

 	export int opApply(int delegate(ref Object) dg) { return 0; }
 	export IEnumerator GetEnumerator() {return null;}

 	export  property bool IsFixedSize() {return false;}
 	export  property bool IsReadOnly() {return false;}

 	export void Clear() { return; }
 	export bool Contains(Object Value) {return false;}
 	export Object opIndex(ulong Index) {return null;}
 	export void opIndexAssign(Object Value, ulong Index) {}
 	export int IndexOf(Object Value) {return 0;}
 	export void Insert(ulong Index, Object Value) {}
 	export void Remove(Object Value) {}
 	export void RemoveAt(ulong Index) {}
 	private int FooTest(int c) { return c*2; }
 	protected int ProtectedTest(int d) { return d*3; }
 }



 This is the DI file as exported with the generation changes:

 // D import file generated from 'Basic.d'
 module Horizon.Collections.Basic;
 export void TestFunc(int a);
 const gss = "__gshared";
 export interface IEnumerator
 {
      export  property Object Current();

      export bool MoveNext();
      export void Reset();
 }

 export interface ICollection
 {
      export  property uint Count();

      export int opApply(int delegate(ref Object) dg);
      export IEnumerator GetEnumerator();
 }

 export interface IList : ICollection
 {
      export  property bool IsFixedSize();

      export  property bool IsReadOnly();

      export void Add(Object Value);
      export void Clear();
      export bool Contains(Object Value);
      export Object opIndex(ulong Index);
      export void opIndexAssign(Object Value, ulong Index);
      export int IndexOf(Object Value);
      export void Insert(ulong Index, Object Value);
      export void Remove(Object Value);
      export void RemoveAt(ulong Index);
 }

 export struct StructTesting
 {
      private int i;

      protected int j;

      export void ExportedStructFunc();
 }

 export class List : IList
 {
      export this(uint Capacity);
      private Object[] internal;

      private uint vCapacity;

      export  property uint Capacity();

      private uint vCount;

      export  property uint Count();

      export void Add(Object Value);
      export int opApply(int delegate(ref Object) dg);
      export IEnumerator GetEnumerator();
      export  property bool IsFixedSize();

      export  property bool IsReadOnly();

      export void Clear();
      export bool Contains(Object Value);
      export Object opIndex(ulong Index);
      export void opIndexAssign(Object Value, ulong Index);
      export int IndexOf(Object Value);
      export void Insert(ulong Index, Object Value);
      export void Remove(Object Value);
      export void RemoveAt(ulong Index);
      protected int ProtectedTest(int d);
 }

 Let me know what you think!





-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 19 2011
prev sibling next sibling parent reply so <so so.so> writes:
On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw gmail.com> wrote:


 Everything else is left alone. Templates and mixins are not addressed  
 with this code and *should* not be modified. That's where I need your  
 help, the test cases I have written cover some basic scenarios but I  
 don't have the capability to test these changes with the diverse code  
 base that the community has created.


I am not exactly sure about your problem with templates and mixins but  
i'll give it a try regardless :)
Since with templates there is no distinction between definition and  
decleration,
exposing them IMO should be solely based on thier module access signatures.

private struct A() // hide
public struct B()  // expose

Now if B or some another exposed structure in ".di" should call A,
compiler will take care of it by outputting an error as usual.
Dec 19 2011
next sibling parent so <so so.so> writes:
On Mon, 19 Dec 2011 14:11:31 +0200, so <so so.so> wrote:


 On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw gmail.com>  
 wrote:


 Everything else is left alone. Templates and mixins are not addressed  
 with this code and *should* not be modified. That's where I need your  
 help, the test cases I have written cover some basic scenarios but I  
 don't have the capability to test these changes with the diverse code  
 base that the community has created.


 I am not exactly sure about your problem with templates and mixins but  
 i'll give it a try regardless :)
 Since with templates there is no distinction between definition and  
 decleration,
 exposing them IMO should be solely based on thier module access  
 signatures.

 private struct A() // hide
 public struct B()  // expose

 Now if B or some another exposed structure in ".di" should call A,
 compiler will take care of it by outputting an error as usual.


One last thing!
Interfaces shouldn't have any business with "export", they should be  
treated just like templates.
Dec 19 2011
prev sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 19 December 2011 at 12:11:32 UTC, so wrote:

 On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson 
 <flyboynw gmail.com> wrote:


 Everything else is left alone. Templates and mixins are not 
 addressed with this code and *should* not be modified. That's 
 where I need your help, the test cases I have written cover 
 some basic scenarios but I don't have the capability to test 
 these changes with the diverse code base that the community 
 has created.


 I am not exactly sure about your problem with templates and 
 mixins but i'll give it a try regardless :)
 Since with templates there is no distinction between definition 
 and decleration,
 exposing them IMO should be solely based on thier module access 
 signatures.

 private struct A() // hide
 public struct B()  // expose

 Now if B or some another exposed structure in ".di" should call 
 A,
 compiler will take care of it by outputting an error as usual.


And if the public template tries to access the private one?

Private module members must be treated like any other unless the 
compiler can prove it removed all references to the private 
member.
Dec 19 2011
parent reply so <so so.so> writes:
On Mon, 19 Dec 2011 14:43:07 +0200, Jakob Ovrum <jakobovrum gmail.com>  
wrote:


 On Monday, 19 December 2011 at 12:11:32 UTC, so wrote:

 On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw gmail.com>  
 wrote:


 Everything else is left alone. Templates and mixins are not addressed  
 with this code and *should* not be modified. That's where I need your  
 help, the test cases I have written cover some basic scenarios but I  
 don't have the capability to test these changes with the diverse code  
 base that the community has created.


 I am not exactly sure about your problem with templates and mixins but  
 i'll give it a try regardless :)
 Since with templates there is no distinction between definition and  
 decleration,
 exposing them IMO should be solely based on thier module access  
 signatures.

 private struct A() // hide
 public struct B()  // expose

 Now if B or some another exposed structure in ".di" should call A,
 compiler will take care of it by outputting an error as usual.


 And if the public template tries to access the private one?

 Private module members must be treated like any other unless the  
 compiler can prove it removed all references to the private member.


You are right it was obscure, i'll try to clarify.
By "B or some another exposed structure" i meant things resides  
"completely" in ".di" file.
Things like Templates, mixins and if we get them one day inlined functions.
In this scenario a "original.d" file should be interpreted as:

original.d
   public void A()()
   private void B()()

whatitwasmeant.di
   public void A()()

whatitwasmeant.d
   private void B()()

Here B can call A but not the opposite.
A simple doesn't see B here. Things that resides completely in ".di" file  
have no way to access the things in its ".d" file unless it wasn't  
"exported".
Imagine the final/generated ".di" file IS the user and the original ".d"  
file is the library designer. While i think this is the right way, it  
probably not that easy to implement.
Dec 19 2011
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 19 Dec 2011 04:57:54 -0800, so <so so.so> wrote:


 On Mon, 19 Dec 2011 14:43:07 +0200, Jakob Ovrum <jakobovrum gmail.com>  
 wrote:


 On Monday, 19 December 2011 at 12:11:32 UTC, so wrote:

 On Mon, 19 Dec 2011 10:11:25 +0200, Adam Wilson <flyboynw gmail.com>  
 wrote:


 Everything else is left alone. Templates and mixins are not addressed  
 with this code and *should* not be modified. That's where I need your  
 help, the test cases I have written cover some basic scenarios but I  
 don't have the capability to test these changes with the diverse code  
 base that the community has created.


 I am not exactly sure about your problem with templates and mixins but  
 i'll give it a try regardless :)
 Since with templates there is no distinction between definition and  
 decleration,
 exposing them IMO should be solely based on thier module access  
 signatures.

 private struct A() // hide
 public struct B()  // expose

 Now if B or some another exposed structure in ".di" should call A,
 compiler will take care of it by outputting an error as usual.


 And if the public template tries to access the private one?

 Private module members must be treated like any other unless the  
 compiler can prove it removed all references to the private member.


 You are right it was obscure, i'll try to clarify.
 By "B or some another exposed structure" i meant things resides  
 "completely" in ".di" file.
 Things like Templates, mixins and if we get them one day inlined  
 functions.
 In this scenario a "original.d" file should be interpreted as:

 original.d
    public void A()()
    private void B()()

 whatitwasmeant.di
    public void A()()

 whatitwasmeant.d
    private void B()()

 Here B can call A but not the opposite.
 A simple doesn't see B here. Things that resides completely in ".di"  
 file have no way to access the things in its ".d" file unless it wasn't  
 "exported".
 Imagine the final/generated ".di" file IS the user and the original ".d"  
 file is the library designer. While i think this is the right way, it  
 probably not that easy to implement.


So the consensus is to keep all private members/imports/etc?

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 19 2011
parent reply so <so so.so> writes:
On Mon, 19 Dec 2011 22:24:18 +0200, Adam Wilson <flyboynw gmail.com> wrote:


 So the consensus is to keep all private members/imports/etc?


Ideally to get most out of the auto generation of di files (by definition  
of module system) we need to strip ALL private stuff.
Only this way we can say the automation is on par with handcrafting.
But if it does not look doable somehow, we can always handcraft them  
(which i'll generally prefer),
Dec 19 2011
next sibling parent so <so so.so> writes:
On Mon, 19 Dec 2011 23:04:27 +0200, so <so so.so> wrote:


 On Mon, 19 Dec 2011 22:24:18 +0200, Adam Wilson <flyboynw gmail.com>  
 wrote:


 So the consensus is to keep all private members/imports/etc?


 Ideally to get most out of the auto generation of di files (by  
 definition of module system) we need to strip ALL private stuff.
 Only this way we can say the automation is on par with handcrafting.
 But if it does not look doable somehow, we can always handcraft them  
 (which i'll generally prefer),


ALL "except" imports :)
Dec 19 2011
prev sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 19 December 2011 at 21:04:28 UTC, so wrote:

 On Mon, 19 Dec 2011 22:24:18 +0200, Adam Wilson 
 <flyboynw gmail.com> wrote:


 So the consensus is to keep all private members/imports/etc?


 Ideally to get most out of the auto generation of di files (by 
 definition of module system) we need to strip ALL private stuff.
 Only this way we can say the automation is on par with 
 handcrafting.
 But if it does not look doable somehow, we can always handcraft 
 them (which i'll generally prefer),


Private members can still be used from public templates and 
whatnot. You can't take them out unless you can statically proved 
all references within the same .di to that private member was 
removed in the stripping process.

foo.d
------

private i = 42;

// Is a public template, has to be left in
void bar()()
{
    /* Uses 'i' */
}

void baz()
{
    /* Also uses 'i' */
}
------

foo.di
------

void bar()()
{
    /* Still uses 'i', where did it go!? */
}

void baz(); // Fine, no longer need 'i'
------

Also, on a side note, function parameters can use types from 
private imports too. Be it function, data or import, it has to be 
left in even if private.
Dec 19 2011
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 19 Dec 2011 15:53:02 -0800, Jakob Ovrum <jakobovrum gmail.com>  
wrote:


 On Monday, 19 December 2011 at 21:04:28 UTC, so wrote:

 On Mon, 19 Dec 2011 22:24:18 +0200, Adam Wilson <flyboynw gmail.com>  
 wrote:


 So the consensus is to keep all private members/imports/etc?


 Ideally to get most out of the auto generation of di files (by  
 definition of module system) we need to strip ALL private stuff.
 Only this way we can say the automation is on par with handcrafting.
 But if it does not look doable somehow, we can always handcraft them  
 (which i'll generally prefer),


 Private members can still be used from public templates and whatnot. You  
 can't take them out unless you can statically proved all references  
 within the same .di to that private member was removed in the stripping  
 process.


It sounds at this point we will just be leaving everything private in the  
DI files, sans implementations.


 foo.d
 ------

 private i = 42;

 // Is a public template, has to be left in
 void bar()()
 {
     /* Uses 'i' */
 }

 void baz()
 {
     /* Also uses 'i' */
 }
 ------

 foo.di
 ------

 void bar()()
 {
     /* Still uses 'i', where did it go!? */
 }

 void baz(); // Fine, no longer need 'i'
 ------

 Also, on a side note, function parameters can use types from private  
 imports too. Be it function, data or import, it has to be left in even  
 if private.


This was discovered by drey_ on IRC and will be fixed by leaving all  
privates intact. I'll upload a patch to my git account this evening.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 19 2011
prev sibling next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 19.12.2011 09:11, Adam Wilson wrote:

 Currently, I feel that private imports should be stripped from the DI
 file as they are intended to be internal to the module. However, I want
 to put it to the community to decide, and I would especially appreciate
 Mr. Bright's opinion on private imports in DI files.


I'd very much like the dependencies to be shrinked in the di file, but I 
think you can only remove private imports if you can guarantee that no 
symbols are referenced by the generated di file (e.g. function 
signatures, variable types, templates).

IIRC removing imports might also have an influence on generated static 
ctor execution order, because there are some optimizations that remove 
dependencies if no imports with static ctors are found in a referenced 
module.

Otherwise, having a version of the di file without any implementation 
sounds like a good idea.
Dec 19 2011
parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Mon, 19 Dec 2011 14:22:10 +0100, Rainer Schuetze <r.sagitario gmx.de>  
wrote:


 On 19.12.2011 09:11, Adam Wilson wrote:

 Currently, I feel that private imports should be stripped from the DI
 file as they are intended to be internal to the module. However, I want
 to put it to the community to decide, and I would especially appreciate
 Mr. Bright's opinion on private imports in DI files.


 I'd very much like the dependencies to be shrinked in the di file, but I  
 think you can only remove private imports if you can guarantee that no  
 symbols are referenced by the generated di file (e.g. function  
 signatures, variable types, templates).

 IIRC removing imports might also have an influence on generated static  
 ctor execution order, because there are some optimizations that remove  
 dependencies if no imports with static ctors are found in a referenced  
 module.


No it should not, an import will not actually create the module info that
contains the import dependencies. So the initialization order will still  
rely
solely on the compiled module.
OTOH not declaring private static ctors can lead to unreferenced module  
infos
that would otherwise end up in the final executable.


 Otherwise, having a version of the di file without any implementation  
 sounds like a good idea.
Dec 19 2011
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 19 Dec 2011 06:27:20 -0800, Martin Nowak <dawg dawgfoto.de> wrote:


 On Mon, 19 Dec 2011 14:22:10 +0100, Rainer Schuetze <r.sagitario gmx.de>  
 wrote:


 On 19.12.2011 09:11, Adam Wilson wrote:

 Currently, I feel that private imports should be stripped from the DI
 file as they are intended to be internal to the module. However, I want
 to put it to the community to decide, and I would especially appreciate
 Mr. Bright's opinion on private imports in DI files.


 I'd very much like the dependencies to be shrinked in the di file, but  
 I think you can only remove private imports if you can guarantee that  
 no symbols are referenced by the generated di file (e.g. function  
 signatures, variable types, templates).

 IIRC removing imports might also have an influence on generated static  
 ctor execution order, because there are some optimizations that remove  
 dependencies if no imports with static ctors are found in a referenced  
 module.


 No it should not, an import will not actually create the module info that
 contains the import dependencies. So the initialization order will still  
 rely
 solely on the compiled module.
 OTOH not declaring private static ctors can lead to unreferenced module  
 infos
 that would otherwise end up in the final executable.


 Otherwise, having a version of the di file without any implementation  
 sounds like a good idea.




At this point it sounds like I need to keep all private members and  
imports. No problem, that's easy enough.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 19 2011
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/19/11 2:11 AM, Adam Wilson wrote:

 As you may all be aware, I've been trying to improve the automated
 generation of .di files and I now have something that I feel is
 testable. Currently the new code only makes the following changes.

 1. Function Implementations are removed
 2. Private Function Declarations are removed.
 3. Variable Initializers, except for const, are removed.


Don't forget immutable.


 Everything else is left alone. Templates and mixins are not addressed
 with this code and *should* not be modified. That's where I need your
 help, the test cases I have written cover some basic scenarios but I
 don't have the capability to test these changes with the diverse code
 base that the community has created.

 drey_ from IRC was kind enough to test build Derelict with the changes
 and has discovered a potential issue around private imports. Derelict
 uses private imports that contain types which are used in function alias
 declarations. As one would expect, this caused many compiler errors.
 Currently, I feel that private imports should be stripped from the DI
 file as they are intended to be internal to the module. However, I want
 to put it to the community to decide, and I would especially appreciate
 Mr. Bright's opinion on private imports in DI files.


I suspect you'd still need the private imports because template code may 
use them.

This is great work. It's almost a textbook example of how one can make a 
great positive impact on D's development by finding an area of 
improvement and working on it. Congratulations!

You may want to generate DIs for Phobos and take a look at them. Phobos 
uses a vast array of D's capabilities so it's an effective unittest for 
DI generation.


Thanks,

Andrei
Dec 19 2011
next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 19 Dec 2011 08:53:21 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 12/19/11 2:11 AM, Adam Wilson wrote:

 As you may all be aware, I've been trying to improve the automated
 generation of .di files and I now have something that I feel is
 testable. Currently the new code only makes the following changes.

 1. Function Implementations are removed
 2. Private Function Declarations are removed.
 3. Variable Initializers, except for const, are removed.


 Don't forget immutable.


Added to my list. :-)


 Everything else is left alone. Templates and mixins are not addressed
 with this code and *should* not be modified. That's where I need your
 help, the test cases I have written cover some basic scenarios but I
 don't have the capability to test these changes with the diverse code
 base that the community has created.

 drey_ from IRC was kind enough to test build Derelict with the changes
 and has discovered a potential issue around private imports. Derelict
 uses private imports that contain types which are used in function alias
 declarations. As one would expect, this caused many compiler errors.
 Currently, I feel that private imports should be stripped from the DI
 file as they are intended to be internal to the module. However, I want
 to put it to the community to decide, and I would especially appreciate
 Mr. Bright's opinion on private imports in DI files.


 I suspect you'd still need the private imports because template code may  
 use them.


Ok, this is a good reason, I'd still like to hear from Walter on the  
subject, but I think he'll probably agree. I'll add it to my list.


 This is great work. It's almost a textbook example of how one can make a  
 great positive impact on D's development by finding an area of  
 improvement and working on it. Congratulations!


Thank you! I have to admit that reason I took it on is because it was  
annoying the daylights out of me on my project, but then I guess that's  
how most open-source work gets done. :-)


 You may want to generate DIs for Phobos and take a look at them. Phobos  
 uses a vast array of D's capabilities so it's an effective unittest for  
 DI generation.


Now that is a good idea. I may need some help getting the test setup as  
I've never built anything as big as phobos before, but it would be a  
fantastic way to nail down the DI behavior and maybe help push Phobos to  
become a shared library.


 Thanks,

 Andrei



-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 19 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Btw Adam, a long long time ago (well, few years ago) there was an
effort for better DLL support in D: http://dsource.org/projects/ddl/

There was also a presentation here: http://vimeo.com/2264486 and
slides: http://replay.waybackmachine.org/20081203030930/http://team0xf.com/conference/DDL.pdf

Anyway, it might be a useful resource for what seem to be D's DLL problems.
Dec 19 2011
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 19 Dec 2011 09:36:27 -0800, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 Btw Adam, a long long time ago (well, few years ago) there was an
 effort for better DLL support in D: http://dsource.org/projects/ddl/

 There was also a presentation here: http://vimeo.com/2264486 and
 slides:  
 http://replay.waybackmachine.org/20081203030930/http://team0xf.com/conference/DDL.pdf

 Anyway, it might be a useful resource for what seem to be D's DLL  
 problems.


Those are awesome links. Thanks Andrej!

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 19 2011
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, December 19, 2011 10:53:21 Andrei Alexandrescu wrote:

 You may want to generate DIs for Phobos and take a look at them. Phobos
 uses a vast array of D's capabilities so it's an effective unittest for
 DI generation.


It's a great test ground, but I wouldn't want to see Phobos actually be 
released with .di files instead of d files. It would seriously harm inlining
and 
CTFE.

- Jonathan M Davis
Dec 19 2011
prev sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 19 Dec 2011 08:53:21 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 12/19/11 2:11 AM, Adam Wilson wrote:

 As you may all be aware, I've been trying to improve the automated
 generation of .di files and I now have something that I feel is
 testable. Currently the new code only makes the following changes.

 1. Function Implementations are removed
 2. Private Function Declarations are removed.
 3. Variable Initializers, except for const, are removed.


 Don't forget immutable.


I did it, but it wasn't pretty. I had to pass the immutable state via the  
HeaderGenState struct.


 Everything else is left alone. Templates and mixins are not addressed
 with this code and *should* not be modified. That's where I need your
 help, the test cases I have written cover some basic scenarios but I
 don't have the capability to test these changes with the diverse code
 base that the community has created.

 drey_ from IRC was kind enough to test build Derelict with the changes
 and has discovered a potential issue around private imports. Derelict
 uses private imports that contain types which are used in function alias
 declarations. As one would expect, this caused many compiler errors.
 Currently, I feel that private imports should be stripped from the DI
 file as they are intended to be internal to the module. However, I want
 to put it to the community to decide, and I would especially appreciate
 Mr. Bright's opinion on private imports in DI files.


 I suspect you'd still need the private imports because template code may  
 use them.


Privates are now all in.


 This is great work. It's almost a textbook example of how one can make a  
 great positive impact on D's development by finding an area of  
 improvement and working on it. Congratulations!

 You may want to generate DIs for Phobos and take a look at them. Phobos  
 uses a vast array of D's capabilities so it's an effective unittest for  
 DI generation.


 Thanks,

 Andrei



-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 19 2011
prev sibling next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 19 Dec 2011 00:11:25 -0800, Adam Wilson <flyboynw gmail.com> wrote:

The latest DI generation code is now on my Git account and ready for  
testing. It fixes the following issues:

1.	Privates should exist in the DI file to support public templates.
2.	Template classes and functions retain their implementations.
3.	Immutable types should retain their initializers.

At this point I could really use testing; you can download them from my  
git account here: https://LightBender github.com/LightBender/dmd.git
I am trying to get myself setup for building phobos as a test but this is  
proving to be a lengthy process.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 20 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Derelict works ok now, good work!

However, the .di files end up eating newlines.

Before:
double ALLEGRO_USECS_TO_SECS(long x)
{
return x / 1e+06;
}
double ALLEGRO_MSECS_TO_SECS(long x)
{
return x / 1000;
}
double ALLEGRO_BPS_TO_SECS(int x)
{
return 1 / x;
}

After:
double ALLEGRO_USECS_TO_SECS(long x);double ALLEGRO_MSECS_TO_SECS(long
x);double ALLEGRO_BPS_TO_SECS(int x);

I've tried merging
https://github.com/D-Programming-Language/dmd/pull/538 but it doesn't
fix this.
Dec 20 2011
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 20 Dec 2011 03:49:35 -0800, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 Derelict works ok now, good work!


This is fantastic news! Thanks for helping to test these changes. :-)


 However, the .di files end up eating newlines.

 Before:
 double ALLEGRO_USECS_TO_SECS(long x)
 {
 return x / 1e+06;
 }
 double ALLEGRO_MSECS_TO_SECS(long x)
 {
 return x / 1000;
 }
 double ALLEGRO_BPS_TO_SECS(int x)
 {
 return 1 / x;
 }

 After:
 double ALLEGRO_USECS_TO_SECS(long x);double ALLEGRO_MSECS_TO_SECS(long
 x);double ALLEGRO_BPS_TO_SECS(int x);

 I've tried merging
 https://github.com/D-Programming-Language/dmd/pull/538 but it doesn't
 fix this.


I noticed similar issues before I made changes to DI generation, but  
fixing it wasn't as high a priority as getting it working. For now I am  
going to continue testing the current changes, however, I would be open to  
prettifying it later. I think the simple fix is to add an extra newline  
between those functions.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 21 2011
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/20/11 2:03 AM, Adam Wilson wrote:

 On Mon, 19 Dec 2011 00:11:25 -0800, Adam Wilson <flyboynw gmail.com> wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Privates should exist in the DI file to support public templates.
 2. Template classes and functions retain their implementations.
 3. Immutable types should retain their initializers.


Great!


 At this point I could really use testing; you can download them from my
 git account here: https://LightBender github.com/LightBender/dmd.git
 I am trying to get myself setup for building phobos as a test but this
 is proving to be a lengthy process.


Nah, it's much easier than you might think. The posix.mak makefile is 
very small for what it does, and you need to literally change one line 
of code to make it generate .di headers.


Andrei
Dec 20 2011
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 20 Dec 2011 04:41:11 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 12/20/11 2:03 AM, Adam Wilson wrote:

 On Mon, 19 Dec 2011 00:11:25 -0800, Adam Wilson <flyboynw gmail.com>  
 wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Privates should exist in the DI file to support public templates.
 2. Template classes and functions retain their implementations.
 3. Immutable types should retain their initializers.


 Great!


 At this point I could really use testing; you can download them from my
 git account here: https://LightBender github.com/LightBender/dmd.git
 I am trying to get myself setup for building phobos as a test but this
 is proving to be a lengthy process.


 Nah, it's much easier than you might think. The posix.mak makefile is  
 very small for what it does, and you need to literally change one line  
 of code to make it generate .di headers.


 Andrei


Is it be add the proper -H options? I'll try it tomorrow, events pretty  
much overran my day today. If the new DI gen can build valid Phobos DI's I  
think it is time to consider opening a pull request and get feedback from  
the maintainers. Any disagreement with that?

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 21 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/21/11 2:12 AM, Adam Wilson wrote:

 On Tue, 20 Dec 2011 04:41:11 -0800, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 12/20/11 2:03 AM, Adam Wilson wrote:

 On Mon, 19 Dec 2011 00:11:25 -0800, Adam Wilson <flyboynw gmail.com>
 wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Privates should exist in the DI file to support public templates.
 2. Template classes and functions retain their implementations.
 3. Immutable types should retain their initializers.


 Great!


 At this point I could really use testing; you can download them from my
 git account here: https://LightBender github.com/LightBender/dmd.git
 I am trying to get myself setup for building phobos as a test but this
 is proving to be a lengthy process.


 Nah, it's much easier than you might think. The posix.mak makefile is
 very small for what it does, and you need to literally change one line
 of code to make it generate .di headers.


 Andrei


 Is it be add the proper -H options? I'll try it tomorrow, events pretty
 much overran my day today. If the new DI gen can build valid Phobos DI's
 I think it is time to consider opening a pull request and get feedback
 from the maintainers. Any disagreement with that?


I don't think we'll use .di for phobos for the time being. I was 
suggesting you do so in order to look over the generated .di files and 
make sure they work.

Andrei
Dec 21 2011
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 21 Dec 2011 08:41:43 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 12/21/11 2:12 AM, Adam Wilson wrote:

 On Tue, 20 Dec 2011 04:41:11 -0800, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 12/20/11 2:03 AM, Adam Wilson wrote:

 On Mon, 19 Dec 2011 00:11:25 -0800, Adam Wilson <flyboynw gmail.com>
 wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Privates should exist in the DI file to support public templates.
 2. Template classes and functions retain their implementations.
 3. Immutable types should retain their initializers.


 Great!


 At this point I could really use testing; you can download them from  
 my
 git account here: https://LightBender github.com/LightBender/dmd.git
 I am trying to get myself setup for building phobos as a test but this
 is proving to be a lengthy process.


 Nah, it's much easier than you might think. The posix.mak makefile is
 very small for what it does, and you need to literally change one line
 of code to make it generate .di headers.


 Andrei


 Is it be add the proper -H options? I'll try it tomorrow, events pretty
 much overran my day today. If the new DI gen can build valid Phobos DI's
 I think it is time to consider opening a pull request and get feedback
 from the maintainers. Any disagreement with that?


 I don't think we'll use .di for phobos for the time being. I was  
 suggesting you do so in order to look over the generated .di files and  
 make sure they work.

 Andrei


Oops! I mean't a pull on dmd for the new DI code. Sorry for the confusion!

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 21 2011
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/21/11 10:42 AM, Adam Wilson wrote:

 Oops! I mean't a pull on dmd for the new DI code. Sorry for the confusion!


Yes, absolutely!

Andrei
Dec 21 2011
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Wednesday, 21 December 2011 at 16:42:59 UTC, Adam Wilson wrote:

 Oops! I mean't a pull on dmd for the new DI code. Sorry for the 
 confusion!


Pull requests are always appreciated! :)

Remember, it's more like 
"please-review-and-pull-if-of-acceptable-quality request" :P
Dec 21 2011
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 19.12.2011 09:11, schrieb Adam Wilson:

 As you may all be aware, I've been trying to improve the automated
 generation of .di files and I now have something that I feel is
 testable. Currently the new code only makes the following changes.

 1. Function Implementations are removed
 2. Private Function Declarations are removed.
 3. Variable Initializers, except for const, are removed.

 Everything else is left alone. Templates and mixins are not addressed
 with this code and *should* not be modified. That's where I need your
 help, the test cases I have written cover some basic scenarios but I
 don't have the capability to test these changes with the diverse code
 base that the community has created.

 drey_ from IRC was kind enough to test build Derelict with the changes
 and has discovered a potential issue around private imports. Derelict
 uses private imports that contain types which are used in function alias
 declarations. As one would expect, this caused many compiler errors.
 Currently, I feel that private imports should be stripped from the DI
 file as they are intended to be internal to the module. However, I want
 to put it to the community to decide, and I would especially appreciate
 Mr. Bright's opinion on private imports in DI files.

 If anyone wishes to test my changes against their code, you can download
 them from my git account here:
 https://LightBender github.com/LightBender/dmd.git
 I only ask that you inform me of anything broken and provide a test case
 that reproduces the error.

 The following is my current test case, it comes from a module in my
 project that has been modified to include the various tests I and others
 have been able to think of. I know for a fact that it is missing a great
 number of features available in D, and I would really appreciate any
 tests you might wish to add to this case.

 module Horizon.Collections.Basic;

 export void TestFunc(int a) {}
 private void TestFunc2(int b) {}

 const gss = "__gshared";

 export interface IEnumerator
 {
 export  property Object Current();

 export bool MoveNext();
 export void Reset();
 }

 export interface ICollection
 {
 export  property uint Count();

 export int opApply(int delegate(ref Object) dg);
 export IEnumerator GetEnumerator();
 }

 export interface IList : ICollection
 {
 export  property bool IsFixedSize();
 export  property bool IsReadOnly();

 export void Add(Object Value);
 export void Clear();
 export bool Contains(Object Value);
 export Object opIndex(ulong Index);
 export void opIndexAssign(Object Value, ulong Index);
 export int IndexOf(Object Value);
 export void Insert(ulong Index, Object Value);
 export void Remove(Object Value);
 export void RemoveAt(ulong Index);
 }

 export struct StructTesting
 {
 private int i = 0;
 protected int j = 1;

 export void ExportedStructFunc() {}
 private void PrivateStructFunc() {}
 }

 export class List : IList
 {
 export this(uint Capacity)
 {
 vCapacity = Capacity;
 internal = new Object[4];
 }

 private Object[] internal;

 private uint vCapacity = 0;
 export  property uint Capacity()
 {
 return vCapacity;
 }

 private uint vCount = 0;
 export  property uint Count()
 {
 return vCount;
 }

 export void Add(Object Value)
 {
 if(vCount == vCapacity) internal.length *= 2;

 internal[++vCount] = Value;
 }

 export int opApply(int delegate(ref Object) dg) { return 0; }
 export IEnumerator GetEnumerator() {return null;}

 export  property bool IsFixedSize() {return false;}
 export  property bool IsReadOnly() {return false;}

 export void Clear() { return; }
 export bool Contains(Object Value) {return false;}
 export Object opIndex(ulong Index) {return null;}
 export void opIndexAssign(Object Value, ulong Index) {}
 export int IndexOf(Object Value) {return 0;}
 export void Insert(ulong Index, Object Value) {}
 export void Remove(Object Value) {}
 export void RemoveAt(ulong Index) {}
 private int FooTest(int c) { return c*2; }
 protected int ProtectedTest(int d) { return d*3; }
 }



 This is the DI file as exported with the generation changes:

 // D import file generated from 'Basic.d'
 module Horizon.Collections.Basic;
 export void TestFunc(int a);
 const gss = "__gshared";
 export interface IEnumerator
 {
 export  property Object Current();

 export bool MoveNext();
 export void Reset();
 }

 export interface ICollection
 {
 export  property uint Count();

 export int opApply(int delegate(ref Object) dg);
 export IEnumerator GetEnumerator();
 }

 export interface IList : ICollection
 {
 export  property bool IsFixedSize();

 export  property bool IsReadOnly();

 export void Add(Object Value);
 export void Clear();
 export bool Contains(Object Value);
 export Object opIndex(ulong Index);
 export void opIndexAssign(Object Value, ulong Index);
 export int IndexOf(Object Value);
 export void Insert(ulong Index, Object Value);
 export void Remove(Object Value);
 export void RemoveAt(ulong Index);
 }

 export struct StructTesting
 {
 private int i;

 protected int j;

 export void ExportedStructFunc();
 }

 export class List : IList
 {
 export this(uint Capacity);
 private Object[] internal;

 private uint vCapacity;

 export  property uint Capacity();

 private uint vCount;

 export  property uint Count();

 export void Add(Object Value);
 export int opApply(int delegate(ref Object) dg);
 export IEnumerator GetEnumerator();
 export  property bool IsFixedSize();

 export  property bool IsReadOnly();

 export void Clear();
 export bool Contains(Object Value);
 export Object opIndex(ulong Index);
 export void opIndexAssign(Object Value, ulong Index);
 export int IndexOf(Object Value);
 export void Insert(ulong Index, Object Value);
 export void Remove(Object Value);
 export void RemoveAt(ulong Index);
 protected int ProtectedTest(int d);
 }

 Let me know what you think!


Just out of curiousity, could removing private function delcerations 
lead to a different vtable layout? (And therefore crash the application)

-- 
Kind Regards
Benjamin Thaut
Dec 21 2011
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 12/21/2011 04:58 PM, Benjamin Thaut wrote:

 Am 19.12.2011 09:11, schrieb Adam Wilson:

 As you may all be aware, I've been trying to improve the automated
 generation of .di files and I now have something that I feel is
 testable. Currently the new code only makes the following changes.

 1. Function Implementations are removed
 2. Private Function Declarations are removed.
 3. Variable Initializers, except for const, are removed.

 Everything else is left alone. Templates and mixins are not addressed
 with this code and *should* not be modified. That's where I need your
 help, the test cases I have written cover some basic scenarios but I
 don't have the capability to test these changes with the diverse code
 base that the community has created.

 drey_ from IRC was kind enough to test build Derelict with the changes
 and has discovered a potential issue around private imports. Derelict
 uses private imports that contain types which are used in function alias
 declarations. As one would expect, this caused many compiler errors.
 Currently, I feel that private imports should be stripped from the DI
 file as they are intended to be internal to the module. However, I want
 to put it to the community to decide, and I would especially appreciate
 Mr. Bright's opinion on private imports in DI files.

 If anyone wishes to test my changes against their code, you can download
 them from my git account here:
 https://LightBender github.com/LightBender/dmd.git
 I only ask that you inform me of anything broken and provide a test case
 that reproduces the error.

 The following is my current test case, it comes from a module in my
 project that has been modified to include the various tests I and others
 have been able to think of. I know for a fact that it is missing a great
 number of features available in D, and I would really appreciate any
 tests you might wish to add to this case.

 module Horizon.Collections.Basic;

 export void TestFunc(int a) {}
 private void TestFunc2(int b) {}

 const gss = "__gshared";

 export interface IEnumerator
 {
 export  property Object Current();

 export bool MoveNext();
 export void Reset();
 }

 export interface ICollection
 {
 export  property uint Count();

 export int opApply(int delegate(ref Object) dg);
 export IEnumerator GetEnumerator();
 }

 export interface IList : ICollection
 {
 export  property bool IsFixedSize();
 export  property bool IsReadOnly();

 export void Add(Object Value);
 export void Clear();
 export bool Contains(Object Value);
 export Object opIndex(ulong Index);
 export void opIndexAssign(Object Value, ulong Index);
 export int IndexOf(Object Value);
 export void Insert(ulong Index, Object Value);
 export void Remove(Object Value);
 export void RemoveAt(ulong Index);
 }

 export struct StructTesting
 {
 private int i = 0;
 protected int j = 1;

 export void ExportedStructFunc() {}
 private void PrivateStructFunc() {}
 }

 export class List : IList
 {
 export this(uint Capacity)
 {
 vCapacity = Capacity;
 internal = new Object[4];
 }

 private Object[] internal;

 private uint vCapacity = 0;
 export  property uint Capacity()
 {
 return vCapacity;
 }

 private uint vCount = 0;
 export  property uint Count()
 {
 return vCount;
 }

 export void Add(Object Value)
 {
 if(vCount == vCapacity) internal.length *= 2;

 internal[++vCount] = Value;
 }

 export int opApply(int delegate(ref Object) dg) { return 0; }
 export IEnumerator GetEnumerator() {return null;}

 export  property bool IsFixedSize() {return false;}
 export  property bool IsReadOnly() {return false;}

 export void Clear() { return; }
 export bool Contains(Object Value) {return false;}
 export Object opIndex(ulong Index) {return null;}
 export void opIndexAssign(Object Value, ulong Index) {}
 export int IndexOf(Object Value) {return 0;}
 export void Insert(ulong Index, Object Value) {}
 export void Remove(Object Value) {}
 export void RemoveAt(ulong Index) {}
 private int FooTest(int c) { return c*2; }
 protected int ProtectedTest(int d) { return d*3; }
 }



 This is the DI file as exported with the generation changes:

 // D import file generated from 'Basic.d'
 module Horizon.Collections.Basic;
 export void TestFunc(int a);
 const gss = "__gshared";
 export interface IEnumerator
 {
 export  property Object Current();

 export bool MoveNext();
 export void Reset();
 }

 export interface ICollection
 {
 export  property uint Count();

 export int opApply(int delegate(ref Object) dg);
 export IEnumerator GetEnumerator();
 }

 export interface IList : ICollection
 {
 export  property bool IsFixedSize();

 export  property bool IsReadOnly();

 export void Add(Object Value);
 export void Clear();
 export bool Contains(Object Value);
 export Object opIndex(ulong Index);
 export void opIndexAssign(Object Value, ulong Index);
 export int IndexOf(Object Value);
 export void Insert(ulong Index, Object Value);
 export void Remove(Object Value);
 export void RemoveAt(ulong Index);
 }

 export struct StructTesting
 {
 private int i;

 protected int j;

 export void ExportedStructFunc();
 }

 export class List : IList
 {
 export this(uint Capacity);
 private Object[] internal;

 private uint vCapacity;

 export  property uint Capacity();

 private uint vCount;

 export  property uint Count();

 export void Add(Object Value);
 export int opApply(int delegate(ref Object) dg);
 export IEnumerator GetEnumerator();
 export  property bool IsFixedSize();

 export  property bool IsReadOnly();

 export void Clear();
 export bool Contains(Object Value);
 export Object opIndex(ulong Index);
 export void opIndexAssign(Object Value, ulong Index);
 export int IndexOf(Object Value);
 export void Insert(ulong Index, Object Value);
 export void Remove(Object Value);
 export void RemoveAt(ulong Index);
 protected int ProtectedTest(int d);
 }

 Let me know what you think!


 Just out of curiousity, could removing private function delcerations
 lead to a different vtable layout? (And therefore crash the application)


No, private implies final.
Dec 21 2011
prev sibling next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
The latest DI generation code is now on my Git account and ready for  
testing. It fixes the following issues:

1.	Variables in template functions/classes would be stripped of their  
initializers.
2.	Manifest Constants would lose their initializers.

DI generation is getting close! Every Phobos DI module I've tried has  
passed the VisualD parser/lexer (for whatever that is worth). I want to do  
some build testing with those files, but even complicated modules like  
std.algorithm and std.regex are passing the lexer/parser. If everything  
goes well with that testing I'll open up a pull for dmd. Hopefully proper  
DI generation will make it into 2.058!
You can get the latest from my git account here:  
https://LightBender github.com/LightBender/dmd.git

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 21 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-22 08:07, Adam Wilson wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Variables in template functions/classes would be stripped of their
 initializers.
 2. Manifest Constants would lose their initializers.

 DI generation is getting close! Every Phobos DI module I've tried has
 passed the VisualD parser/lexer (for whatever that is worth). I want to
 do some build testing with those files, but even complicated modules
 like std.algorithm and std.regex are passing the lexer/parser. If
 everything goes well with that testing I'll open up a pull for dmd.
 Hopefully proper DI generation will make it into 2.058!
 You can get the latest from my git account here:
 https://LightBender github.com/LightBender/dmd.git


Can the generator handle this:

id objc_msgSend (ARGS...)(id theReceiver, SEL theSelector, ARGS args)
{
     alias extern (C) id function (id, SEL, ARGS) fp;
     return (cast(fp)&c_objc_msgSend)(theReceiver, theSelector, args);
}

I know it had problems with that, at least in previous versions of DMD.

-- 
/Jacob Carlborg
Dec 21 2011
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 21 Dec 2011 23:43:56 -0800, Jacob Carlborg <doob me.com> wrote:


 On 2011-12-22 08:07, Adam Wilson wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Variables in template functions/classes would be stripped of their
 initializers.
 2. Manifest Constants would lose their initializers.

 DI generation is getting close! Every Phobos DI module I've tried has
 passed the VisualD parser/lexer (for whatever that is worth). I want to
 do some build testing with those files, but even complicated modules
 like std.algorithm and std.regex are passing the lexer/parser. If
 everything goes well with that testing I'll open up a pull for dmd.
 Hopefully proper DI generation will make it into 2.058!
 You can get the latest from my git account here:
 https://LightBender github.com/LightBender/dmd.git


 Can the generator handle this:

 id objc_msgSend (ARGS...)(id theReceiver, SEL theSelector, ARGS args)
 {
      alias extern (C) id function (id, SEL, ARGS) fp;
      return (cast(fp)&c_objc_msgSend)(theReceiver, theSelector, args);
 }

 I know it had problems with that, at least in previous versions of DMD.


Well, without the accompanying types that are used in the declaration I'm  
not entire sure how I could give you a good answer. However, since it's a  
template function, I can say that template functions in DI files are  
basically straight implementation dumps of the D file. Can you tell me  
where DI generation hiccups in that code? If the hiccup is inside the  
function impl than I doubt anything I've changed will fix your error.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-22 18:33, Adam Wilson wrote:

 On Wed, 21 Dec 2011 23:43:56 -0800, Jacob Carlborg <doob me.com> wrote:


 On 2011-12-22 08:07, Adam Wilson wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Variables in template functions/classes would be stripped of their
 initializers.
 2. Manifest Constants would lose their initializers.

 DI generation is getting close! Every Phobos DI module I've tried has
 passed the VisualD parser/lexer (for whatever that is worth). I want to
 do some build testing with those files, but even complicated modules
 like std.algorithm and std.regex are passing the lexer/parser. If
 everything goes well with that testing I'll open up a pull for dmd.
 Hopefully proper DI generation will make it into 2.058!
 You can get the latest from my git account here:
 https://LightBender github.com/LightBender/dmd.git


 Can the generator handle this:

 id objc_msgSend (ARGS...)(id theReceiver, SEL theSelector, ARGS args)
 {
 alias extern (C) id function (id, SEL, ARGS) fp;
 return (cast(fp)&c_objc_msgSend)(theReceiver, theSelector, args);
 }

 I know it had problems with that, at least in previous versions of DMD.


 Well, without the accompanying types that are used in the declaration
 I'm not entire sure how I could give you a good answer. However, since
 it's a template function, I can say that template functions in DI files
 are basically straight implementation dumps of the D file. Can you tell
 me where DI generation hiccups in that code? If the hiccup is inside the
 function impl than I doubt anything I've changed will fix your error.


Never mind, it seems to be working.

-- 
/Jacob Carlborg
Dec 22 2011
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Thu, 22 Dec 2011 13:02:49 -0800, Jacob Carlborg <doob me.com> wrote:


 On 2011-12-22 18:33, Adam Wilson wrote:

 On Wed, 21 Dec 2011 23:43:56 -0800, Jacob Carlborg <doob me.com> wrote:


 On 2011-12-22 08:07, Adam Wilson wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Variables in template functions/classes would be stripped of their
 initializers.
 2. Manifest Constants would lose their initializers.

 DI generation is getting close! Every Phobos DI module I've tried has
 passed the VisualD parser/lexer (for whatever that is worth). I want  
 to
 do some build testing with those files, but even complicated modules
 like std.algorithm and std.regex are passing the lexer/parser. If
 everything goes well with that testing I'll open up a pull for dmd.
 Hopefully proper DI generation will make it into 2.058!
 You can get the latest from my git account here:
 https://LightBender github.com/LightBender/dmd.git


 Can the generator handle this:

 id objc_msgSend (ARGS...)(id theReceiver, SEL theSelector, ARGS args)
 {
 alias extern (C) id function (id, SEL, ARGS) fp;
 return (cast(fp)&c_objc_msgSend)(theReceiver, theSelector, args);
 }

 I know it had problems with that, at least in previous versions of DMD.


 Well, without the accompanying types that are used in the declaration
 I'm not entire sure how I could give you a good answer. However, since
 it's a template function, I can say that template functions in DI files
 are basically straight implementation dumps of the D file. Can you tell
 me where DI generation hiccups in that code? If the hiccup is inside the
 function impl than I doubt anything I've changed will fix your error.


 Never mind, it seems to be working.


Did you check it against my forked code? I want to make sure that there  
are as few errors as possible before I open up the pull. :-)

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-12-23 01:22, Adam Wilson wrote:

 On Thu, 22 Dec 2011 13:02:49 -0800, Jacob Carlborg <doob me.com> wrote:

 Never mind, it seems to be working.


 Did you check it against my forked code? I want to make sure that there
 are as few errors as possible before I open up the pull. :-)


No, I checked with DMD 1.072.

-- 
/Jacob Carlborg
Dec 23 2011
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/22/11 1:07 AM, Adam Wilson wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Variables in template functions/classes would be stripped of their
 initializers.
 2. Manifest Constants would lose their initializers.

 DI generation is getting close! Every Phobos DI module I've tried has
 passed the VisualD parser/lexer (for whatever that is worth). I want to
 do some build testing with those files, but even complicated modules
 like std.algorithm and std.regex are passing the lexer/parser. If
 everything goes well with that testing I'll open up a pull for dmd.
 Hopefully proper DI generation will make it into 2.058!
 You can get the latest from my git account here:
 https://LightBender github.com/LightBender/dmd.git


Great! Did you fix indentation?

Also, I think testing with Visual Studio is insufficient. One good test 
is to extract all unittest code from std modules, and then make sure it 
compiles and runs when you only import the .di files.

Andrei
Dec 22 2011
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Thu, 22 Dec 2011 08:46:39 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 12/22/11 1:07 AM, Adam Wilson wrote:

 The latest DI generation code is now on my Git account and ready for
 testing. It fixes the following issues:

 1. Variables in template functions/classes would be stripped of their
 initializers.
 2. Manifest Constants would lose their initializers.

 DI generation is getting close! Every Phobos DI module I've tried has
 passed the VisualD parser/lexer (for whatever that is worth). I want to
 do some build testing with those files, but even complicated modules
 like std.algorithm and std.regex are passing the lexer/parser. If
 everything goes well with that testing I'll open up a pull for dmd.
 Hopefully proper DI generation will make it into 2.058!
 You can get the latest from my git account here:
 https://LightBender github.com/LightBender/dmd.git


 Great! Did you fix indentation?

 Also, I think testing with Visual Studio is insufficient. One good test  
 is to extract all unittest code from std modules, and then make sure it  
 compiles and runs when you only import the .di files.

 Andrei


No I haven't, there is an open pull for something about indentations, so I  
wasn't sure if I needed to. I will certainly do what I can for  
indentations, but the generation process is a bit messy so I can't promise  
it will be perfect.

I completely agree, it's just the baseline sanity check I'm using while I  
hack on it. Unless there is a way to export the unit tests automatically,  
I don't have the time to hand port all the unit tests for Phobos. I think  
I'll just pick a few modules with differing levels of complexity and use  
those unit tests.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Dec 22 2011
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/19/11, Adam Wilson <flyboynw gmail.com> wrote:

 If anyone wishes to test my changes against their code, you can download
 them from my git account here:
 https://LightBender github.com/LightBender/dmd.git


This seems to work nicely even with the latest release. It's much
better than the current .di generation for sure. Any plans on merging
this with mainline before it goes stale?
Mar 14 2012
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 14 Mar 2012 12:30:00 -0700, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 On 12/19/11, Adam Wilson <flyboynw gmail.com> wrote:

 If anyone wishes to test my changes against their code, you can download
 them from my git account here:
 https://LightBender github.com/LightBender/dmd.git


 This seems to work nicely even with the latest release. It's much
 better than the current .di generation for sure. Any plans on merging
 this with mainline before it goes stale?


I am currently maintaining this against DMD HEAD, but until this bug  
(http://d.puremagic.com/issues/show_bug.cgi?id=7423) gets fixed I can't  
open a pull request because the patch will break both the Phobos and  
DRuntime builds badly without it.
I would fix the bug myself, but I am getting married in a month and am  
rather short on time. Earliest I'll likely be able to investigate a fix is  
May. If anyone else wants to take a shot at a fix before then, I'd happily  
open up a pull request once they're done.
Mar 14 2012
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/14/12, Adam Wilson <flyboynw gmail.com> wrote:

 I would fix the bug myself, but I am getting married in a month and am
 rather short on time.


That's ok I'm in no hurry just curious. Congrats on getting married! :)
Mar 14 2012
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 14 Mar 2012 14:23:19 -0700, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 On 3/14/12, Adam Wilson <flyboynw gmail.com> wrote:

 I would fix the bug myself, but I am getting married in a month and am
 rather short on time.


 That's ok I'm in no hurry just curious. Congrats on getting married! :)


Thank you! Hehe, well, I have big plans for D, but I need DI files to work  
better to execute them, so I am a little bit of a hurry, but real life  
wins for now. ;-)

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Mar 14 2012
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/14/12, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:

 This seems to work nicely even with the latest release. It's much
 better than the current .di generation for sure. Any plans on merging
 this with mainline before it goes stale?


Although I would really like to be able to keep documentation comments
in the .di files, e.g.:

class Foo {
    /** Commented ctor. */
    this() { }
}

->
class Foo {
    /** Commented ctor. */
    this();
}
Mar 14 2012
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 14 Mar 2012 12:35:03 -0700, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:


 On 3/14/12, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:

 This seems to work nicely even with the latest release. It's much
 better than the current .di generation for sure. Any plans on merging
 this with mainline before it goes stale?


 Although I would really like to be able to keep documentation comments
 in the .di files, e.g.:

 class Foo {
     /** Commented ctor. */
     this() { }
 }

 ->
 class Foo {
     /** Commented ctor. */
     this();
 }


I'll look into it when I get the chance. :-)

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Mar 14 2012
prev sibling parent reply Alvaro <alvaroDotSegura gmail.com> writes:
El 19/12/2011 9:11, Adam Wilson escribió:

 1. Function Implementations are removed


What about function inlining? would it work from a different module?

I think the implementation of small functions should be kept so that 
client code can inline them.

The current DMD does this apparently, it keeps small functions in .di files.
Mar 14 2012
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 14 Mar 2012 13:45:13 -0700, Alvaro <alvaroDotSegura gmail.com>  =

wrote:


 El 19/12/2011 9:11, Adam Wilson escribi=F3:

 1. Function Implementations are removed


 What about function inlining? would it work from a different module?

 I think the implementation of small functions should be kept so that  =



 client code can inline them.

 The current DMD does this apparently, it keeps small functions in .di =


 =


 files.


The problem is that in DI generation, at least as near as I can tell, D =
 =

has now idea how big a function is during DI generation. In my experienc=
e  =

it was keeping all functions not just small ones. And frankly DI  =

generation is targeted at library builders who are well aware of the  =

inlining trade-offs. And then comes the question of how do you define  =

"small functions" to an adequately technical level. Because theoreticall=
y  =

you can inline ANYTHING. Yes, there are rules the prevent inlining, but =
 =

you'll also note that the state of these rules is not guaranteed to be  =

known at generation time.

DI generation currently works as such. After the code has been parsed in=
to  =

an AST, the compiler runs a special set of virtual functions called  =

toCBuffer (and it's variants) that is used to print out the AST in sourc=
e  =

form again. NO semantic analysis of any kind has been performed on the A=
ST  =

yet. And you wouldn't want to generate DI's after semantic analysis as t=
he  =

analysis fundamentally alters the AST such that you would not get the sa=
me  =

code back out and some things would be missing from the DI files that yo=
u  =

intended to be there. The AST at DI generation is an incredibly naive  =

representation of the SOURCE not the SEMANTICS; which is what you would =
 =

need to determine the eligibility of inlining.

The answer that Walter has always given for objections about how DI file=
s  =

are built is that if you want anything customized about the output, you =
 =

have to do it yourself. DI generation will probably NEVER be as perfect =
as  =

everyone wants. But I think this solution represents a best effort to ge=
t  =

DI files to a point where the community agrees that they would be most  =

useful in achieving their intended purpose, which is as interface files =
 =

for compiled libraries. It's not perfect, but it gets you A LOT further =
 =

than the current one, if you need customization beyond that, well, D let=
s  =

you do that too. :-)

-- =

Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Mar 14 2012
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
Adam Wilson:

I think I said this before, but for what it's worth,
I think your new approach is spot-on correct.
Mar 14 2012
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 14 Mar 2012 14:17:28 -0700, Adam D. Ruppe  
<destructionator gmail.com> wrote:


 Adam Wilson:

 I think I said this before, but for what it's worth,
 I think your new approach is spot-on correct.


Thank you. More importantly though, Walter and Andrei also agree, so in  
the end, the inlinaholics can just deal. :-)

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Mar 14 2012
prev sibling parent reply Alvaro <alvaroDotSegura gmail.com> writes:
El 14/03/2012 22:13, Adam Wilson escribió:

 On Wed, 14 Mar 2012 13:45:13 -0700, Alvaro <alvaroDotSegura gmail.com>
 wrote:

 The problem is that in DI generation, at least as near as I can tell, D
 has now idea how big a function is during DI generation. In my
 experience it was keeping all functions not just small ones. And frankly
 DI generation is targeted at library builders who are well aware of the
 inlining trade-offs. And then comes the question of how do you define
 "small functions" to an adequately technical level.>


OK, I rechecked. DMD -H is at least omitting function bodies containing 
foreach() loops (and maybe other cases). Maybe that led me to assume things.

AFAIK some C++ compilers use heuristics (don't really know what that is 
exactly) to decide what "small functions" deserved automatic inlining.

And well, C++ library builders are aware of that and usually keep the 
small functions they want inlined in the header file (e.g. in the class 
declaration). I can see a lot of cases where D libraries would benefit 
from allowing some of their functions to be inlined. Think of 
 properties, they're often pretty short.

Other than this detail, the improvement in DI generation is very welcome.
Mar 14 2012
parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 14 Mar 2012 14:42:54 -0700, Alvaro <alvaroDotSegura gmail.com>  =

wrote:


 El 14/03/2012 22:13, Adam Wilson escribi=F3:
  > On Wed, 14 Mar 2012 13:45:13 -0700, Alvaro <alvaroDotSegura gmail.c=


om>

  > wrote:
  >
  > The problem is that in DI generation, at least as near as I can tel=


l,  =


 D
  > has now idea how big a function is during DI generation. In my
  > experience it was keeping all functions not just small ones. And  =



 frankly
  > DI generation is targeted at library builders who are well aware of=


  =


 the
  > inlining trade-offs. And then comes the question of how do you defi=


ne

  > "small functions" to an adequately technical level.>

 OK, I rechecked. DMD -H is at least omitting function bodies containin=


g  =


 foreach() loops (and maybe other cases). Maybe that led me to assume  =



 things.


Hmmm, I hadn't seen it do this. But the whole point of DI files is to  =

create an interface to compiled libraries, not be the absolute fastest i=
t  =

can be. It is generally accepted that you are going to take a minuscule =
 =

(and on modern machines it is truly minuscule) hit every time you cross =
a  =

library boundary, unless you import the whole source of said library,  =

which is functionally what DI generation does now.


 AFAIK some C++ compilers use heuristics (don't really know what that i=


s  =


 exactly) to decide what "small functions" deserved automatic inlining.=



Those heuristics are in the semantic analysis step and I really don't  =

think building the DI file against the constantly changing AST during  =

semantic analysis is a wise idea. DI's get built before that, and as suc=
h  =

do no have access to those heuristics. If the compiler is doing somethin=
g  =

it's based on its own special, and limited analysis prior to the semanti=
c  =

analysis (I think I know which function is doing this, but it doesn't wo=
rk  =

very well at all). Also, trying to build DI's during semantic analysis  =

would most likely drastically slow down the compiler as the DI's would  =

need to be constantly updated to make the AST changes. Basically, it's a=
n  =

incredible amount of work for a minuscule speed improvement in the final=
  =

step. There are areas of D that are MUCH more ripe for speed improvement=
s  =

than inlining in DI files ... *cough*GC*cough*


 And well, C++ library builders are aware of that and usually keep the =


 =


 small functions they want inlined in the header file (e.g. in the clas=


s  =


 declaration). I can see a lot of cases where D libraries would benefit=


  =


 from allowing some of their functions to be inlined. Think of  =



  properties, they're often pretty short.


Well, that's where the do-it-yourself part into play. This is an automat=
ed  =

system for generating headers based on a naive AST. It will not be  =

perfect. But C++ doesn't even have that and they have to hand maintain  =

everything, so pretty much anything we do is an improvement. :-)
 properties are a valid concern, and also one that can be addressed in t=
he  =

current method of DI generation as they are a valid token that can be  =

checked for and excluded. If the community wishes for  properties to  =

retain their implementations that would an easy thing to do. But be  =

warned, you wont be able to call private functions as those are dropped =
 =

COMPLETELY from the DI file. Private data will of course be accessible.


 Other than this detail, the improvement in DI generation is very welco=


me.

-- =

Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Mar 14 2012