www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Visual D Build + DMD Bugginess = Bad

reply dsimcha <dsimcha yahoo.com> writes:
I've noticed that, when passed multiple files at once, DMD is generally buggy
in ways that I can't reproduce in small test cases.  This includes things like
magically ignoring __gshared variables, and not being able to convert a type
to an alias to the same type, for example, not being able to convert a size_t
to a uint on 32-bit, or not being able to convert a float[] to an R where R is
a template parameter instantiated as float[].

I have two questions:

1.  Has anyone experienced similar things and if so are they further along
than me at creating decent test cases?

2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
seems to work a lot better for the most part.  However, by default it seems to
pass all project files to the compiler at once, where Code::Blocks compiles
one file at a time.  This means I have to deal with DMD's bugginess when
passed mutliple files at once.  Does anyone know how to make Visual D work
Code::Blocks style, i.e. compile each file to an object file and link them
afterwords?
Oct 13 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"dsimcha" <dsimcha yahoo.com> wrote in message 
news:i9529m$2d4b$1 digitalmars.com...

 I've noticed that, when passed multiple files at once, DMD is generally 
 buggy
 in ways that I can't reproduce in small test cases.  This includes things 
 like
 magically ignoring __gshared variables, and not being able to convert a 
 type
 to an alias to the same type, for example, not being able to convert a 
 size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where 
 R is
 a template parameter instantiated as float[].

 I have two questions:

 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?

 2.  I'm used to using Code::Blocks, but want to switch to Visual D because 
 it
 seems to work a lot better for the most part.  However, by default it 
 seems to
 pass all project files to the compiler at once, where Code::Blocks 
 compiles
 one file at a time.  This means I have to deal with DMD's bugginess when
 passed mutliple files at once.  Does anyone know how to make Visual D work
 Code::Blocks style, i.e. compile each file to an object file and link them
 afterwords?


Don't they support setting up custom tools? I always do that and just define 
my own command line calls.
Oct 13 2010
prev sibling next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
dsimcha wrote:

 I've noticed that, when passed multiple files at once, DMD is generally buggy
 in ways that I can't reproduce in small test cases.  This includes things like
 magically ignoring __gshared variables, and not being able to convert a type
 to an alias to the same type, for example, not being able to convert a size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where R is
 a template parameter instantiated as float[].
 
 I have two questions:
 
 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?


I haven't run into something similar lately, but what you describe 
sounds like issues with forward references, maybe in combination with 
circular imports. Building multiple files at once can change the point 
where you enter the vicious import circle.

Could it be related to any of these?

http://d.puremagic.com/issues/show_bug.cgi?id=190
http://d.puremagic.com/issues/show_bug.cgi?id=3979


 
 2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
 seems to work a lot better for the most part.  However, by default it seems to
 pass all project files to the compiler at once, where Code::Blocks compiles
 one file at a time.  This means I have to deal with DMD's bugginess when
 passed mutliple files at once.  Does anyone know how to make Visual D work
 Code::Blocks style, i.e. compile each file to an object file and link them
 afterwords?


I have that option on my todo list, but didn't implement it so far 
because I was not aware of any problems with compiling multiple files.

I don't know if this is really an option for a larger project, but what 
you can do is select "Custom Build Tool" for all files in the "Common 
Properties" setting of the project property dialog and enter your own 
command line and output file. But be warned: you will not get any 
automatic dependency detection.

As editing the settings for a lot of files might be very annoying, you 
might want to set it for one file and then edit the project file with a 
text editor to copy the settings to other files.

Rainer
Oct 13 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Rainer Schuetze (r.sagitario gmx.de)'s article

 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979


Yep, these look like at least part of the problem.  Glad someone else has
already
reduced them to sane test cases so I don't have to.  Reducing compiler bugs that
only seem to occur in non-trivial, multi-module projects to decent test cases
is a
huge PITA.

Anyhow, the project in question is a very messy codebase because it grew very
organically.  It's basically a haphazard collection of research prototype
algorithms for predicting gene expression from DNA sequence, and every time I
think of a new idea, I tend to just put it wherever I can make it fit and almost
never bother with non-trivial refactoring.  Therefore, there are cyclic imports
**everywhere**.


 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer


Since build process automation is by far the biggest reason why I use an IDE
instead of a plain old editor, I'd rather just stick with Code::Blocks for now.
Could you please bump this up the todo list, given that building multiple files
simultaneously is buggy in ways that probably aren't going to get fixed too
soon?
Oct 13 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"dsimcha" <dsimcha yahoo.com> wrote in message 
news:i957sg$2otm$1 digitalmars.com...

 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article

 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer


 Since build process automation is by far the biggest reason why I use an 
 IDE
 instead of a plain old editor, I'd rather just stick with Code::Blocks for 
 now.
 Could you please bump this up the todo list, given that building multiple 
 files
 simultaneously is buggy in ways that probably aren't going to get fixed 
 too soon?


I find that rdmd works just as well as any IDE (if not better) for 
automating the build process of D programs. Particularly if you apply the 
"Combined" patch here: http://d.puremagic.com/issues/show_bug.cgi?id=4930

Although it doesn't do one-at-a-time building ATM, so I guess that wouldn't 
help in your case.
Oct 13 2010
prev sibling next sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
I'll push the issue up the todo list. Maybe I'm not getting a lot of 
these errors because I have applied the dmd patches in the bug reports 
190 and 4753...


dsimcha wrote:

 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article

 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979


 
 Yep, these look like at least part of the problem.  Glad someone else has
already
 reduced them to sane test cases so I don't have to.  Reducing compiler bugs
that
 only seem to occur in non-trivial, multi-module projects to decent test cases
is a
 huge PITA.
 
 Anyhow, the project in question is a very messy codebase because it grew very
 organically.  It's basically a haphazard collection of research prototype
 algorithms for predicting gene expression from DNA sequence, and every time I
 think of a new idea, I tend to just put it wherever I can make it fit and
almost
 never bother with non-trivial refactoring.  Therefore, there are cyclic imports
 **everywhere**.
 

 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer


 
 Since build process automation is by far the biggest reason why I use an IDE
 instead of a plain old editor, I'd rather just stick with Code::Blocks for now.
 Could you please bump this up the todo list, given that building multiple files
 simultaneously is buggy in ways that probably aren't going to get fixed too
soon?
Oct 14 2010
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Here's a preliminary new version that includes an option on the general 
project configuration page to switch to single file compilation:

http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe

I've used ddmd as a medium sized project for testing. It has more than 
400 files with most modules mutually importing each other. Some remarks:

- compilation speed is a lot worse, even when compared to what you would 
expect from a similar C/C++ project. It takes almost 8 minutes to 
compile ddmd on my system with single file compilation, while the 
standard compilation takes about 10 seconds.

- the dependency files written by dmd are so bloated that the check that 
selects the files to compile takes almost 20 seconds. This is because 
the size of each dependency file is about 700 kB, summing up to 220 MB 
to scan for imported files. (I had to do some optimizations and file 
operation caching to get the time down from several minutes).

- I consider the single compilation mode only sensible for a large 
collection of modules that have only limited dependencies.

- probably, compilation speed could be way better if semantic analysis 
would only be done on symbols that are actually directly or indirectly 
referred to by the modules passed on the command line. This would need a 
slight modification of the semantic analysis, but would also put the 
promise of independence of lexical order to the test - and would expose 
more of the problems you are currently getting.

This new version also includes some experimental debug/version statement 
highlighting, which has not yet been extensively tested. If it causes 
too many troubles, it can be disabled on the global property page 
Tools->Options->TextEditor->D->Colorizer.

Rainer


dsimcha wrote:

 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article

 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979


 
 Yep, these look like at least part of the problem.  Glad someone else has
already
 reduced them to sane test cases so I don't have to.  Reducing compiler bugs
that
 only seem to occur in non-trivial, multi-module projects to decent test cases
is a
 huge PITA.
 
 Anyhow, the project in question is a very messy codebase because it grew very
 organically.  It's basically a haphazard collection of research prototype
 algorithms for predicting gene expression from DNA sequence, and every time I
 think of a new idea, I tend to just put it wherever I can make it fit and
almost
 never bother with non-trivial refactoring.  Therefore, there are cyclic imports
 **everywhere**.
 

 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer


 
 Since build process automation is by far the biggest reason why I use an IDE
 instead of a plain old editor, I'd rather just stick with Code::Blocks for now.
 Could you please bump this up the todo list, given that building multiple files
 simultaneously is buggy in ways that probably aren't going to get fixed too
soon?
Oct 17 2010
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
with newer versions of VisualD (mago?). Do I need to set it up to be
used in VS over the default one or is it selected by default after
installation?

On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:

 Here's a preliminary new version that includes an option on the general
 project configuration page to switch to single file compilation:

 http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe

 I've used ddmd as a medium sized project for testing. It has more than
 400 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would
 expect from a similar C/C++ project. It takes almost 8 minutes to
 compile ddmd on my system with single file compilation, while the
 standard compilation takes about 10 seconds.

 - the dependency files written by dmd are so bloated that the check that
 selects the files to compile takes almost 20 seconds. This is because
 the size of each dependency file is about 700 kB, summing up to 220 MB
 to scan for imported files. (I had to do some optimizations and file
 operation caching to get the time down from several minutes).

 - I consider the single compilation mode only sensible for a large
 collection of modules that have only limited dependencies.

 - probably, compilation speed could be way better if semantic analysis
 would only be done on symbols that are actually directly or indirectly
 referred to by the modules passed on the command line. This would need a
 slight modification of the semantic analysis, but would also put the
 promise of independence of lexical order to the test - and would expose
 more of the problems you are currently getting.

 This new version also includes some experimental debug/version statement
 highlighting, which has not yet been extensively tested. If it causes
 too many troubles, it can be disabled on the global property page
 Tools->Options->TextEditor->D->Colorizer.

 Rainer


 dsimcha wrote:

 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article

 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979


 Yep, these look like at least part of the problem.  Glad someone else has
 already
 reduced them to sane test cases so I don't have to.  Reducing compiler
 bugs that
 only seem to occur in non-trivial, multi-module projects to decent test
 cases is a
 huge PITA.

 Anyhow, the project in question is a very messy codebase because it grew
 very
 organically.  It's basically a haphazard collection of research prototype
 algorithms for predicting gene expression from DNA sequence, and every
 time I
 think of a new idea, I tend to just put it wherever I can make it fit and
 almost
 never bother with non-trivial refactoring.  Therefore, there are cyclic
 imports
 **everywhere**.


 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer


 Since build process automation is by far the biggest reason why I use an
 IDE
 instead of a plain old editor, I'd rather just stick with Code::Blocks for
 now.
 Could you please bump this up the todo list, given that building multiple
 files
 simultaneously is buggy in ways that probably aren't going to get fixed
 too soon?
Oct 17 2010
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
Andrej Mitrovic wrote:

 Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
 with newer versions of VisualD (mago?). Do I need to set it up to be
 used in VS over the default one or is it selected by default after
 installation?
 


If you did not uncheck the mago debugger when installing, you can simply 
select the debug engine to be used on the project property page 
"Debugging" (visual studio internal debugger or mago).
Oct 17 2010
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I've just tried it. It doesn't seem to be able to find any locals,
even if I add a variable to a watchlist it can't find the symbol. I
guess this is still experimental. Anyway, thanks for the info.

On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:

 Andrej Mitrovic wrote:

 Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
 with newer versions of VisualD (mago?). Do I need to set it up to be
 used in VS over the default one or is it selected by default after
 installation?


 If you did not uncheck the mago debugger when installing, you can simply
 select the debug engine to be used on the project property page
 "Debugging" (visual studio internal debugger or mago).
Oct 17 2010
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
That's an unfortunate regression in the linker that comes with dmd 
2.049. There's already an updated linker available 
(http://ftp.digitalmars.com/link.8.00.8.zip). See also the project page 
http://dsource.org/projects/mago_debugger for more info.

Andrej Mitrovic wrote:

 I've just tried it. It doesn't seem to be able to find any locals,
 even if I add a variable to a watchlist it can't find the symbol. I
 guess this is still experimental. Anyway, thanks for the info.
 
 On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:

 Andrej Mitrovic wrote:

 Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
 with newer versions of VisualD (mago?). Do I need to set it up to be
 used in VS over the default one or is it selected by default after
 installation?


 If you did not uncheck the mago debugger when installing, you can simply
 select the debug engine to be used on the project property page
 "Debugging" (visual studio internal debugger or mago).
Oct 17 2010
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Ah, ok. I'll have a look, thanks.

On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:

 That's an unfortunate regression in the linker that comes with dmd
 2.049. There's already an updated linker available
 (http://ftp.digitalmars.com/link.8.00.8.zip). See also the project page
 http://dsource.org/projects/mago_debugger for more info.

 Andrej Mitrovic wrote:

 I've just tried it. It doesn't seem to be able to find any locals,
 even if I add a variable to a watchlist it can't find the symbol. I
 guess this is still experimental. Anyway, thanks for the info.

 On 10/17/10, Rainer Schuetze <r.sagitario gmx.de> wrote:

 Andrej Mitrovic wrote:

 Cool stuff, Rainer. Btw, I've noticed there's a new debugger installed
 with newer versions of VisualD (mago?). Do I need to set it up to be
 used in VS over the default one or is it selected by default after
 installation?


 If you did not uncheck the mago debugger when installing, you can simply
 select the debug engine to be used on the project property page
 "Debugging" (visual studio internal debugger or mago).
Oct 17 2010
prev sibling next sibling parent dsimcha <dsimcha yahoo.com> writes:
Awesome.  I frankly don't care about compile speeds because the projects in
question are fairly small, so even with a horribly inefficient build process
build
times are negligible.  FWIW I use the one at a time builds in CodeBlocks, too,
though I parallelize them on my quad core.

I only wanted the one at a time compilation feature as a temporary workaround
for
compiler bugs that will hopefully be fixed by the time people start using D2 for
larger projects.  I'll try Visual D again tomorrow and provide some feedback.
From my limited playing around with it, I've already noticed that the debugger
works a ton better than any other D debugger I've tried, i.e. possibly good
enough
that I'll stop using writeln as my main debugging method.

== Quote from Rainer Schuetze (r.sagitario gmx.de)'s article

 Here's a preliminary new version that includes an option on the general
 project configuration page to switch to single file compilation:
 http://www.dsource.org/projects/visuald/browser/downloads/VisualD-v0.3.18rc1.exe
 I've used ddmd as a medium sized project for testing. It has more than
 400 files with most modules mutually importing each other. Some remarks:
 - compilation speed is a lot worse, even when compared to what you would
 expect from a similar C/C++ project. It takes almost 8 minutes to
 compile ddmd on my system with single file compilation, while the
 standard compilation takes about 10 seconds.
 - the dependency files written by dmd are so bloated that the check that
 selects the files to compile takes almost 20 seconds. This is because
 the size of each dependency file is about 700 kB, summing up to 220 MB
 to scan for imported files. (I had to do some optimizations and file
 operation caching to get the time down from several minutes).
 - I consider the single compilation mode only sensible for a large
 collection of modules that have only limited dependencies.
 - probably, compilation speed could be way better if semantic analysis
 would only be done on symbols that are actually directly or indirectly
 referred to by the modules passed on the command line. This would need a
 slight modification of the semantic analysis, but would also put the
 promise of independence of lexical order to the test - and would expose
 more of the problems you are currently getting.
 This new version also includes some experimental debug/version statement
 highlighting, which has not yet been extensively tested. If it causes
 too many troubles, it can be disabled on the global property page
 Tools->Options->TextEditor->D->Colorizer.
 Rainer
 dsimcha wrote:

 == Quote from Rainer Schuetze (r.sagitario gmx.de)'s article

 Could it be related to any of these?
 http://d.puremagic.com/issues/show_bug.cgi?id=190
 http://d.puremagic.com/issues/show_bug.cgi?id=3979


 Yep, these look like at least part of the problem.  Glad someone else has
already
 reduced them to sane test cases so I don't have to.  Reducing compiler bugs
that
 only seem to occur in non-trivial, multi-module projects to decent test cases
is a
 huge PITA.

 Anyhow, the project in question is a very messy codebase because it grew very
 organically.  It's basically a haphazard collection of research prototype
 algorithms for predicting gene expression from DNA sequence, and every time I
 think of a new idea, I tend to just put it wherever I can make it fit and
almost
 never bother with non-trivial refactoring.  Therefore, there are cyclic imports
 **everywhere**.


 I have that option on my todo list, but didn't implement it so far
 because I was not aware of any problems with compiling multiple files.
 I don't know if this is really an option for a larger project, but what
 you can do is select "Custom Build Tool" for all files in the "Common
 Properties" setting of the project property dialog and enter your own
 command line and output file. But be warned: you will not get any
 automatic dependency detection.
 As editing the settings for a lot of files might be very annoying, you
 might want to set it for one file and then edit the project file with a
 text editor to copy the settings to other files.
 Rainer


 Since build process automation is by far the biggest reason why I use an IDE
 instead of a plain old editor, I'd rather just stick with Code::Blocks for now.
 Could you please bump this up the todo list, given that building multiple files
 simultaneously is buggy in ways that probably aren't going to get fixed too
soon?
Oct 17 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Rainer Schuetze" <r.sagitario gmx.de> wrote in message 
news:i9f2ce$30re$1 digitalmars.com...

 I've used ddmd as a medium sized project for testing. It has more than 400 
 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would 
 expect from a similar C/C++ project. It takes almost 8 minutes to compile 
 ddmd on my system with single file compilation, while the standard 
 compilation takes about 10 seconds.


Whaa...? Standard compilation of ddmd takes me a full two minutes.
Oct 18 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:i9ibi0$ddn$1 digitalmars.com...

 "Rainer Schuetze" <r.sagitario gmx.de> wrote in message 
 news:i9f2ce$30re$1 digitalmars.com...

 I've used ddmd as a medium sized project for testing. It has more than 
 400 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would 
 expect from a similar C/C++ project. It takes almost 8 minutes to compile 
 ddmd on my system with single file compilation, while the standard 
 compilation takes about 10 seconds.


 Whaa...? Standard compilation of ddmd takes me a full two minutes.


As a sidebar, it's the *only* D program I've compiled that's ever taken more 
than a few seconds for me.
Oct 18 2010
prev sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 19 Oct 2010 00:41:54 +0400, Nick Sabalausky <a a.a> wrote:


 "Rainer Schuetze" <r.sagitario gmx.de> wrote in message
 news:i9f2ce$30re$1 digitalmars.com...

 I've used ddmd as a medium sized project for testing. It has more than  
 400
 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would
 expect from a similar C/C++ project. It takes almost 8 minutes to  
 compile
 ddmd on my system with single file compilation, while the standard
 compilation takes about 10 seconds.


 Whaa...? Standard compilation of ddmd takes me a full two minutes.


Do you compile the code on 486 or something? :) Running the build script  
takes 7.967 seconds for me (timed with timeit, part of the Windows Server  
2003 Resource Kit Tools).
Oct 18 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Denis Koroskin" <2korden gmail.com> wrote in message 
news:op.vkshh4bdo7cclz korden-pc...

 On Tue, 19 Oct 2010 00:41:54 +0400, Nick Sabalausky <a a.a> wrote:


 "Rainer Schuetze" <r.sagitario gmx.de> wrote in message
 news:i9f2ce$30re$1 digitalmars.com...

 I've used ddmd as a medium sized project for testing. It has more than 
 400
 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you would
 expect from a similar C/C++ project. It takes almost 8 minutes to 
 compile
 ddmd on my system with single file compilation, while the standard
 compilation takes about 10 seconds.


 Whaa...? Standard compilation of ddmd takes me a full two minutes.


 Do you compile the code on 486 or something? :)


Heh :)  Compared to your and Rainer's systems it probably might as well be.


 Running the build script  takes 7.967 seconds for me (timed with timeit, 
 part of the Windows Server  2003 Resource Kit Tools).


My guess is it may be the multi-threaded "process multiple modules in 
parallel" optimization. DDMD does have a *lot* of source files. And I'm on a 
single-core (yea, yea, I know, but it works for me and I have no money).

I'm curious, is it possible for you or Rainer to time it running dmd in a 
"force single-core" mode? Ideally using the main system/OS core (if there 
even is one)? I would expect a normal out-of-the-box dmd would be noticably 
faster for you two just because of that multi-thread/core optimization and 
the cores themselves just simply being faster, but 120 sec vs 8 sec still 
seems an excessive difference.

And then considering also, outside of dmd bugs, I've never once had any D 
program besides ddmd take more than a few seconds - despite the fact that 
much of what I do complie is fairly template/mixin/ctfe-heavy, and ddmd 
appears to use very little (if any) of that - it's certainly still a 
conceivable speed difference, but it does make me wonder if something else 
might be going on.

BTW, anyone know of an easy way to check how much time in a process is spent 
waiting on disk IO? It's entirely possible that my disk is just fragmented 
all to hell.
Oct 18 2010
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 19 Oct 2010 06:26:23 +0400, Nick Sabalausky <a a.a> wrote:


 "Denis Koroskin" <2korden gmail.com> wrote in message
 news:op.vkshh4bdo7cclz korden-pc...

 On Tue, 19 Oct 2010 00:41:54 +0400, Nick Sabalausky <a a.a> wrote:


 "Rainer Schuetze" <r.sagitario gmx.de> wrote in message
 news:i9f2ce$30re$1 digitalmars.com...

 I've used ddmd as a medium sized project for testing. It has more than
 400
 files with most modules mutually importing each other. Some remarks:

 - compilation speed is a lot worse, even when compared to what you  
 would
 expect from a similar C/C++ project. It takes almost 8 minutes to
 compile
 ddmd on my system with single file compilation, while the standard
 compilation takes about 10 seconds.


 Whaa...? Standard compilation of ddmd takes me a full two minutes.


 Do you compile the code on 486 or something? :)


 Heh :)  Compared to your and Rainer's systems it probably might as well  
 be.


 Running the build script  takes 7.967 seconds for me (timed with timeit,
 part of the Windows Server  2003 Resource Kit Tools).


 My guess is it may be the multi-threaded "process multiple modules in
 parallel" optimization. DDMD does have a *lot* of source files. And I'm  
 on a
 single-core (yea, yea, I know, but it works for me and I have no money).

 I'm curious, is it possible for you or Rainer to time it running dmd in a
 "force single-core" mode? Ideally using the main system/OS core (if there
 even is one)? I would expect a normal out-of-the-box dmd would be  
 noticably
 faster for you two just because of that multi-thread/core optimization  
 and
 the cores themselves just simply being faster, but 120 sec vs 8 sec still
 seems an excessive difference.

 And then considering also, outside of dmd bugs, I've never once had any D
 program besides ddmd take more than a few seconds - despite the fact that
 much of what I do complie is fairly template/mixin/ctfe-heavy, and ddmd
 appears to use very little (if any) of that - it's certainly still a
 conceivable speed difference, but it does make me wonder if something  
 else
 might be going on.

 BTW, anyone know of an easy way to check how much time in a process is  
 spent
 waiting on disk IO? It's entirely possible that my disk is just  
 fragmented
 all to hell.


dmd doesn't use multiple core for anything BUT asynchronous reading from  
disk. Everything else is done is a single thread. I don't think it has  
THAT big of impact, but I'll try to recompile with all the cores disabled.
Oct 18 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Denis Koroskin" <2korden gmail.com> wrote in message 
news:op.vksxyn15o7cclz korden-pc...

 On Tue, 19 Oct 2010 06:26:23 +0400, Nick Sabalausky <a a.a> wrote:


 "Denis Koroskin" <2korden gmail.com> wrote in message

 Running the build script  takes 7.967 seconds for me (timed with timeit,
 part of the Windows Server  2003 Resource Kit Tools).


 My guess is it may be the multi-threaded "process multiple modules in
 parallel" optimization. DDMD does have a *lot* of source files. And I'm 
 on a
 single-core (yea, yea, I know, but it works for me and I have no money).

 I'm curious, is it possible for you or Rainer to time it running dmd in a
 "force single-core" mode? Ideally using the main system/OS core (if there
 even is one)? I would expect a normal out-of-the-box dmd would be 
 noticably
 faster for you two just because of that multi-thread/core optimization 
 and
 the cores themselves just simply being faster, but 120 sec vs 8 sec still
 seems an excessive difference.

 And then considering also, outside of dmd bugs, I've never once had any D
 program besides ddmd take more than a few seconds - despite the fact that
 much of what I do complie is fairly template/mixin/ctfe-heavy, and ddmd
 appears to use very little (if any) of that - it's certainly still a
 conceivable speed difference, but it does make me wonder if something 
 else
 might be going on.

 BTW, anyone know of an easy way to check how much time in a process is 
 spent
 waiting on disk IO? It's entirely possible that my disk is just 
 fragmented
 all to hell.


 dmd doesn't use multiple core for anything BUT asynchronous reading from 
 disk. Everything else is done is a single thread.


Ahh, I see. So much for that theory then.


 I don't think it has  THAT big of impact, but I'll try to recompile with 
 all the cores disabled.


Thanks, that'll be interesting.

I did go ahead and re-time the compile. Apparently I must have been 
remembering it wrong, because this time it only took 1 min 20 sec (and this 
was with a ton of stuff running - bunch of misc apps, FF2 with 20 tabs, an 
HDD SMART monitor, a torrent manager and a bunch of other servers (but no 
clients connected)). But that's still quite a lot of time to compile a D 
app, even for my machine.

I also tried grabbing the latest ddmd, rebooted, killed all non-essential 
processes, and tried that way. Got it down to just slightly under one 
minute.

I thought about maybe it being a limited-memory issue (remembering that dmd 
never frees anything until it's done - or is that just CTFE?), but I don't 
think that's it - the highest memory usage it ever got was about 200MB, and 
I have 1GB, and it still took a whole minute with almost everything besides 
XP shut down, so I'm not sure that was it. (I could have sworn I had 2GB at 
one point, but I think I probably cannibalized one of the sticks when I 
built my linux box - not that that's really relevant ;) )

FWIW, this is all with compiling the just the debug version of ddmd only. 
Ie, not including building the release version or the one-time initial setup 
of building dmd.lib.
Oct 18 2010
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 19 Oct 2010 07:53:45 +0400, Nick Sabalausky <a a.a> wrote:


 "Denis Koroskin" <2korden gmail.com> wrote in message
 news:op.vksxyn15o7cclz korden-pc...

 On Tue, 19 Oct 2010 06:26:23 +0400, Nick Sabalausky <a a.a> wrote:


 "Denis Koroskin" <2korden gmail.com> wrote in message

 Running the build script  takes 7.967 seconds for me (timed with  
 timeit,
 part of the Windows Server  2003 Resource Kit Tools).


 My guess is it may be the multi-threaded "process multiple modules in
 parallel" optimization. DDMD does have a *lot* of source files. And I'm
 on a
 single-core (yea, yea, I know, but it works for me and I have no  
 money).

 I'm curious, is it possible for you or Rainer to time it running dmd  
 in a
 "force single-core" mode? Ideally using the main system/OS core (if  
 there
 even is one)? I would expect a normal out-of-the-box dmd would be
 noticably
 faster for you two just because of that multi-thread/core optimization
 and
 the cores themselves just simply being faster, but 120 sec vs 8 sec  
 still
 seems an excessive difference.

 And then considering also, outside of dmd bugs, I've never once had  
 any D
 program besides ddmd take more than a few seconds - despite the fact  
 that
 much of what I do complie is fairly template/mixin/ctfe-heavy, and ddmd
 appears to use very little (if any) of that - it's certainly still a
 conceivable speed difference, but it does make me wonder if something
 else
 might be going on.

 BTW, anyone know of an easy way to check how much time in a process is
 spent
 waiting on disk IO? It's entirely possible that my disk is just
 fragmented
 all to hell.


 dmd doesn't use multiple core for anything BUT asynchronous reading from
 disk. Everything else is done is a single thread.


 Ahh, I see. So much for that theory then.


 I don't think it has  THAT big of impact, but I'll try to recompile with
 all the cores disabled.


 Thanks, that'll be interesting.

 I did go ahead and re-time the compile. Apparently I must have been
 remembering it wrong, because this time it only took 1 min 20 sec (and  
 this
 was with a ton of stuff running - bunch of misc apps, FF2 with 20 tabs,  
 an
 HDD SMART monitor, a torrent manager and a bunch of other servers (but no
 clients connected)). But that's still quite a lot of time to compile a D
 app, even for my machine.

 I also tried grabbing the latest ddmd, rebooted, killed all non-essential
 processes, and tried that way. Got it down to just slightly under one
 minute.

 I thought about maybe it being a limited-memory issue (remembering that  
 dmd
 never frees anything until it's done - or is that just CTFE?), but I  
 don't
 think that's it - the highest memory usage it ever got was about 200MB,  
 and
 I have 1GB, and it still took a whole minute with almost everything  
 besides
 XP shut down, so I'm not sure that was it. (I could have sworn I had 2GB  
 at
 one point, but I think I probably cannibalized one of the sticks when I
 built my linux box - not that that's really relevant ;) )

 FWIW, this is all with compiling the just the debug version of ddmd only.
 Ie, not including building the release version or the one-time initial  
 setup
 of building dmd.lib.


It's 8.6 seconds for a single cores, 8.3s for all 4 cores (Core2 Quad  
Q8300   2.5Ghz, Windows).
Oct 18 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Denis Koroskin" <2korden gmail.com> wrote in message 
news:op.vks9nlljo7cclz korden-pc...

 On Tue, 19 Oct 2010 07:53:45 +0400, Nick Sabalausky <a a.a> wrote:


 "Denis Koroskin" <2korden gmail.com> wrote in message
 news:op.vksxyn15o7cclz korden-pc...

 I don't think it has  THAT big of impact, but I'll try to recompile with
 all the cores disabled.


 Thanks, that'll be interesting.

 I did go ahead and re-time the compile. Apparently I must have been
 remembering it wrong, because this time it only took 1 min 20 sec (and 
 this
 was with a ton of stuff running - bunch of misc apps, FF2 with 20 tabs, 
 an
 HDD SMART monitor, a torrent manager and a bunch of other servers (but no
 clients connected)). But that's still quite a lot of time to compile a D
 app, even for my machine.

 I also tried grabbing the latest ddmd, rebooted, killed all non-essential
 processes, and tried that way. Got it down to just slightly under one
 minute.

 I thought about maybe it being a limited-memory issue (remembering that 
 dmd
 never frees anything until it's done - or is that just CTFE?), but I 
 don't
 think that's it - the highest memory usage it ever got was about 200MB, 
 and
 I have 1GB, and it still took a whole minute with almost everything 
 besides
 XP shut down, so I'm not sure that was it. (I could have sworn I had 2GB 
 at
 one point, but I think I probably cannibalized one of the sticks when I
 built my linux box - not that that's really relevant ;) )

 FWIW, this is all with compiling the just the debug version of ddmd only.
 Ie, not including building the release version or the one-time initial 
 setup
 of building dmd.lib.


 It's 8.6 seconds for a single cores, 8.3s for all 4 cores (Core2 Quad 
 Q8300   2.5Ghz, Windows).


Hmm... that does make my situation seem odd then. Mine's a 1.7 GHz Celeron, 
so a little more than half the clock speed of yours. Of course, I'm well 
aware that my older architecture and less cache make mine slower than 
(2.5/1.7) of your speed, but that still seems like a strangely large 
difference.

I wonder if maybe RAM speed could account for it.
Oct 19 2010
parent "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:i9jr29$6af$1 digitalmars.com...

 "Denis Koroskin" <2korden gmail.com> wrote in message 
 news:op.vks9nlljo7cclz korden-pc...

 On Tue, 19 Oct 2010 07:53:45 +0400, Nick Sabalausky <a a.a> wrote:


 "Denis Koroskin" <2korden gmail.com> wrote in message
 news:op.vksxyn15o7cclz korden-pc...

 I don't think it has  THAT big of impact, but I'll try to recompile 
 with
 all the cores disabled.


 Thanks, that'll be interesting.

 I did go ahead and re-time the compile. Apparently I must have been
 remembering it wrong, because this time it only took 1 min 20 sec (and 
 this
 was with a ton of stuff running - bunch of misc apps, FF2 with 20 tabs, 
 an
 HDD SMART monitor, a torrent manager and a bunch of other servers (but 
 no
 clients connected)). But that's still quite a lot of time to compile a D
 app, even for my machine.

 I also tried grabbing the latest ddmd, rebooted, killed all 
 non-essential
 processes, and tried that way. Got it down to just slightly under one
 minute.

 I thought about maybe it being a limited-memory issue (remembering that 
 dmd
 never frees anything until it's done - or is that just CTFE?), but I 
 don't
 think that's it - the highest memory usage it ever got was about 200MB, 
 and
 I have 1GB, and it still took a whole minute with almost everything 
 besides
 XP shut down, so I'm not sure that was it. (I could have sworn I had 2GB 
 at
 one point, but I think I probably cannibalized one of the sticks when I
 built my linux box - not that that's really relevant ;) )

 FWIW, this is all with compiling the just the debug version of ddmd 
 only.
 Ie, not including building the release version or the one-time initial 
 setup
 of building dmd.lib.


 It's 8.6 seconds for a single cores, 8.3s for all 4 cores (Core2 Quad 
 Q8300   2.5Ghz, Windows).


 Hmm... that does make my situation seem odd then. Mine's a 1.7 GHz 
 Celeron, so a little more than half the clock speed of yours. Of course, 
 I'm well aware that my older architecture and less cache make mine slower 
 than (2.5/1.7) of your speed, but that still seems like a strangely large 
 difference.

 I wonder if maybe RAM speed could account for it.


s/(2.5/1.7)/(1.7/2.5)

Too late in the day for math...
Oct 19 2010
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
dsimcha wrote:

 I've noticed that, when passed multiple files at once, DMD is generally buggy
 in ways that I can't reproduce in small test cases.  This includes things like
 magically ignoring __gshared variables, and not being able to convert a type
 to an alias to the same type, for example, not being able to convert a size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where R is
 a template parameter instantiated as float[].
 
 I have two questions:
 
 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?


Phobos and Druntime are built by passing all the files to dmd at once, and it 
works without problems. No attempt was made to tweak the order the files were 
presented to dmd to make it work, I just aggregated them all.
Oct 13 2010
parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
On Wed, 13 Oct 2010 14:15:00 -0700, Walter Bright wrote:


 dsimcha wrote:

 I've noticed that, when passed multiple files at once, DMD is generally
 buggy in ways that I can't reproduce in small test cases.  This
 includes things like magically ignoring __gshared variables, and not
 being able to convert a type to an alias to the same type, for example,
 not being able to convert a size_t to a uint on 32-bit, or not being
 able to convert a float[] to an R where R is a template parameter
 instantiated as float[].
 
 I have two questions:
 
 1.  Has anyone experienced similar things and if so are they further
 along than me at creating decent test cases?


 
 Phobos and Druntime are built by passing all the files to dmd at once,
 and it works without problems. No attempt was made to tweak the order
 the files were presented to dmd to make it work, I just aggregated them
 all.


It's not quite true that it works without problems.  I reported bug 3979 
because I was unable to include the new std.process in Phobos without 
changing the order of the files passed to DMD.

-Lars
Oct 13 2010
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
I'm also having problems with VisualD + DMD, I made a posting in the 
bugs forums about it, but unfortunately no one did answer yet:

http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D.bugs&artnum=25536

Am 13.10.2010 21:45, schrieb dsimcha:

 I've noticed that, when passed multiple files at once, DMD is generally buggy
 in ways that I can't reproduce in small test cases.  This includes things like
 magically ignoring __gshared variables, and not being able to convert a type
 to an alias to the same type, for example, not being able to convert a size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where R is
 a template parameter instantiated as float[].

 I have two questions:

 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?

 2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
 seems to work a lot better for the most part.  However, by default it seems to
 pass all project files to the compiler at once, where Code::Blocks compiles
 one file at a time.  This means I have to deal with DMD's bugginess when
 passed mutliple files at once.  Does anyone know how to make Visual D work
 Code::Blocks style, i.e. compile each file to an object file and link them
 afterwords?



-- 
Kind Regards
Benjamin Thaut
Oct 13 2010
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday 13 October 2010 23:28:40 Benjamin Thaut wrote:

 I'm also having problems with VisualD + DMD, I made a posting in the
 bugs forums about it, but unfortunately no one did answer yet:
 
 http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group
 =digitalmars.D.bugs&artnum=25536


The bugs forum is not for posting to. The bugtracker sends information on the 
creation of and changes to bug reports. You sign up for the list if you want to 
see those messages. Pretty much no one is going to pay attention to anything 
that you post to the bug list directly. Typically, you'd post to the D list or 
the D-Learn list (D being for general stuff and Learn for questions about how
the 
language works and anything related to learning D).

- Jonathan M Davis
Oct 13 2010
prev sibling next sibling parent klickverbot <see klickverbot.at> writes:
On 10/14/10 8:28 AM, Benjamin Thaut wrote:

 I'm also having problems with VisualD + DMD, I made a posting in the
 bugs forums about it, but unfortunately no one did answer yet:


Uh, afaik the digitalmars.D.bugs forum just mirrors the changes from the 
puremagic D bugzilla, and thus is not quite the best place for discussions.
Oct 14 2010
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday 13 October 2010 23:58:16 Jonathan M Davis wrote:

 The bugs forum is not for posting to.


Ack. I meant the bug mailing list, not forum. Oh well, it should be fairly 
obvious what I meant.

- Jonathan M Davis
Oct 14 2010
prev sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <ludwig informatik.uni-luebeck.de> writes:
Am 13.10.2010 21:45, schrieb dsimcha:

 I've noticed that, when passed multiple files at once, DMD is generally buggy
 in ways that I can't reproduce in small test cases.  This includes things like
 magically ignoring __gshared variables, and not being able to convert a type
 to an alias to the same type, for example, not being able to convert a size_t
 to a uint on 32-bit, or not being able to convert a float[] to an R where R is
 a template parameter instantiated as float[].

 I have two questions:

 1.  Has anyone experienced similar things and if so are they further along
 than me at creating decent test cases?

 2.  I'm used to using Code::Blocks, but want to switch to Visual D because it
 seems to work a lot better for the most part.  However, by default it seems to
 pass all project files to the compiler at once, where Code::Blocks compiles
 one file at a time.  This means I have to deal with DMD's bugginess when
 passed mutliple files at once.  Does anyone know how to make Visual D work
 Code::Blocks style, i.e. compile each file to an object file and link them
 afterwords?


Right now, I'm in exactly the same situation. However, the problems 
might be a bit more obvious: Sometimes symbols are missing at link time 
that should not and sometimes some symbols (most of the time phobos 
symbols) are defined twice but with exactly the same module path.

(Error: multiple definition of conv_208_199f: 
_D53C:\D\dmd2\windows\bin\..\..\src\phobos\std\conv.d.52012__ModuleInfoZ 
and conv: 
_D53C:\D\dmd2\windows\bin\..\..\src\phobos\std\conv.d.52012__ModuleInfoZ)

On Mac OS, using all-at-once compilation I get defective .lib files (the 
ar tool complains that the .o file is smaller than it reports or 
something similar) so I have to compile the code of all static libs 
together with the application code.

The problem is that it is really difficult (almost impossible) to 
isolate the problem in any way because sometimes appears erratically and 
the code base is quite large. And also I cannot share the source code 
for debugging.
Oct 14 2010