www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D2 port of xfBuild (alpha version)

reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
xfBuild is a D1 & D2 build tool, it was written for D1 and Tango by
Tomasz Stachowiak. I've ported it to D2 and Phobos, which will
hopefully enable it to grow thanks to the fairly active D2 community.

Grab it from:
https://github.com/AndrejMitrovic/xfBuild

Currently it compiles simple projects, and was tested on XP SP3 and
Lubuntu. It also appears some multithreaded bugs have disappeared
since I'm now using the new std.parallelism module.

*Note*: This is an alpha-release. I'm pretty sure there are many
lurking bugs ready to be discovered. Don't try to use it on production
code!

Why did I do this? Because I have time to burn. :-) (also I've heard
Tomasz used his tool on very large codebases, so it's a project worth
porting)
Nov 23 2011
next sibling parent reply Gour <gour atmarama.net> writes:
On Thu, 24 Nov 2011 04:44:44 +0100
Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:


 Grab it from:
 https://github.com/AndrejMitrovic/xfBuild


Thank you. I just wanted to ask about recommended tool to build D
projects. ;)


 *Note*: This is an alpha-release. I'm pretty sure there are many
 lurking bugs ready to be discovered. Don't try to use it on production
 code!


We don't have it (yet) :-D


 Why did I do this? Because I have time to burn. :-) (also I've heard
 Tomasz used his tool on very large codebases, so it's a project worth
 porting)


Excuse my ignorance, but I wonder if xfbuild tool is meant/capable to
build C-libs whih might get included within the project and then e.g.
build bindings (e.g. using SWIG) etc.?

What do you think about capabilities ala Cmake's CPack to be able to
prepare installers/packages for different systems/OS-es?



Sincerely,
Gour

--=20
In this endeavor there is no loss or diminution,=20
and a little advancement on this path can protect=20
one from the most dangerous type of fear.

http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810
Nov 24 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Right now my goal is to get xfBuild stable (fix bugs I might have
introduced), and try to fix any bugs it had from before.

I'm certainly interested in adding major new features, but I don't
know whether I should bloat the xfBuild codebase or start a new
project for that. And anyway that will come at a later time. Bug fixes
are a priority right now.

On 11/24/11, Gour <gour atmarama.net> wrote:

 What do you think about capabilities ala Cmake's CPack to be able to
 prepare installers/packages for different systems/OS-es?


As far as packages go, Jacob Carlborg is working on Orbit which is a
package manager. I really think builders and packagers are two
different beasts and should probably be done separately.

On 11/24/11, Gour <gour atmarama.net> wrote:

 Excuse my ignorance, but I wonder if xfbuild tool is meant/capable to
 build C-libs which might get included within the project and then e.g.
 build bindings (e.g. using SWIG) etc.?


No, xfBuild was made strictly for compiling D code by automatically
tracking dependencies based on imports and deps files generated by DMD
(maybe even GDC, I'm not sure yet). C libs come with their own build
systems and shell scripts, and you'd have more luck solving the
halting problem than making a tool that can autobuild and do SWIG
wrapping of arbitrary C libraries. :p
Nov 24 2011
parent reply Gour <gour atmarama.net> writes:
On Thu, 24 Nov 2011 18:43:30 +0100
Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:


 Right now my goal is to get xfBuild stable (fix bugs I might have
 introduced), and try to fix any bugs it had from before.


OK. Fair-enough.


 I'm certainly interested in adding major new features, but I don't
 know whether I should bloat the xfBuild codebase or start a new
 project for that. And anyway that will come at a later time. Bug fixes
 are a priority right now.


OK.


 As far as packages go, Jacob Carlborg is working on Orbit which is a
 package manager. I really think builders and packagers are two
 different beasts and should probably be done separately.


Right, like cmake & cpack.


 No, xfBuild was made strictly for compiling D code by automatically
 tracking dependencies based on imports and deps files generated by DMD
 (maybe even GDC, I'm not sure yet).=20


OK.


 C libs come with their own build systems and shell scripts, and you'd
 have more luck solving the halting problem than making a tool that can
 autobuild and do SWIG wrapping of arbitrary C libraries. :p


Well, in my case I soon plan to work on D bindings for 3rd party C-lib
which is usually shipped along with the project using it. So, in order
to build the application we need to build C lib first, then create D
bindings (e.g. by using SWIG) and finally build application which uses
that bindings.


Sincerely,
Gour

--=20
Nov 24 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Well you could always ship a build script, .bat/.sh or even a .d
script runnable via RDMD or something.
Nov 24 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-11-24 21:36, Andrej Mitrovic wrote:

 Well you could always ship a build script, .bat/.sh or even a .d
 script runnable via RDMD or something.


For executables a shell script executing RDMD works quite well.

-- 
/Jacob Carlborg
Nov 24 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
FWIW a couple of bugs from the old issue tracker were fixed in the new
fork. Bugs were reported here:
https://bitbucket.org/h3r3tic/xfbuild/issue/16/xfbuild-crashes-if-full-path-to-deps-objs
https://bitbucket.org/h3r3tic/xfbuild/issue/17/xfbuild-crashes-optlink-on-windows-if-o

There was also a multithreaded bug that manifested itself on Windows
where printing to the console would be interleaved and left the
message scrambled. This is now fixed.

Also, the +xpath option was added, first discussed here:
https://bitbucket.org/h3r3tic/xfbuild/issue/1/add-a-xpath-option

I'll have to look into the whole template instantiation "fiasco" and
see where we are at that and if it can be resolved. Stay tuned.. :)

https://github.com/AndrejMitrovic/xfbuild
Nov 29 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-11-29 12:00, Andrej Mitrovic wrote:

 FWIW a couple of bugs from the old issue tracker were fixed in the new
 fork. Bugs were reported here:
 https://bitbucket.org/h3r3tic/xfbuild/issue/16/xfbuild-crashes-if-full-path-to-deps-objs
 https://bitbucket.org/h3r3tic/xfbuild/issue/17/xfbuild-crashes-optlink-on-windows-if-o

 There was also a multithreaded bug that manifested itself on Windows
 where printing to the console would be interleaved and left the
 message scrambled. This is now fixed.

 Also, the +xpath option was added, first discussed here:
 https://bitbucket.org/h3r3tic/xfbuild/issue/1/add-a-xpath-option

 I'll have to look into the whole template instantiation "fiasco" and
 see where we are at that and if it can be resolved. Stay tuned.. :)

 https://github.com/AndrejMitrovic/xfbuild


Walter mentioned something that if the "-lib" flag is used it will 
output templates to all object files. If that was what you were 
referring to.

-- 
/Jacob Carlborg
Nov 29 2011
parent reply torhu <no spam.invalid> writes:
On 29.11.2011 13:20, Jacob Carlborg wrote:

 On 2011-11-29 12:00, Andrej Mitrovic wrote:

  FWIW a couple of bugs from the old issue tracker were fixed in the new
  fork. Bugs were reported here:
  https://bitbucket.org/h3r3tic/xfbuild/issue/16/xfbuild-crashes-if-full-path-to-deps-objs
  https://bitbucket.org/h3r3tic/xfbuild/issue/17/xfbuild-crashes-optlink-on-windows-if-o

  There was also a multithreaded bug that manifested itself on Windows
  where printing to the console would be interleaved and left the
  message scrambled. This is now fixed.

  Also, the +xpath option was added, first discussed here:
  https://bitbucket.org/h3r3tic/xfbuild/issue/1/add-a-xpath-option

  I'll have to look into the whole template instantiation "fiasco" and
  see where we are at that and if it can be resolved. Stay tuned.. :)

  https://github.com/AndrejMitrovic/xfbuild


 Walter mentioned something that if the "-lib" flag is used it will
 output templates to all object files. If that was what you were
 referring to.


The -lib flag leads to bloated executables.  Maybe the linker doesn't 
get rid of the duplicates, I don't know.
Nov 29 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-11-29 22:14, torhu wrote:

 On 29.11.2011 13:20, Jacob Carlborg wrote:

 On 2011-11-29 12:00, Andrej Mitrovic wrote:

 FWIW a couple of bugs from the old issue tracker were fixed in the new
 fork. Bugs were reported here:
 https://bitbucket.org/h3r3tic/xfbuild/issue/16/xfbuild-crashes-if-full-path-to-deps-objs

 https://bitbucket.org/h3r3tic/xfbuild/issue/17/xfbuild-crashes-optlink-on-windows-if-o


 There was also a multithreaded bug that manifested itself on Windows
 where printing to the console would be interleaved and left the
 message scrambled. This is now fixed.

 Also, the +xpath option was added, first discussed here:
 https://bitbucket.org/h3r3tic/xfbuild/issue/1/add-a-xpath-option

 I'll have to look into the whole template instantiation "fiasco" and
 see where we are at that and if it can be resolved. Stay tuned.. :)

 https://github.com/AndrejMitrovic/xfbuild


 Walter mentioned something that if the "-lib" flag is used it will
 output templates to all object files. If that was what you were
 referring to.


 The -lib flag leads to bloated executables. Maybe the linker doesn't get
 rid of the duplicates, I don't know.


That may be so, but at least the application works, if Walter is correct 
about the -lib flag. He has also said he doesn't want to add a flag that 
does what's necessary with the templates.

-- 
/Jacob Carlborg
Nov 29 2011
parent reply torhu <no spam.invalid> writes:
On 30.11.2011 08:15, Jacob Carlborg wrote:

 On 2011-11-29 22:14, torhu wrote:

  On 29.11.2011 13:20, Jacob Carlborg wrote:

  On 2011-11-29 12:00, Andrej Mitrovic wrote:

  FWIW a couple of bugs from the old issue tracker were fixed in the new
  fork. Bugs were reported here:
  https://bitbucket.org/h3r3tic/xfbuild/issue/16/xfbuild-crashes-if-full-path-to-deps-objs

  https://bitbucket.org/h3r3tic/xfbuild/issue/17/xfbuild-crashes-optlink-on-windows-if-o


  There was also a multithreaded bug that manifested itself on Windows
  where printing to the console would be interleaved and left the
  message scrambled. This is now fixed.

  Also, the +xpath option was added, first discussed here:
  https://bitbucket.org/h3r3tic/xfbuild/issue/1/add-a-xpath-option

  I'll have to look into the whole template instantiation "fiasco" and
  see where we are at that and if it can be resolved. Stay tuned.. :)

  https://github.com/AndrejMitrovic/xfbuild


  Walter mentioned something that if the "-lib" flag is used it will
  output templates to all object files. If that was what you were
  referring to.


  The -lib flag leads to bloated executables. Maybe the linker doesn't get
  rid of the duplicates, I don't know.


 That may be so, but at least the application works, if Walter is correct
 about the -lib flag. He has also said he doesn't want to add a flag that
 does what's necessary with the templates.


Maybe this is about something else than I was thinking of.  I was 
thinking of the linker errors you will get if you do partial rebuilds. 
Missing symbols for struct initializers and things like that.  And 
templates too, I guess.  Can't remember the details anymore.  But the 
-lib flags wouldn't help with that anyway.
Nov 30 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-11-30 21:13, torhu wrote:

 Maybe this is about something else than I was thinking of. I was
 thinking of the linker errors you will get if you do partial rebuilds.
 Missing symbols for struct initializers and things like that. And
 templates too, I guess. Can't remember the details anymore. But the -lib
 flags wouldn't help with that anyway.


I think I'm referring to the same, or something similar. Apparently DMD 
doesn't emit template instances to all object files that need them. When 
doing incremental builds it can happen that some of the template 
instances gets removed when an object file is replaced.

https://bitbucket.org/h3r3tic/xfbuild/issue/7/make-incremental

This link suggests that the -lib flag will put each output the template 
instances to all object files that need it: 
http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=96168

-- 
/Jacob Carlborg
Nov 30 2011
prev sibling parent reply Joel Christensen <joelcnz gmail.com> writes:
Hi,

I'm trying to get xfbuild working.

First I found with WindowsAPI the line 70 of directx\d3d9.d has an 
error. But fixing it I run into another problem:

Cannot use win32.winsock without Win32_Winsock1 defined.
winsock.d(31): Error: static assert  (false) is false

I'm not sure where dcollections fits in.

Maybe I should just try again will further updates.

Thanks for any help,

-joelcnz
Nov 30 2011
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I knew I should have just distributed win32 with xfbuild. Someone
screwed up the API headers.

I'll distribute the API with xfbuild in a few minutes. Thanks for
letting me know about this.
Nov 30 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Or maybe I just cut off the directx stuff. Heh, maybe I'm to blame.
Anyway I'll fix this shortly.
Nov 30 2011
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Ok it should work now.
Nov 30 2011
parent reply Joel Christensen <joelcnz gmail.com> writes:
On 01-Dec-11 2:28 PM, Andrej Mitrovic wrote:

 Ok it should work now.


Thanks Andrej. Works now. Or, I have to change how StopWatch 
(std.datetime) works in my programs, because putting '*.start;' twice 
fails an assert.

Also, it doesn't seem to work with Visual D.

One of my programs uses it in the command prompt now. :-)

-joelcnz
Nov 30 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/1/11, Joel Christensen <joelcnz gmail.com> wrote:

 Thanks Andrej. Works now. Or, I have to change how StopWatch
 (std.datetime) works in my programs, because putting '*.start;' twice
 fails an assert.


Hmm, I'm not sure what you're referring to. Is something broken about
the profiler in xfBuild (it uses stopwatch), or are you referring to
your own code?


 Also, it doesn't seem to work with Visual D.


I haven't tested it with VisualD, but I'll give it some testing.
VisualD does have it's own builder, of course. How are you invoking it
/ what exactly is failing?


 One of my programs uses it in the command prompt now. :-)


Cool. If there are any issues (and I know of some where xfBuild likes
to crash, some parts need to be refactored), feel free to file them so
I can fix them as soon as possible.
Dec 01 2011
parent reply Joel Christensen <joelcnz gmail.com> writes:
On 01-Dec-11 11:08 PM, Andrej Mitrovic wrote:

 On 12/1/11, Joel Christensen<joelcnz gmail.com>  wrote:

 Thanks Andrej. Works now. Or, I have to change how StopWatch
 (std.datetime) works in my programs, because putting '*.start;' twice
 fails an assert.


 Hmm, I'm not sure what you're referring to. Is something broken about
 the profiler in xfBuild (it uses stopwatch), or are you referring to
 your own code?


My own code. I used '*.reset' instead of putting '*.start' in again.


 Also, it doesn't seem to work with Visual D.


 I haven't tested it with VisualD, but I'll give it some testing.
 VisualD does have it's own builder, of course. How are you invoking it
 / what exactly is failing?


VisualD has more than one source file in the compiler arguments.


 One of my programs uses it in the command prompt now. :-)


 Cool. If there are any issues (and I know of some where xfBuild likes
 to crash, some parts need to be refactored), feel free to file them so
 I can fix them as soon as possible.


It doesn't handle deprecated stuff, (at lest without the '-d', but I 
haven't tried that).
Dec 01 2011
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 01.12.2011 21:07, Joel Christensen wrote:

 On 01-Dec-11 11:08 PM, Andrej Mitrovic wrote:

 On 12/1/11, Joel Christensen<joelcnz gmail.com> wrote:

 Thanks Andrej. Works now. Or, I have to change how StopWatch
 (std.datetime) works in my programs, because putting '*.start;' twice
 fails an assert.


 Hmm, I'm not sure what you're referring to. Is something broken about
 the profiler in xfBuild (it uses stopwatch), or are you referring to
 your own code?


 My own code. I used '*.reset' instead of putting '*.start' in again.


 Also, it doesn't seem to work with Visual D.


 I haven't tested it with VisualD, but I'll give it some testing.
 VisualD does have it's own builder, of course. How are you invoking it
 / what exactly is failing?


 VisualD has more than one source file in the compiler arguments.


You could set the "Build tool" for all files but the main module to 
"None", so they will not be placed on the command line (doing it on a 
multi selection does not seem to work, though).

If I understand correctly, xfbuilds' big feature is to collect imported 
modules and only compile files that are modified or that import modified 
files, but in a single compiler run. These are linked together 
afterwards with the remaining object files.

I have considered adding such a compilation mode to Visual D aswell, but 
I don't see a large advantage to compiling the whole project because the 
import dependencies easily spread across all files like a virus. Even 
creating di interface files does not help, as you must not remove 
imports because it can ruin the order static constructors are run.

If there are natural independent groups of modules, I put them into 
libraries anyway - otherwise the compiler can run into memory trouble.

What's your experience? Does xfbuild work significantly faster than 
building all the modules like rdmd?
Dec 02 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Honestly I don't think xfBuild should be used in IDEs, an IDE already
has all the information about module dependencies and has a better
idea on what to compile.

Currently xfBuild is actually slower than RDMD on full builds (both
the D1 and D2 branch). When I further refactor the codebase I'll see
what is slowing it down. However on incremental builds xfBuild is
faster. It still likes to crash sometime though, so RDMD is safer to
use at the moment.

Anyway RDMD is DMD-specific, but I want xfBuild to work with multiple
compilers (DMD/GDC/LDC/SDC) and linkers.
Dec 02 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-12-02 18:38, Andrej Mitrovic wrote:

 Honestly I don't think xfBuild should be used in IDEs, an IDE already
 has all the information about module dependencies and has a better
 idea on what to compile.

 Currently xfBuild is actually slower than RDMD on full builds (both
 the D1 and D2 branch). When I further refactor the codebase I'll see
 what is slowing it down. However on incremental builds xfBuild is
 faster. It still likes to crash sometime though, so RDMD is safer to
 use at the moment.

 Anyway RDMD is DMD-specific, but I want xfBuild to work with multiple
 compilers (DMD/GDC/LDC/SDC) and linkers.


Both GDC and LDC ships with DMD wrappers.

-- 
/Jacob Carlborg
Dec 04 2011