www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - OPTLINK needs to die.

reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
That's it -- I'm finished.  I'm tired of constantly rearranging code
to appease OPTLINK and its outdated object format.

This is, for me, the number one blocker of most of my more complex
code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
and Eldar sounds like he s about to give up on D after dealing with
OPTLINK's crashes.

OpenWatcom is not an option.  It seems to be buggy with regards to
weak symbols.  Furthermore its documentation is sparse at best and its
option syntax is completely different from OPTLINK's.

I *do not care* how fast OPTLINK is.  I *do not care* how many hours
of hand-coded assembly went into it.  I want a linker that *works*,
and preferably one that doesn't use an object format that *no other
modern compilers use*.

Please do not underestimate the importance of this, Walter.  OMF and
OPTLINK will continue to be a thorn in DMD's side unless you do
something about it now.
Feb 16 2009
next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Jarrett Billingsley wrote:

 That's it -- I'm finished.  I'm tired of constantly rearranging code
 to appease OPTLINK and its outdated object format.
 
 This is, for me, the number one blocker of most of my more complex
 code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
 and Eldar sounds like he s about to give up on D after dealing with
 OPTLINK's crashes.


Hehehe, agreed :P Out of desperation I've even tried not using OPTLINK 
for one of my programs and loading it at runtime from .obj files using 
DDL instead :P It took 3 seconds to link since DDL needs many 
optimizations, but at least it didn't crash like OPTLINK would :P



 OpenWatcom is not an option.  It seems to be buggy with regards to
 weak symbols.  Furthermore its documentation is sparse at best and its
 option syntax is completely different from OPTLINK's.


Indeed, I've tried Open Watcom's wlink two days ago and it borks on some 
weak symbols from snn.lib, claiming that they are unresolved. The old 
version on its author's site doesn't like long symbol names (since DMD 
and OPTLINK use an extension for the OMF format to allow longer names).
As for the documentation, you probably haven't found it :P There's a 
huge PDF file with examples :)



 I *do not care* how fast OPTLINK is.  I *do not care* how many hours
 of hand-coded assembly went into it.  I want a linker that *works*,
 and preferably one that doesn't use an object format that *no other
 modern compilers use*.


Agreed :P



 Please do not underestimate the importance of this, Walter.  OMF and
 OPTLINK will continue to be a thorn in DMD's side unless you do
 something about it now.


Perhaps our best bet would be fixing the bug in Open Watcom's linker? Or 
rolling something on our own using the code from DDL? I wouldn't mind 
DMD emitting COFF obj files, that the MS linker could process, either :P


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Feb 16 2009
parent BCS <none anon.com> writes:
Hello Tom,


 Perhaps our best bet would be fixing the bug in Open Watcom's linker?
 Or rolling something on our own using the code from DDL? I wouldn't
 mind DMD emitting COFF obj files, that the MS linker could process,
 either :P
 


Well Walter is up to speed on object file generators right now. I second 
the motion for DMD's next target to be a better object file format on windows. 
Also, if I understand the situation correctly, it wouldn't be hard to make 
a DMC release to match.
Feb 16 2009
prev sibling next sibling parent reply "Tim M" <a b.com> writes:
On Tue, 17 Feb 2009 13:27:14 +1300, Jarrett Billingsley  
<jarrett.billingsley gmail.com> wrote:


 That's it -- I'm finished.  I'm tired of constantly rearranging code
 to appease OPTLINK and its outdated object format.

 This is, for me, the number one blocker of most of my more complex
 code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
 and Eldar sounds like he s about to give up on D after dealing with
 OPTLINK's crashes.

 OpenWatcom is not an option.  It seems to be buggy with regards to
 weak symbols.  Furthermore its documentation is sparse at best and its
 option syntax is completely different from OPTLINK's.

 I *do not care* how fast OPTLINK is.  I *do not care* how many hours
 of hand-coded assembly went into it.  I want a linker that *works*,
 and preferably one that doesn't use an object format that *no other
 modern compilers use*.

 Please do not underestimate the importance of this, Walter.  OMF and
 OPTLINK will continue to be a thorn in DMD's side unless you do
 something about it now.



Just checking you are aware of objconv which can be found here  
http://www.agner.org/optimize/.
Feb 16 2009
next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Tim M wrote:

 Just checking you are aware of objconv which can be found here 
 http://www.agner.org/optimize/.


Have you successfully used it to convert all OMF obj files of some 
application to another format and link them? It gives me a lot of: 
"Error 2316: Incompatible relocation method: 16+32 bit far".


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Feb 16 2009
parent reply "Tim M" <a b.com> writes:
On Tue, 17 Feb 2009 14:35:55 +1300, Tom S  
<h3r3tic remove.mat.uni.torun.pl> wrote:


 Tim M wrote:

 Just checking you are aware of objconv which can be found here  
 http://www.agner.org/optimize/.


 Have you successfully used it to convert all OMF obj files of some  
 application to another format and link them? It gives me a lot of:  
 "Error 2316: Incompatible relocation method: 16+32 bit far".



No I convert MS COFF files to OMF and I totally agree with the multiple  
formats problems. I don't think making another converter or improving  
exisitng converters is the sollution though. Native COFF support is the  
way.
Feb 16 2009
parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 8:48 PM, Tim M <a b.com> wrote:

 No I convert MS COFF files to OMF and I totally agree with the multiple
 formats problems. I don't think making another converter or improving
 exisitng converters is the sollution though. Native COFF support is the way.


Oh, that might be useful for linking to COFF libraries, but it
certainly doesn't help with the other major problems with OPTLINK and
the OMF format.
Feb 16 2009
prev sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 8:19 PM, Tim M <a b.com> wrote:

 On Tue, 17 Feb 2009 13:27:14 +1300, Jarrett Billingsley
 <jarrett.billingsley gmail.com> wrote:


 That's it -- I'm finished.  I'm tired of constantly rearranging code
 to appease OPTLINK and its outdated object format.

 This is, for me, the number one blocker of most of my more complex
 code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
 and Eldar sounds like he s about to give up on D after dealing with
 OPTLINK's crashes.

 OpenWatcom is not an option.  It seems to be buggy with regards to
 weak symbols.  Furthermore its documentation is sparse at best and its
 option syntax is completely different from OPTLINK's.

 I *do not care* how fast OPTLINK is.  I *do not care* how many hours
 of hand-coded assembly went into it.  I want a linker that *works*,
 and preferably one that doesn't use an object format that *no other
 modern compilers use*.

 Please do not underestimate the importance of this, Walter.  OMF and
 OPTLINK will continue to be a thorn in DMD's side unless you do
 something about it now.



 Just checking you are aware of objconv which can be found here
 http://www.agner.org/optimize/.


I'd rather not have to add in an _object conversion_ step to my build
process.  And I wouldn't want to have to force users of my libraries
to do so either.
Feb 16 2009
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
One thing you can try is:

1. download imagecfg.exe from http://www.robpol86.com/pages/imagecfg.php

2. run the command:

	imagecfg -a 0x1 link.exe

This will set optlink to use only one core. I've suspected for a while 
that the multithreading locks in it don't work right for multicore. I 
have a single core machine and cannot test multicore behavior.
Feb 16 2009
parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 9:54 PM, Walter Bright
<newshound1 digitalmars.com> wrote:

 One thing you can try is:

 1. download imagecfg.exe from http://www.robpol86.com/pages/imagecfg.php

 2. run the command:

        imagecfg -a 0x1 link.exe

 This will set optlink to use only one core. I've suspected for a while that
 the multithreading locks in it don't work right for multicore. I have a
 single core machine and cannot test multicore behavior.


Tom S does all of his development on a single-core machine, and I have
this problem on my single-core laptop as well.  It is not
multithreading.
Feb 16 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Jarrett Billingsley wrote:

 On Mon, Feb 16, 2009 at 9:54 PM, Walter Bright
 <newshound1 digitalmars.com> wrote:

 One thing you can try is:

 1. download imagecfg.exe from http://www.robpol86.com/pages/imagecfg.php

 2. run the command:

        imagecfg -a 0x1 link.exe

 This will set optlink to use only one core. I've suspected for a while that
 the multithreading locks in it don't work right for multicore. I have a
 single core machine and cannot test multicore behavior.


 
 Tom S does all of his development on a single-core machine, and I have
 this problem on my single-core laptop as well.  It is not
 multithreading.


But I downloaded the offending obj file from bugzilla, and it links and 
runs without error:

http://d.puremagic.com/issues/show_bug.cgi?id=2436
Feb 16 2009
parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 11:08 PM, Walter Bright
<newshound1 digitalmars.com> wrote:

 But I downloaded the offending obj file from bugzilla, and it links and runs
 without error:

 http://d.puremagic.com/issues/show_bug.cgi?id=2436


http://d.puremagic.com/issues/show_bug.cgi?id=424

is *much* older, and you already seem to have classed it as unfixable.
 It's the "too many fixups" bug (although instead of popping up a
message to that effect OPTLINK just unhelpfully crashes and dumps its
registers).  This is the one that bites me all the time.  I'm not
doing terribly uncommon or large things.  Even something like Pyd is
liable to kill OPTLINK with relatively small wrappings.

Splitting code across multiple modules _sometimes_ helps and
_sometimes_ does not.  Besides being a crude workaround it's sometimes
much more difficult than it sounds (because as we all know, DMD does
*not* like circular imports, and sometimes massive modules are the
result of working around _that_ bug).  -g affects it as it's now
generating lots of extra debug symbols, pushing the number of fixups
up.  There's no way to tell how many symbols I'm generating and I
shouldn't _have_ to worry about that kind of crap.

Worst of all, DMDWin is the *only* compiler with these problems.  So
I'm doing all this work to get around bugs in a single compiler on a
single platform.  It makes me want to just abandon it.

Do you think that with the new uber-templated Phobos2 that the problem
is going to be getting any better?
Feb 16 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Jarrett Billingsley wrote:

 http://d.puremagic.com/issues/show_bug.cgi?id=424



I just added this to the bug report:

======================
The compiler already has a switch to generate multiple obj files from 
one source file: -multiobj. I use it for debugging. But it would be a 
royal pain to use if there's a lot.

A better way is to compile to a library with the -lib switch. That 
splits the module into a zillion obj files, and stuffs them all into the 
library. Then, link with the library (the main() program will still have 
to be separate).
=======================

Can you give it a try?
Feb 16 2009
next sibling parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Mon, Feb 16, 2009 at 11:32 PM, Walter Bright
<newshound1 digitalmars.com> wrote:

 Jarrett Billingsley wrote:

 http://d.puremagic.com/issues/show_bug.cgi?id=424



 I just added this to the bug report:

 ======================
 The compiler already has a switch to generate multiple obj files from one
 source file: -multiobj. I use it for debugging. But it would be a royal pain
 to use if there's a lot.


Why isn't it documented?  And you're damn right it would be a pain.
Especially when I'm compiling three external libraries and a large
library of my own.


 A better way is to compile to a library with the -lib switch. That splits
 the module into a zillion obj files, and stuffs them all into the library.
 Then, link with the library (the main() program will still have to be
 separate).
 =======================

 Can you give it a try?


Not easily.  Under my normal compilation scheme I'm not building a
library and I have no idea how well bud/rebuild will cope with a
zillion object files.

In any case the -lib switch is just sidestepping the problems with the
linker and object format by pushing some of the linker's
responsibilities into the compiler.  DMD (and the D language itself,
if you want to count the 16MB limit on static data, which has no place
in a portable language specification!) already makes far too many
concessions to deal with the ridiculous constraints of the linker.
Please, stop designing around the problem and fix it.
Feb 16 2009
parent Walter Bright <newshound1 digitalmars.com> writes:
Jarrett Billingsley wrote:

 On Mon, Feb 16, 2009 at 11:32 PM, Walter Bright
 <newshound1 digitalmars.com> wrote:

 Jarrett Billingsley wrote:

 http://d.puremagic.com/issues/show_bug.cgi?id=424


 I just added this to the bug report:

 ======================
 The compiler already has a switch to generate multiple obj files from one
 source file: -multiobj. I use it for debugging. But it would be a royal pain
 to use if there's a lot.


 
 Why isn't it documented?  And you're damn right it would be a pain.
 Especially when I'm compiling three external libraries and a large
 library of my own.


Because it's pretty useless.



 Can you give it a try?


 Not easily.  Under my normal compilation scheme I'm not building a
 library and I have no idea how well bud/rebuild will cope with a
 zillion object files.


The point of building a library directly is that you never see the 
zillion object files. dmd generates the library directly.


 In any case the -lib switch is just sidestepping the problems with the
 linker and object format by pushing some of the linker's
 responsibilities into the compiler.  DMD (and the D language itself,
 if you want to count the 16MB limit on static data, which has no place
 in a portable language specification!) already makes far too many
 concessions to deal with the ridiculous constraints of the linker.
 Please, stop designing around the problem and fix it.


The -lib option has nothing to do with linker problems. It exists and is 
highly useful on linux and osx. It enables building libraries in one 
operation without having to dink around with managing the object files.
Feb 16 2009
prev sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:

 Jarrett Billingsley wrote:

 http://d.puremagic.com/issues/show_bug.cgi?id=424


 
 
 I just added this to the bug report:
 
 ======================
 The compiler already has a switch to generate multiple obj files from 
 one source file: -multiobj. I use it for debugging. But it would be a 
 royal pain to use if there's a lot.
 
 A better way is to compile to a library with the -lib switch. That 
 splits the module into a zillion obj files, and stuffs them all into the 
 library. Then, link with the library (the main() program will still have 
 to be separate).
 =======================
 
 Can you give it a try?


I've tried it before and unfortunately there's a problem with this 
approach: static ctors from modules linked from a .lib don't get executed:


cat Main.d


void main() {
}


cat Mod.d


extern (C) int printf(char*, ...);


static this() {
         printf("Running a static ctor for Mod"\n);
}

// compiling Mod into an .obj

dmd -c Main.d
dmd -c Mod.d
dmd Main.obj Mod.obj && Main.exe


Running a static ctor for Mod

// compiling Mod into a .lib

dmd -lib Mod.d
dmd Main.obj Mod.lib && Main.exe



-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Feb 16 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:

 I've tried it before and unfortunately there's a problem with this 
 approach: static ctors from modules linked from a .lib don't get executed:


Not everything is linked in from a library, just referenced things 
(that's the whole point of a library). There is nothing referenced in 
mod.d. But rewriting as:

---------------
import mod;

void main()
{
     foo();
}
---------------
import std.c.stdio;

void foo() { }

static this() {
         printf("Running a static ctor for Mod"\n);
}
---------------

Compiling and running:

C:>dmd -lib mod

C:>dmd test5 mod.lib

C:>test5
Running a static ctor for Mod


Or, compile the module with just the static constructor in it separately 
and add it explicitly to the link list.
Feb 16 2009
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:

 Tom S wrote:

 I've tried it before and unfortunately there's a problem with this 
 approach: static ctors from modules linked from a .lib don't get 
 executed:


 
 Not everything is linked in from a library, just referenced things 
 (that's the whole point of a library). There is nothing referenced in 
 mod.d. But rewriting as:
 
 ---------------
 import mod;
 
 void main()
 {
     foo();
 }
 ---------------
 import std.c.stdio;
 
 void foo() { }
 
 static this() {
         printf("Running a static ctor for Mod"\n);
 }
 ---------------
 
 Compiling and running:
 
 C:>dmd -lib mod
 
 C:>dmd test5 mod.lib
 
 C:>test5
 Running a static ctor for Mod
 
 
 Or, compile the module with just the static constructor in it separately 
 and add it explicitly to the link list.


This one still has problems.

---- Main.d:
import Mod;

void main() {
	foo();
}
---- Mod.d:
import Mod2;

extern (C) int printf(char*, ...);

void foo() {
	foo2();
}

static this() {
	printf("Running a static ctor for Mod"\n);
}
---- Mod2.d:
extern (C) int printf(char*, ...);

void foo2() {
}

static this() {
	printf("Running a static ctor for Mod2"\n);
}
----

Compiling these modules as:
dmd -c Main.d
dmd -lib Mod.d Mod2.d
dmd Main.obj Mod.lib
... and running Main.exe yields:
Running a static ctor for Mod

So it seems that Main.d would have to reference symbols from each module 
in the program. Ouch.

Additionally, using such an approach breaks any possibility to use 
incremental compilation. So there's also the option of having a .lib per 
.d file and linking that all together. So I've just tried it:
dmd -c Main.d
dmd -lib Mod.d
dmd -lib Mod2.d
dmd Main.obj Mod.lib Mod2.d
... and OPTLINK crashed with the familiar MsgBox containing register values.

While we're at incremental compilation, this bug: 
http://d.puremagic.com/issues/show_bug.cgi?id=1629 often kills it for me 
and it's been around for a few whiles already.


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Feb 17 2009
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:

 Additionally, using such an approach breaks any possibility to use 
 incremental compilation. So there's also the option of having a .lib per 
 .d file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.


It does not crash when I try it. Are you sure you're not running a 
multicore computer?
Feb 17 2009
parent reply "Danny Wilson" <bluezenix gmail.com> writes:
Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright  
<newshound1 digitalmars.com>:


 It does not crash when I try it. Are you sure you're not running a  
 multicore computer?


It's pretty hard to _NOT_ get a multicore computer these days. Please kill  
OPTLINK.
Feb 17 2009
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Danny Wilson wrote:

 
 Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:
 

 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?


 
 It's pretty hard to _NOT_ get a multicore computer these days. Please 
 kill OPTLINK.


What I'm asking is for someone with a multicore computer who is 
experiencing these crashes to try the patch I outlined to restrict it to 
one core.
Feb 17 2009
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright <newshound1 digitalmars.com>
wrote:


 Danny Wilson wrote:

  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright  
 <newshound1 digitalmars.com>:


 It does not crash when I try it. Are you sure you're not running a  
 multicore computer?


  It's pretty hard to _NOT_ get a multicore computer these days. Please  
 kill OPTLINK.


 What I'm asking is for someone with a multicore computer who is  
 experiencing these crashes to try the patch I outlined to restrict it to  
 one core.


It is responded that three people (Tom S, Jarrett, Max Samukha) are running
single-core machines and still experiencing constant crashes.
Feb 17 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Denis Koroskin wrote:

 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 

 Danny Wilson wrote:

  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:


 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?


  It's pretty hard to _NOT_ get a multicore computer these days. 
 Please kill OPTLINK.


 What I'm asking is for someone with a multicore computer who is 
 experiencing these crashes to try the patch I outlined to restrict it 
 to one core.


 
 It is responded that three people (Tom S, Jarrett, Max Samukha) are 
 running single-core machines and still experiencing constant crashes.


It doesn't crash on my machine on *any* of the examples that produce a 
crash on your machines. Can you please just try the patch anyway? It may 
also be related to the hyperthreading on a single core machine.
Feb 17 2009
next sibling parent reply HOSOKAWA Kenchi <hskwk inter7.jp> writes:
It's not sure but some processors which have SMT act as pseudo-multicore
manner, like Pentium 4 and Atom.
Unfortunately I don't have them.


Walter Bright Wrote:

 Denis Koroskin wrote:

 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 

 Danny Wilson wrote:

  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:


 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?


  It's pretty hard to _NOT_ get a multicore computer these days. 
 Please kill OPTLINK.


 What I'm asking is for someone with a multicore computer who is 
 experiencing these crashes to try the patch I outlined to restrict it 
 to one core.


 
 It is responded that three people (Tom S, Jarrett, Max Samukha) are 
 running single-core machines and still experiencing constant crashes.


 
 It doesn't crash on my machine on *any* of the examples that produce a 
 crash on your machines. Can you please just try the patch anyway? It may 
 also be related to the hyperthreading on a single core machine.
Feb 17 2009
parent HOSOKAWA Kenchi <hskwk inter7.jp> writes:
I'm sorry that my eyes leaped over the word "hyperthreading".

I just tested OPTLINK bug I reported on the Issue Tracking System,
http://d.puremagic.com/issues/show_bug.cgi?id=2378
with fixed link.exe by imagecfg.

The issue is not solved.


HOSOKAWA Kenchi Wrote:


 It's not sure but some processors which have SMT act as pseudo-multicore
manner, like Pentium 4 and Atom.
 Unfortunately I don't have them.
 
 
 Walter Bright Wrote:

 Denis Koroskin wrote:

 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 

 Danny Wilson wrote:

  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:


 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?


  It's pretty hard to _NOT_ get a multicore computer these days. 
 Please kill OPTLINK.


 What I'm asking is for someone with a multicore computer who is 
 experiencing these crashes to try the patch I outlined to restrict it 
 to one core.


 
 It is responded that three people (Tom S, Jarrett, Max Samukha) are 
 running single-core machines and still experiencing constant crashes.


 
 It doesn't crash on my machine on *any* of the examples that produce a 
 crash on your machines. Can you please just try the patch anyway? It may 
 also be related to the hyperthreading on a single core machine.
Feb 17 2009
prev sibling parent reply Max Samukha <samukha voliacable.com.removethis> writes:
On Tue, 17 Feb 2009 04:04:18 -0800, Walter Bright
<newshound1 digitalmars.com> wrote:


Denis Koroskin wrote:

 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 

 Danny Wilson wrote:

  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:


 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?


  It's pretty hard to _NOT_ get a multicore computer these days. 
 Please kill OPTLINK.


 What I'm asking is for someone with a multicore computer who is 
 experiencing these crashes to try the patch I outlined to restrict it 
 to one core.


 
 It is responded that three people (Tom S, Jarrett, Max Samukha) are 
 running single-core machines and still experiencing constant crashes.


It doesn't crash on my machine on *any* of the examples that produce a 
crash on your machines. Can you please just try the patch anyway? It may 
also be related to the hyperthreading on a single core machine.


Have you tried to link my example with /co as I asked you in my last
post here http://d.puremagic.com/issues/show_bug.cgi?id=2436#c4 ?

It fails on AMD Athlon 1800, which is rather old. Changing the afinity
mask doesn't help
Feb 17 2009
next sibling parent Max Samukha <samukha voliacable.com.removethis> writes:
On Tue, 17 Feb 2009 14:54:09 +0200, Max Samukha
<samukha voliacable.com.removethis> wrote:


On Tue, 17 Feb 2009 04:04:18 -0800, Walter Bright
<newshound1 digitalmars.com> wrote:


Denis Koroskin wrote:

 On Tue, 17 Feb 2009 14:24:10 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 

 Danny Wilson wrote:

  Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:


 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?


  It's pretty hard to _NOT_ get a multicore computer these days. 
 Please kill OPTLINK.


 What I'm asking is for someone with a multicore computer who is 
 experiencing these crashes to try the patch I outlined to restrict it 
 to one core.


 
 It is responded that three people (Tom S, Jarrett, Max Samukha) are 
 running single-core machines and still experiencing constant crashes.


It doesn't crash on my machine on *any* of the examples that produce a 
crash on your machines. Can you please just try the patch anyway? It may 
also be related to the hyperthreading on a single core machine.


Have you tried to link my example with /co as I asked you in my last
post here http://d.puremagic.com/issues/show_bug.cgi?id=2436#c4 ?

It fails on AMD Athlon 1800, which is rather old. Changing the afinity
mask doesn't help 


"affinity" it should have been. Please don't ignore me because of my
grammer :)
Feb 17 2009
prev sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Max Samukha wrote:

 Have you tried to link my example with /co as I asked you in my last
 post here http://d.puremagic.com/issues/show_bug.cgi?id=2436#c4 ?


No, I hadn't tried it with /co, and yes, I can reproduce the failure 
with /co. Thanks!
Feb 17 2009
prev sibling parent "Nick Sabalausky" <a a.a> writes:
"Danny Wilson" <bluezenix gmail.com> wrote in message 
news:op.uphrgb2pbmuhq1 danny-wilsons-ep45-ds4.local...

 Op Tue, 17 Feb 2009 11:06:43 +0100 schreef Walter Bright 
 <newshound1 digitalmars.com>:


 It does not crash when I try it. Are you sure you're not running a 
 multicore computer?


 It's pretty hard to _NOT_ get a multicore computer these days. Please kill 
 OPTLINK.


Minor pet peeve: Just because something stops being offered in stores does 
*not* mean it isn't in heavy use.
Feb 17 2009
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:

 So it seems that Main.d would have to reference symbols from each module 
 in the program. Ouch.


I found that problem, and fixed it.


 Additionally, using such an approach breaks any possibility to use 
 incremental compilation.


It depends. If instead of a.obj, b.obj, and c.obj, think a.lib, b.lib, 
c.lib, you can incrementally compile.


 So there's also the option of having a .lib per 
 .d file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.


Can't repro it. I'm pretty sure it has something to do with how 
threading works on newer processors (mine is old).

Vendor string:    GenuineIntel
Processor string: Intel(R) Pentium(R) 4 CPU 1.60GHz
Signature:        Family=15 Model=1 Stepping=2
Features:         MMX FXSR SSE SSE2 HTT
Multithreading:   1 threads / 1 cores

What's your CPU?


 While we're at incremental compilation, this bug: 
 http://d.puremagic.com/issues/show_bug.cgi?id=1629 often kills it for me 
 and it's been around for a few whiles already.


Ok.
Feb 17 2009
next sibling parent reply Max Samukha <samukha voliacable.com.removethis> writes:
On Tue, 17 Feb 2009 04:16:35 -0800, Walter Bright
<newshound1 digitalmars.com> wrote:


Tom S wrote:

 So it seems that Main.d would have to reference symbols from each module 
 in the program. Ouch.


I found that problem, and fixed it.


 Additionally, using such an approach breaks any possibility to use 
 incremental compilation.


It depends. If instead of a.obj, b.obj, and c.obj, think a.lib, b.lib, 
c.lib, you can incrementally compile.


 So there's also the option of having a .lib per 
 .d file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.


Can't repro it. I'm pretty sure it has something to do with how 
threading works on newer processors (mine is old).


Probably you typed literally. The last one should be 'dmd Main.obj
Mod.lib Mod2.lib'

Vendor string:    GenuineIntel
Processor string: Intel(R) Pentium(R) 4 CPU 1.60GHz
Signature:        Family=15 Model=1 Stepping=2
Features:         MMX FXSR SSE SSE2 HTT
Multithreading:   1 threads / 1 cores

What's your CPU?



 While we're at incremental compilation, this bug: 
 http://d.puremagic.com/issues/show_bug.cgi?id=1629 often kills it for me 
 and it's been around for a few whiles already.


Ok.


Checked on an old Celeron and Athlon. Both fail
Feb 17 2009
next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Max Samukha wrote:

 Probably you typed literally.


I cut & pasted to be sure!


 The last one should be 'dmd Main.obj
 Mod.lib Mod2.lib'


Thank you. That does repro it.
Feb 17 2009
prev sibling parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Max Samukha wrote:

 On Tue, 17 Feb 2009 04:16:35 -0800, Walter Bright
 <newshound1 digitalmars.com> wrote:
 

 Tom S wrote:

 So there's also the option of having a .lib per 
 .d file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.


 Can't repro it. I'm pretty sure it has something to do with how 
 threading works on newer processors (mine is old).


 
 Probably you typed literally. The last one should be 'dmd Main.obj
 Mod.lib Mod2.lib'


Argh! Sorry, I knew I'd make a typo there since I didn't copy-paste the 
console commands :( Yea, it should be 'dmd Main.obj Mod.lib Mod2.lib'


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Feb 17 2009
prev sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:

 Tom S wrote:

 So it seems that Main.d would have to reference symbols from each 
 module in the program. Ouch.


 
 I found that problem, and fixed it.


Cool, thanks :)



 Additionally, using such an approach breaks any possibility to use 
 incremental compilation.


 
 It depends. If instead of a.obj, b.obj, and c.obj, think a.lib, b.lib, 
 c.lib, you can incrementally compile.


Yea, that's why I tried it next ;)



 So there's also the option of having a .lib per .d file and linking 
 that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register 
 values.


 
 Can't repro it. I'm pretty sure it has something to do with how 
 threading works on newer processors (mine is old).


I'll try DMD 1.040 and different OPTLINK versions then. I'm currently using:

Digital Mars D Compiler v1.039
OPTLINK (R) for Win32  Release 8.00.1
Digital Mars Librarian Version 8.02n  ( I don't think it's used in this 
case :P )



 Vendor string:    GenuineIntel
 Processor string: Intel(R) Pentium(R) 4 CPU 1.60GHz
 Signature:        Family=15 Model=1 Stepping=2
 Features:         MMX FXSR SSE SSE2 HTT
 Multithreading:   1 threads / 1 cores
 
 What's your CPU?


Vendor string:    GenuineIntel
Processor string: Intel(R) Pentium(R) M processor 1.70GHz
Signature:        Family=6 Model=13 Stepping=6
Features:         MMX FXSR SSE SSE2
Multithreading:   1 threads / 1 cores


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Feb 17 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
The problem appears to be that optlink cannot deal correctly with weak 
references. I've modified the compiler to not emit weak references, and 
the problem goes away. It's not a perfect fix, but it works.
Feb 19 2009
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:

 The problem appears to be that optlink cannot deal correctly with weak 
 references. I've modified the compiler to not emit weak references, and 
 the problem goes away. It's not a perfect fix, but it works.


With the modified compiler, does the following link? If so, is the 
output correct?

----
module a;
import c;
alias C!(int) aC;
import b;

extern(C) int printf(char*, ...);

void main() {
	printf("&aC.x == %x"\n, &aC.x);
	printf("&bC.x == %x"\n, &bC.x);
	assert (&aC.x is &bC.x);
}
----
module b;
import c;
alias C!(int) bC;
----
module c;
template C(T) {
	int x;
}
----
dmd -c a.d
dmd -c b.d
dmd -c c.d
dmd a.obj b.obj c.obj
a.exe
----

I'm getting:
&aC.x == 419c30
&bC.x == 419c30

I'm afraid that it might refuse to link and then even if OPTLINK allowed 
multiple strong symbols of the same name, I'd expect the assertion to 
fail. Aren't weak symbols essential to properly handling templates?

Of course compiling the modules together causes only one symbol to be 
emitted for the template instantiations. Yet this is not an option for a 
larger project, since compilation times and memory usage can get rather 
high.


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Feb 19 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:

 Walter Bright wrote:

 The problem appears to be that optlink cannot deal correctly with weak 
 references. I've modified the compiler to not emit weak references, 
 and the problem goes away. It's not a perfect fix, but it works.


 
 With the modified compiler, does the following link?


Yes.


 If so, is the output correct?


[...]

 I'm getting:
 &aC.x == 419c30
 &bC.x == 419c30


Yes, that is correct (there should be only one instance in the executable).


 I'm afraid that it might refuse to link and then even if OPTLINK allowed 
 multiple strong symbols of the same name, I'd expect the assertion to 
 fail. Aren't weak symbols essential to properly handling templates?


No. Weak references are: "if this symbol does not get resolved, then 
default initialize it to this other symbol". In other words, weak 
references do not cause things to be pulled in from the library.

Multiple symbols with the same name and same contents, across different 
object files, are called COMDATs and the linker picks one and discards 
the duplicates.

Where the compiler was emitting weak references was in the list of which 
modules are imported by the object file, so that if nobody actually 
referred to the import's contents beyond the import statement, the 
import would not get linked in.



 Of course compiling the modules together causes only one symbol to be 
 emitted for the template instantiations. Yet this is not an option for a 
 larger project, since compilation times and memory usage can get rather 
 high.



I know. This is why COMDATs were invented (for C++ templates). C++ drove 
a number of improvements in obj files.
Feb 19 2009
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Walter Bright wrote:

 Tom S wrote:

 Walter Bright wrote:

 The problem appears to be that optlink cannot deal correctly with 
 weak references. I've modified the compiler to not emit weak 
 references, and the problem goes away. It's not a perfect fix, but it 
 works.


 With the modified compiler, does the following link?


 
 Yes.
 

 If so, is the output correct?


 [...]

 I'm getting:
 &aC.x == 419c30
 &bC.x == 419c30


 
 Yes, that is correct (there should be only one instance in the executable).
 

 I'm afraid that it might refuse to link and then even if OPTLINK 
 allowed multiple strong symbols of the same name, I'd expect the 
 assertion to fail. Aren't weak symbols essential to properly handling 
 templates?


 
 No. Weak references are: "if this symbol does not get resolved, then 
 default initialize it to this other symbol". In other words, weak 
 references do not cause things to be pulled in from the library.
 
 Multiple symbols with the same name and same contents, across different 
 object files, are called COMDATs and the linker picks one and discards 
 the duplicates.
 
 Where the compiler was emitting weak references was in the list of which 
 modules are imported by the object file, so that if nobody actually 
 referred to the import's contents beyond the import statement, the 
 import would not get linked in.
 
 

 Of course compiling the modules together causes only one symbol to be 
 emitted for the template instantiations. Yet this is not an option for 
 a larger project, since compilation times and memory usage can get 
 rather high.


 
 
 I know. This is why COMDATs were invented (for C++ templates). C++ drove 
 a number of improvements in obj files.


I'm glad to be proven wrong :) Thanks for the info! I'm sure it will 
come in handy for the dynamic linking I'm doing with DDL. I got the 
silly idea that you'd disable COMDATs... Another case of "Too much blood 
in my caffeine".


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Feb 19 2009
parent Walter Bright <newshound1 digitalmars.com> writes:
Tom S wrote:

 I'm glad to be proven wrong :) Thanks for the info! I'm sure it will 
 come in handy for the dynamic linking I'm doing with DDL. I got the 
 silly idea that you'd disable COMDATs... Another case of "Too much blood 
 in my caffeine".


dmd would be quite dead if I had to disable COMDATs <g>.
Feb 19 2009
prev sibling next sibling parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 4:49 AM, Tom S <h3r3tic remove.mat.uni.torun.pl> wrote:


 Additionally, using such an approach breaks any possibility to use
 incremental compilation. So there's also the option of having a .lib per .d
 file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register values.


I can confirm that this happens on both my laptop (single-core,
single-threaded Pentium M) and my desktop.  Using imagecfg changes
nothing.
Feb 17 2009
parent Walter Bright <newshound1 digitalmars.com> writes:
Jarrett Billingsley wrote:

 Using imagecfg changes nothing.


Thanks for giving it a try. It saves me a lot of wasted effort looking 
under the wrong rocks :-)
Feb 17 2009
prev sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 9:06 AM, Jarrett Billingsley
<jarrett.billingsley gmail.com> wrote:

 On Tue, Feb 17, 2009 at 4:49 AM, Tom S <h3r3tic remove.mat.uni.torun.pl> wrote:


 Additionally, using such an approach breaks any possibility to use
 incremental compilation. So there's also the option of having a .lib per .d
 file and linking that all together. So I've just tried it:
 dmd -c Main.d
 dmd -lib Mod.d
 dmd -lib Mod2.d
 dmd Main.obj Mod.lib Mod2.d
 ... and OPTLINK crashed with the familiar MsgBox containing register values.


 I can confirm that this happens on both my laptop (single-core,
 single-threaded Pentium M) and my desktop.  Using imagecfg changes
 nothing.


Also, I'm using DMD 1.039 and OPTLINK 8.00.1 on both machines.
Feb 17 2009
prev sibling parent grauzone <none example.net> writes:
Jarrett Billingsley wrote:

 This is, for me, the number one blocker of most of my more complex
 code on Windows.  More than any DMD bug.  Tom S can vouch for it too,
 and Eldar sounds like he s about to give up on D after dealing with
 OPTLINK's crashes.


He's not the only one. I hope Walter is aware how serious the problem 
is. Although D is a really nice language, it becomes unusable with its 
shitty toolchain support. Same with "popular" bugs like forward 
referencing / circular module bugs.

If I tell someone about D, I always add a warning about how shitty the 
toolchain is. Then I hope they won't just ditch D if they hit compiler 
and linker bugs...

Even on Linux, where the GNU linker is used, there are some problems 
with linking. Sometimes, the linker will complain about either multiply 
defined symbols or missing ones. Or the compiled program segfaults... 
then you completely recompile it, and it works.
Feb 16 2009