www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 424] New: Unexpected OPTLINK Termination at EIP=0044C37B

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424

           Summary: Unexpected OPTLINK Termination at EIP=0044C37B
           Product: D
           Version: unspecified
          Platform: PC
               URL: http://www.digitalmars.com/d/archives/digitalmars/D/bugs
                    /306.html
        OS/Version: Windows
            Status: NEW
          Severity: blocker
          Priority: P1
         Component: DMD
        AssignedTo: bugzilla digitalmars.com
        ReportedBy: someanon yahoo.com


I have some generated D code in a single source file ~ 30K LOC. On Windows I
can compile it but cannot link it. (On Linux, there's no problem.)

Some web search leads me to the following, which seems related to the problem. 
I wonder if OPTLINK can be fixed now.

http://www.digitalmars.com/d/archives/digitalmars/D/bugs/306.html

Sun, 30 May 2004 20:54:18 -0700 "Walter" <newshound xx digitalmars.com> 
writes:

This happens when there are more than 16,000 fixups in a single .obj.
Unfortunately, this isn't fixable at the moment.


http://www.digitalmars.com/d/archives/digitalmars/D/bugs/8283.html
http://www.digitalmars.com/d/archives/digitalmars/D/bugs/306.html
http://www.digitalmars.com/d/archives/digitalmars/D/41323.html


--
Oct 10 2006
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424


bugzilla digitalmars.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|blocker                     |normal





While important, I don't think it's a blocker. 30,000 lines of code in one
module is unusually large, and can be split into smaller source files.


--
Oct 14 2006
prev sibling next sibling parent reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424








 While important, I don't think it's a blocker. 30,000 lines of code in one
 module is unusually large, and can be split into smaller source files.
 


It is a blocker at least for one user :-)

OK, let me elaborate a little more where I come from, and all the problems I've
encountered:

When I started to use D on a hobby project, I organize the source code into
modules as everyone typically does, but because of bug 386:

http://d.puremagic.com/issues/show_bug.cgi?id=386

I was so frustrated that I cannot get my code compile separately, and the
compiler error message is useless, each time I spent hours and hours on them
and cannot get them fixed. So I write some script to concat all my source files
(togather with some generated ones) into a single huge .d file, and comment out
all the 'module' and 'import' statements.  Then everything works.

After bug 386 is fixed. I tried to compile each source file separately again,
now I no longer have 'import conflicts' problems, but I still cannot get many
of the file compiled, the compiler just hangs: I guess it run into some
infinite loop.  But unfortunately I cannot isolate the issue to send you a test
case.

(search "infinite loop" on digitalmars.com will show some user report: e.g.
http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=42668
But I'm not sure, it's the same issue as mine.)

Then I went back to my original approach to generate a huge source file (and it
works); but maybe as my project grows, at certain point, it stops to link on
Windows.  (On Linux, it's fine).

And this is a OPTLINK problem. I hope you can fix it. Or as a workaround: if
the generate obj is too large (> 16,000 fixups, I really don't understand what
does it mean), can the compiler split the obj into several obj files? e.g.
foo1.obj, foo2.obj, ...

If you ask the user the split the generated source files, first it's tedious,
and it maybe run into some compiler problem, like the import one again.

So please fix this issue:

1) remove the (> 16,000 fixups) constraint.  It is a bug anyway.
2) or, the compiler automatically generate multiple .obj files, if (> 16,000
fixups)

Thanks.


--
Oct 15 2006
parent Walter Bright <newshound digitalmars.com> writes:
d-bugmail puremagic.com wrote:

 After bug 386 is fixed. I tried to compile each source file separately again,
 now I no longer have 'import conflicts' problems, but I still cannot get many
 of the file compiled, the compiler just hangs: I guess it run into some
 infinite loop.  But unfortunately I cannot isolate the issue to send you a test
 case.


Here's how to do it. Make a copy of your project. Create a .bat file 
that compiles it. Verify that it hangs. Now, keep deleting code until 
you have the minimum that produces the hang.

I've never seen one yet that couldn't be isolated down this way, 
including from people who swore it couldn't be done <g>. Don't worry 
about intelligently deciding what lines to delete. Delete something, 
even if it's just one line.
Oct 15 2006
prev sibling next sibling parent reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






Another "infinite loop" user report, maybe related:

http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=D.gnu&artnum=2134


--
Oct 15 2006
parent Walter Bright <newshound digitalmars.com> writes:
d-bugmail puremagic.com wrote:


 Another "infinite loop" user report, maybe related:
 
 http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=D.gnu&artnum=2134


Doesn't infinite loop on current dmd.
Oct 15 2006
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






Let's not clutter this report up with more than one issue.  Please make sure a
seprate bug is filed for the infinite loop issue.  This report is strictly
about th eoptlink fixup limit.

Walter, can you please clarify where the limit exists and if it's authored by
you or microsoft?  I don't do any windows development so I'm unfamiliar with
the pieces.

Filer, would you please attach to this report an example .d file that fails
with the error message?

Rather than discussing why or how this programming practice might be ill
advised, can we talk about where the problem is and how or if it can be fixed. 
Each barrier to the use of D, whatever the reason, is a barrier that should at
least be addressed in a FAQ and not dismissed as a bad practice.

Thanks,
Brad


--
Oct 16 2006
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






any chance this will be fixed soon? it's a bug anyways.


--
Nov 07 2006
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






can this one be fixed before 1.0 (Jan 01 2007)?


--
Dec 27 2006
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






No update?


--
Mar 22 2007
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






Tried the new linker:

http://ftp.digitalmars.com/Digital_Mars_C++/Patch/beta.zip 

In release mode (-d -O -inline -release), it works, and generate the exe file.
But when running the program (using wxd, 0.09 version), the main UI window
doesn't show up.

In debug mode (-d -g -gc -debug), it errors out:
Unexpected OPTLINK Termination at EIP=00412793

GDC (under MingW32) works in both debug & release mode on the same system.


--
Apr 06 2007
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






I think if you can post your object file which makes optlink fails could let
Walter easier to debug this issue


--
Sep 12 2007
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424








 While important, I don't think it's a blocker. 30,000 lines of code in one
 module is unusually large, and can be split into smaller source files.


I believe I'm running into this too, and my files have a reasonable number of
lines.  However, the code is structured so that *everything* is templated, and
doesn't get instantiated until main.d.  This is a very common way to write
heavily generic code.  The actual library in question in which this is
happening OpenMeshD (http://www.dsource.org/projects/openmeshd), which is a
port of a  C++ template library to D.

Why I think I'm seeing this same problem:

1) Optlink is dying and popping up a MsgBox saying "Unexpected OPTLINK
Termination at EIP=0044C37B"  The EIP number is different that the one
previously reported, but I'm guessing that doesn't matter (just a random memory
address?).

2) The thing that triggered the crashing was adding a function with a big block
of type-checking case statements like this:
   else if (ti==typeid(VectorT!(byte,   2))) { 
       renderer_ = new PropRenderer!(VectorT!(byte,   2)); 
   }

   So it's causing a lot of templates to be instantiated, which could be
pushing the number of "fixups" in main.d over the 16K limit.

3) If I start commenting out those "else if" lines one by one, eventually
compilation will work again. It doesn't seem to matter which lines I comment
out.  If I get rid of enough of them, then it compiles.

4) Compiling with -O -release instead of -g -debug allows me to uncomment a few
more of the lines.

5) "main.obj" is between 1-2MB even in release mode.


fixups then with something.

I'm using DMD 1.023 on Windows XP.

I've checked the code into the OpenMesh/D repository in its buggy state so it
can be checked out for debugging.

To reproduce, check out revision 111 of OpenMesh from the svn repository:

If you don't have dsss installed, install it first
(http://www.dsource.org/projects/dsss)

Then:

$ mkdir scratch
$ cd scratch
$ svn co -r111 http://svn.dsource.org/projects/openmeshd/trunk .
$ cd OpenMeshD\OpenMesh\Apps\GLViewer
$ dsss net deps 
$ dsss build -version=OPTLINKIsMyFriend

If there are complaints about Helix things, you may need to go to the Helix dir
and do a "dsss build && dsss install"


--
Nov 14 2007
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424


wbaxter gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |major





Upgrading the severity because I just ran into this again adding to an
unrelated part of the main .d file.  Moving the added code to a different file
fixed it for now but that may not always be so easy to do if the code needs to
be in the generic, templated part of things.  Then it seems it will end up
instantiating in the main .obj file no matter where I put it.

Also further rearrangement of the code started triggering the "previous
definition different blahblah_initZ" bug.

So, yeh, this is a bad one, and it affects more than just code with gigantic
source files.   It appears to affect any template library with a few tens of
thousands of lines of code.


--
Nov 14 2007
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






Fun fun!  Just hit this one again.

See thread digitalmars.com digitalmars.D:67093


--
Mar 04 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424


wbaxter gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Unexpected OPTLINK          |Unexpected OPTLINK
                   |Termination at EIP=0044C37B |Termination at EIP=0044C37B
                   |                            |(too many fixups)





Changed title to include keyword "fixups" for easy finding.


--
Jun 26 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424









--
Nov 12 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424









 





--
Nov 13 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






Exactly


--
Nov 13 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






Just got bitten by this one too.  Voted!


--
Nov 20 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424








 Just got bitten by this one too.  Voted!
 


Unfortunately I think the only real solution for this is a new linker.
Just out of curiosity, in your case is it the lots of templates getting
instantiated in a single file thing?  Or did you find some other way to trigger
it?


--
Nov 20 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424










 Just got bitten by this one too.  Voted!
 


 
 Unfortunately I think the only real solution for this is a new linker.
 Just out of curiosity, in your case is it the lots of templates getting
 instantiated in a single file thing?  Or did you find some other way to trigger
 it?
 


I'm pretty sure that's what's causing the issue I have.  It compiles with n
template instantiations, and fails with n + 1.  Doesn't matter which ones I
remove or add.  And I have a *lot* of them in one file.  And they're the deeply
nested variadic code-generating kind.

I do horrible things.


--
Nov 20 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424







 Unfortunately I think the only real solution for this is a new linker.


I just thought there maybe a simple solution to implement: add a switch to the
DMD compiler, e.g "-mo", means output multiple object files.

So for file foo.d the compiler will output 
foo1.o
foo2.o
foo3.o 
....

each only contains a limited number of fixups, so in this way, the user can add
all these object files to the linker command line. This should work.

What do you think? Walter?


--
Feb 10 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






Actually even without such switch, the compiler should split the object files "
when there are more than 16,000 fixups in a single .obj" -- after all why the
compiler should generate such obj file, if the linker cannot link it, while the
compiler have all the necessary info to take the right action. (I would
consider this is a minor compiler bug in this sense ;-)

And if this happened with no user option specified, the compiler should also
print a friendly message, saying: "too many fixups, multiple object file
foo1.obj, foo2.obj ... fooN.obj generated, please modify your link command".




 Unfortunately I think the only real solution for this is a new linker.


 
 I just thought there maybe a simple solution to implement: add a switch to the
 DMD compiler, e.g "-mo", means output multiple object files.
 
 So for file foo.d the compiler will output 
 foo1.o
 foo2.o
 foo3.o 
 ....
 
 each only contains a limited number of fixups, so in this way, the user can add
 all these object files to the linker command line. This should work.
 
 What do you think? Walter?
 



--
Feb 10 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






I disagree about multiple output obj files from one source. This would make
linking much harder (e.g. - compilation using makefile). I wouldn't like the
build process to depend from some third party tool that would parse compiler
output and adjust my linking options.


--
Feb 10 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424








 I disagree about multiple output obj files from one source. This would make
 linking much harder (e.g. - compilation using makefile). I wouldn't like the
 build process to depend from some third party tool that would parse compiler
 output and adjust my linking options.
 


Then have a compiler switch then "-mo+" means on, and "-mo-" turn it off.

I just wonder if you have ever bitten by this bug, what else you can do? The
user have control of his own build process, but how the user overcome this
linker bug?


--
Feb 10 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






The compiler already has a switch to generate multiple obj files from one
source file: -multiobj. I use it for debugging. But it would be a royal pain to
use if there's a lot.

A better way is to compile to a library with the -lib switch. That splits the
module into a zillion obj files, and stuffs them all into the library. Then,
link with the library (the main() program will still have to be separate).


--
Feb 16 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424








 The compiler already has a switch to generate multiple obj files from one
 source file: -multiobj. I use it for debugging. But it would be a royal pain to
 use if there's a lot.


Yes, tried that: because there are sequence dependency on the order of the
object file, one cannot give "foo*.obj", the linker complains lots of
unresolved symbol.

I just wonder if the linker can be changed to be ".obj" order insensitive. Yes,
it's a royal pain to let the user try to figure out the correct ".obj"
sequence, but the linker should have all the information to find the unresolved
symbols in those .objs.



 
 A better way is to compile to a library with the -lib switch. That splits the
 module into a zillion obj files, and stuffs them all into the library. Then,
 link with the library (the main() program will still have to be separate).
 


Also tried, when linking with the generated .lib file, it also gives
"Unexpected OPTLINK Termination" error, not sure it's the same one. But I guess
so.


--
Feb 17 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






If someone can attach to this an object file with too many fixups, that would
be helpful. Or email it to me.


--
Mar 07 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424


Jarrett Billingsley <jarrett.billingsley gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jarrett.billingsley gmail.c
                   |                            |om





2009-06-20 23:54:07 PDT ---
Open a windows command prompt and run the following line:

for /L %i in (1,1,17000) do  echo void func%i(){} >> test.txt

This creates test.txt with 17,000 functions.

Then create test.d in the same dir:

module test;
mixin(import("test.txt"));
void main() {}

Now, compile *with the following commandline*.

dmd test.d -g -J.

This causes OPTLINK v8.00.1 to crash with EIP=00412793 as sa mentioned.  

For more testing, delete test.txt and run the shell line above with different
values.  16000 does not crash.  Interestingly, 16500 neither succeeds nor
fails; the linker apparently hangs and does not use any CPU.

If you want a real-world test, you're going to have to install some third-party
D libraries.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jun 20 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424






2009-07-27 08:42:22 PDT ---
I know you responded with a "thanks" on the newsgroup, Walter, but.. is there
any followup?  Were you able to reproduce it?  Have you investigated the
problem?  Is there any chance of it being fixed?  Is it a fundamental limit
that can't be changed?  Is this on the back burner?  Is this going to magically
appear in the next release without warning?

Cmooon, communicate!  What is the story?  There are a lot of people who've run
into this bug, a lot who want to see it fixed, and a simple repro.  What more
do you want or need?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jul 27 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424




10:38:43 PST ---
Optlink is written entirely in rather impenetrable assembler code, and is
resistant to understanding and modification. Hence, over the last few months
I've been very slowly converting it to C, function by function.

One might ask, why not convert it to D? The answer is that I don't have a good
test suite for optlink, so I have to be very very careful to not make a mistake
in the translation. That means do one function at a time, rebuild, and retest,
which means the compiled C code has to match the segment, naming and calling
conventions used in optlink. I made a custom version of the dmc compiler to do
this. Also, C can be made to work without any runtime library support at all,
and since optlink does not use the C runtime library, this is useful.

Once it is in C and working, it will be trivial to translate it to D and start
rewriting it.

Anyhow, during this process I stumbled upon what the problem was. Optlink was
apparently trying to account for some Borland obscure extension to the OMF.
Remove this, and it works, although presumably it will no longer link Borland
object files (who cares!).

The fix will go out in the next update, if you need it sooner please email me.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 03 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424




2009-11-05 10:03:36 PST ---



 The fix will go out in the next update, if you need it sooner please email me.


Holy--

Joy upon joys! This is REALLY good news. I can't tell you how happy you'll make
a lot of people with this.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 05 2009
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=424


Walter Bright <bugzilla digitalmars.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED



11:29:04 PST ---
Fixed dmd 1.051 and 2.036

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 06 2009