• Jonathan Marler (20/20) Jul 29 2014 I'm attempting to fix
• Walter Bright (20/36) Jul 29 2014 I'm really the only person, except a couple others who have submitted PR...
• w0rp (4/10) Jul 30 2014 The solution there is to write a comprehensive test suite. Then
• Jonathan Marler (12/12) Jul 30 2014 It looks like the easiest way to fix the bug is to change the
• Kagamin (1/1) Jul 30 2014 Making dmd generate coff would make more sense.
• Rikki Cattermole (4/5) Jul 30 2014 +1
• Joakim (7/12) Jul 30 2014 What makes it craziest is that there's a COFF32 branch lying
• Rikki Cattermole (7/18) Jul 30 2014 If we obsoleted the OMF format output we would need to have a free and
• Joakim (7/33) Jul 30 2014 I don't think dmd comes with a COFF64 linker now, users are just
• Rikki Cattermole (7/38) Jul 30 2014 I'm in agreement with you about just getting people to install it. But
• w0rp (9/44) Jul 30 2014 I think it's important to ship with a linker without requiring
• Daniel Murphy (6/13) Jul 30 2014 FWIW this was the plan when I started ylink. I wanted to make a linker ...
• w0rp (4/20) Jul 30 2014 I didn't know about ylink. The prospect of having a free software
• Daniel Murphy (5/8) Jul 30 2014 I would estimate it's a smaller effort than DDMD. (win32 + omf + cv4)
• Jonathan Marler (23/23) Jul 30 2014 I like the discussion. I do want to remind everyone that OPTLINK
• Kagamin (3/6) Jul 30 2014 Wouldn't it be easier to optimize a linker written in D than
• Jonathan Marler (17/24) Jul 31 2014 No matter how much you optimize your D code, you will only ever
• Kagamin (10/17) Jul 31 2014 Such micro-optimizations give only marginal contribution to
• Daniel Murphy (9/22) Jul 31 2014 Optlink simply doesn't do enough. It doesn't support modern debug info
• Jacob Carlborg (5/7) Aug 06 2014 Just for the record. Currently, Windows in the only platform where this
• Trass3r (3/8) Jul 30 2014 Anybody seriously programming on Windows uses VS anyway.
• Kagamin (3/5) Jul 30 2014 GNU binutils should do.
• Trass3r (1/4) Jul 30 2014 Same procedure as every year.

"Jonathan Marler" <johnnymarler gmail.com> writes:

I'm attempting to fix 
https://issues.dlang.org/show_bug.cgi?id=4831.  I've been 
debugging the optlink assembly and getting familiar with the 
code.  I have a couple questions though:

1. If I have any questions in the future about optlink who and 
where do I ask these questions? For now I'm posting to this forum 
because I'm not sure where else to send them.

2. Why are the assembly labels in this format: L<number>$?  I ask 
because I changed the labels for some functions so when I stepped 
through the assembly I had a better idea of what the code was 
doing.  So I'm not sure if I should leave the new labels or go 
back to the old L<number>$ labels.

3. Is there any documentation for developers who would like to 
contribute to optlink?  I'd like to make sure I'm following any 
style rules or guidelines.

Any information about developing for optlink would be helpful.  I 
haven't found much information online.  Thanks for any help, I'd 
like to start contributing to the D tools, its a crime to let all 
these bugs live on for so long when they are making adopting this 
language a hindrence for others.

Walter Bright <newshound2 digitalmars.com> writes:

On 7/29/2014 10:53 PM, Jonathan Marler wrote:
 I'm attempting to fix https://issues.dlang.org/show_bug.cgi?id=4831.  I've been
 debugging the optlink assembly and getting familiar with the code.  I have a
 couple questions though:

 1. If I have any questions in the future about optlink who and where do I ask
 these questions? For now I'm posting to this forum because I'm not sure where
 else to send them.

I'm really the only person, except a couple others who have submitted PRs 
against optlink.


 2. Why are the assembly labels in this format: L<number>$?  I ask because I
 changed the labels for some functions so when I stepped through the assembly I
 had a better idea of what the code was doing.  So I'm not sure if I should
leave
 the new labels or go back to the old L<number>$ labels.

No known reason.


 3. Is there any documentation for developers who would like to contribute to
 optlink?  I'd like to make sure I'm following any style rules or guidelines.

The fundamental problem with fixing optlink is there is essentially no test 
suite. This means that any fixes to it need to be surgical - as little code 
modified as practical, and pretty great care in doing it. Wholesale refactoring 
"just because" or for "cleanup" are out of the question. Wholesale reformatting 
is also out of the question.

Once what a function does is figured out, and its inputs and outputs
identified, 
adding comments to that effect is appreciated. Most functions in Optlink do not 
identify what registers are used as input, what registers are used for output, 
and which registers must be preserved. Some functions even return results in 
flags. There is no consistency. To identify these, one must look at every call 
of that function.


 Any information about developing for optlink would be helpful.  I haven't found
 much information online.  Thanks for any help, I'd like to start contributing
to
 the D tools, its a crime to let all these bugs live on for so long when they
are
 making adopting this language a hindrence for others.

I appreciate you stepping up to make a difference on this.

What I've done to fix bugs in Optlink is to first identify where in the code 
things go wrong. Then, I convert that section of code to C. The C code, line by 
line, mirrors the assembler. Then I fix the C code. This makes things much 
easier because I can use printf, etc., in the C code.

"w0rp" <devw0rp gmail.com> writes:

On Wednesday, 30 July 2014 at 06:54:52 UTC, Walter Bright wrote:
 The fundamental problem with fixing optlink is there is 
 essentially no test suite. This means that any fixes to it need 
 to be surgical - as little code modified as practical, and 
 pretty great care in doing it. Wholesale refactoring "just 
 because" or for "cleanup" are out of the question. Wholesale 
 reformatting is also out of the question.

The solution there is to write a comprehensive test suite. Then 
it can be refactored. I'm not saying it's an easy solution, it's 
probably a major pain, but that's what it is.

"Jonathan Marler" <johnnymarler gmail.com> writes:

It looks like the easiest way to fix the bug is to change the 
get_filename function to support non HPFS characters.  I'm 
guessing this was originally written to run exclusively on HPFS 
systems?  Is this tool still suppose to support HPFS file 
systems?  If so maybe we could add a runtime check to see what 
kind of fs the program is being run on (or a compile time check 
if we are guaranteed that each system with a different file 
system will use a different build, I'm guessing this is not the 
case.)  If HPFS does not need to be supported then the fix is 
really easy.

I also found a buffer overrun bug in the get_filename function 
(if your filename does not have an ending quote).

"Kagamin" <spam here.lot> writes:

Making dmd generate coff would make more sense.

Rikki Cattermole <alphaglosined gmail.com> writes:

On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.

+1
Most of the code should already be present in dmd, which makes it far 
crazier not to.

"Joakim" <dlang joakim.airpost.net> writes:

On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole 
wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.

 +1
 Most of the code should already be present in dmd, which makes 
 it far crazier not to.

What makes it craziest is that there's a COFF32 branch lying 
around that nobody merges:

http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com

It would be a far better use of Jonathan's time to get COFF32 
merged and obsolete Optlink altogether.

Rikki Cattermole <alphaglosined gmail.com> writes:

On 30/07/2014 8:58 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.

 +1
 Most of the code should already be present in dmd, which makes it far
 crazier not to.

 What makes it craziest is that there's a COFF32 branch lying around that
 nobody merges:

 http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com


 It would be a far better use of Jonathan's time to get COFF32 merged and
 obsolete Optlink altogether.

If we obsoleted the OMF format output we would need to have a free and 
distributed with PE-COFF linker. If we can do this, I think Walter might 
go along with it.

Unless of course we could convince Microsoft to have a download just for 
the linker. We could download that in e.g. the installer. Would be 
better than a full install.

"Joakim" <dlang joakim.airpost.net> writes:

On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole 
wrote:
 On 30/07/2014 8:58 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole 
 wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.

 +1
 Most of the code should already be present in dmd, which 
 makes it far
 crazier not to.

 What makes it craziest is that there's a COFF32 branch lying 
 around that
 nobody merges:

 http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com


 It would be a far better use of Jonathan's time to get COFF32 
 merged and
 obsolete Optlink altogether.

 If we obsoleted the OMF format output we would need to have a 
 free and distributed with PE-COFF linker. If we can do this, I 
 think Walter might go along with it.

 Unless of course we could convince Microsoft to have a download 
 just for the linker. We could download that in e.g. the 
 installer. Would be better than a full install.

I don't think dmd comes with a COFF64 linker now, users are just 
told to install Visual Studio or the Windows SDK for a linker.  
No reason you can't do the same with COFF32.  Optlink can stick 
around with OMF for a couple releases.  I suspect nobody would 
use it when given the choice of COFF32 support.

Rikki Cattermole <alphaglosined gmail.com> writes:

On 30/07/2014 9:17 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole wrote:
 On 30/07/2014 8:58 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.

 +1
 Most of the code should already be present in dmd, which makes it far
 crazier not to.

 What makes it craziest is that there's a COFF32 branch lying around that
 nobody merges:

 http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com



 It would be a far better use of Jonathan's time to get COFF32 merged and
 obsolete Optlink altogether.

 If we obsoleted the OMF format output we would need to have a free and
 distributed with PE-COFF linker. If we can do this, I think Walter
 might go along with it.

 Unless of course we could convince Microsoft to have a download just
 for the linker. We could download that in e.g. the installer. Would be
 better than a full install.

 I don't think dmd comes with a COFF64 linker now, users are just told to
 install Visual Studio or the Windows SDK for a linker. No reason you
 can't do the same with COFF32.  Optlink can stick around with OMF for a
 couple releases.  I suspect nobody would use it when given the choice of
 COFF32 support.

I'm in agreement with you about just getting people to install it. But 
the problem is, its not out of the box enough for Walters liking and I'm 
partially agreeing with him on this. At the end of the day, we don't 
really want to tell new people you have to also install x.

Also not to forget that we could also remove OUR implib versions of 
Windows Dlls. That would solve a lot of issues for people I bet.

"w0rp" <devw0rp gmail.com> writes:

On Wednesday, 30 July 2014 at 09:17:05 UTC, Joakim wrote:
 On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole 
 wrote:
 On 30/07/2014 8:58 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole 
 wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.

 +1
 Most of the code should already be present in dmd, which 
 makes it far
 crazier not to.

 What makes it craziest is that there's a COFF32 branch lying 
 around that
 nobody merges:

 http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com


 It would be a far better use of Jonathan's time to get COFF32 
 merged and
 obsolete Optlink altogether.

 If we obsoleted the OMF format output we would need to have a 
 free and distributed with PE-COFF linker. If we can do this, I 
 think Walter might go along with it.

 Unless of course we could convince Microsoft to have a 
 download just for the linker. We could download that in e.g. 
 the installer. Would be better than a full install.

 I don't think dmd comes with a COFF64 linker now, users are 
 just told to install Visual Studio or the Windows SDK for a 
 linker.  No reason you can't do the same with COFF32.  Optlink 
 can stick around with OMF for a couple releases.  I suspect 
 nobody would use it when given the choice of COFF32 support.

I think it's important to ship with a linker without requiring 
any further installation. One of the things that helped me to 
learn D was being able to download DMD and run RDMD on Windows 
without installing anything else. It's not obvious to very new 
users who don't come from a Windows C++ background where you can 
get a gratis Microsoft compiler or how to configure it. If we 
were going to switch to COFF32, it would be a big bonus to ship a 
COFF32 linker in the installer.

"Daniel Murphy" <yebbliesnospam gmail.com> writes:

"w0rp"  wrote in message news:sinwmhzuvhmevqtunxms forum.dlang.org...

 I think it's important to ship with a linker without requiring any further 
 installation. One of the things that helped me to learn D was being able 
 to download DMD and run RDMD on Windows without installing anything else. 
 It's not obvious to very new users who don't come from a Windows C++ 
 background where you can get a gratis Microsoft compiler or how to 
 configure it. If we were going to switch to COFF32, it would be a big 
 bonus to ship a COFF32 linker in the installer.

FWIW this was the plan when I started ylink.  I wanted to make a linker that 
could link omf and coff together, so I could tackle the object-format and 
runtime-format transitions independently.  It does support mscoff32 hello 
world now, but not much more.  Until it has debug info support it's not a 
viable replacement for optlink unfortunately. 

"w0rp" <devw0rp gmail.com> writes:

On Wednesday, 30 July 2014 at 13:03:30 UTC, Daniel Murphy wrote:
 "w0rp"  wrote in message 
 news:sinwmhzuvhmevqtunxms forum.dlang.org...

 I think it's important to ship with a linker without requiring 
 any further installation. One of the things that helped me to 
 learn D was being able to download DMD and run RDMD on Windows 
 without installing anything else. It's not obvious to very new 
 users who don't come from a Windows C++ background where you 
 can get a gratis Microsoft compiler or how to configure it. If 
 we were going to switch to COFF32, it would be a big bonus to 
 ship a COFF32 linker in the installer.

 FWIW this was the plan when I started ylink.  I wanted to make 
 a linker that could link omf and coff together, so I could 
 tackle the object-format and runtime-format transitions 
 independently.  It does support mscoff32 hello world now, but 
 not much more.  Until it has debug info support it's not a 
 viable replacement for optlink unfortunately.

I didn't know about ylink. The prospect of having a free software 
linker for D on Win32 written in D does sound attractive. I 
assume it would be a lot of work to make it acceptable for usage.

"Daniel Murphy" <yebbliesnospam gmail.com> writes:

"w0rp"  wrote in message news:vnaffnibgvtmqeuhzczx forum.dlang.org...

 I didn't know about ylink. The prospect of having a free software linker 
 for D on Win32 written in D does sound attractive. I assume it would be a 
 lot of work to make it acceptable for usage.

I would estimate it's a smaller effort than DDMD.  (win32 + omf + cv4)

We really don't need to match the performance and memory usage of optlink.

The best part is we can then potentially re-use parts of it inside dmd, 
currently dmd has its own lib generator for each format. 

"Jonathan Marler" <johnnymarler gmail.com> writes:

I like the discussion.  I do want to remind everyone that OPTLINK 
is very fast and switching to a different linker will likely 
result performance hit.  There are advantages to using COFF as it 
seems more compilers use that format and D would be more 
interoperable with other compilers and languages.  I think every 
peice of software has it's own goals which determine what 
tradeoffs to chose.  In my view, the linker should 1) always work 
and 2) be as fast as possible.  OPTLINK passes number 2 with 
flying colors but is does have bugs.  This is obviously due to 
the fact that it is written in assembly but keep in mind that the 
performance of the linker affects everyone who uses D.  Any 
performance hit on the linker will be seen in every single build 
of any D program. Adding 1 second to link time will add 1 second 
to build time since this process cannot be parallelized (as far 
as I know).

That being said, if it were up to me I wouldn't abandon OPTLINK 
so quickly.  I would spend time creating a test suite for 
OPTLINK, and try to comment the code a little better to encourage 
other developers to contribute.  Then I would consider adding 
support in DMD to generate COFF output files for those who want 
to link D object files with another linker. I wouldn't make this 
a huge priority though as one can use an OMF to COFF converter if 
they really needed it.

"Kagamin" <spam here.lot> writes:

On Wednesday, 30 July 2014 at 19:50:35 UTC, Jonathan Marler wrote:
 I like the discussion.  I do want to remind everyone that 
 OPTLINK is very fast and switching to a different linker will 
 likely result performance hit.

Wouldn't it be easier to optimize a linker written in D than 
tinker with optlink?

"Jonathan Marler" <johnnymarler gmail.com> writes:

On Thursday, 31 July 2014 at 06:46:19 UTC, Kagamin wrote:
 On Wednesday, 30 July 2014 at 19:50:35 UTC, Jonathan Marler 
 wrote:
 I like the discussion.  I do want to remind everyone that 
 OPTLINK is very fast and switching to a different linker will 
 likely result performance hit.

 Wouldn't it be easier to optimize a linker written in D than 
 tinker with optlink?

No matter how much you optimize your D code, you will only ever 
be able to use a subset of what assembly can do.  That being 
said, writing perfect assembly code is impossible, so we rely on 
the compiler to take our higher level concepts and have it decide 
on the best way to optimize.  The beauty of higher level 
languages is you can make high level changes resulting an a 
cascade of optimizations.  But the downside is sometimes changes 
can have cascading adverse affects as well.  One of the things I 
like about D is that it does a much better job of allowing the 
programmer to tell the compiler the "right" information it needs 
to know how to optimize.

I would like to see a linker written in D and see how it compares 
to optlink.  But I would reserve making a decision on moving to 
another linker until I got some solid performance data.  Some 
performance data for a D linker compiled with all three D 
compilers as well.

"Kagamin" <spam here.lot> writes:

On Thursday, 31 July 2014 at 07:07:27 UTC, Jonathan Marler wrote:
 No matter how much you optimize your D code, you will only ever 
 be able to use a subset of what assembly can do.

Such micro-optimizations give only marginal contribution to 
performance, which is mostly determined by choice of algorithms. 
The approach to write everything in assembly with manual 
optimizations is obsolete for many years, at least for desktop 
platforms. This is also one of the reasons, why optlink is 
translated to C.

 I would like to see a linker written in D and see how it 
 compares to optlink.  But I would reserve making a decision on 
 moving to another linker until I got some solid performance 
 data.  Some performance data for a D linker compiled with all 
 three D compilers as well.

I only wanted to say, no matter what performance data we get, the 
linker can be optimized by choosing faster algorithms without 
tinkering with assembler.

"Daniel Murphy" <yebbliesnospam gmail.com> writes:

"Jonathan Marler"  wrote in message 
news:xrpdzjuaxtdhyfhpsgcu forum.dlang.org...

 I like the discussion.  I do want to remind everyone that OPTLINK is very 
 fast and switching to a different linker will likely result performance 
 hit.  There are advantages to using COFF as it seems more compilers use 
 that format and D would be more interoperable with other compilers and 
 languages.  I think every peice of software has it's own goals which 
 determine what tradeoffs to chose.  In my view, the linker should 1) 
 always work and 2) be as fast as possible.  OPTLINK passes number 2 with 
 flying colors but is does have bugs.  This is obviously due to the fact 
 that it is written in assembly but keep in mind that the performance of 
 the linker affects everyone who uses D.  Any performance hit on the linker 
 will be seen in every single build of any D program. Adding 1 second to 
 link time will add 1 second to build time since this process cannot be 
 parallelized (as far as I know).

Optlink simply doesn't do enough.  It doesn't support modern debug info 
formats.  It doesn't support mixing omf and coff object files (this is 
probably the most requested feature, because it would allow using the stock 
import libraries).

Worst of all, optlink is hostile to change.

win32 is dying, and optlink will certainly die with it.  Having a more 
accessible linker will let us adapt much faster. 

Jacob Carlborg <doob me.com> writes:

On 2014-07-30 14:51, w0rp wrote:

 I think it's important to ship with a linker without requiring any
 further installation.

Just for the record. Currently, Windows in the only platform where this 
is true.

-- 
/Jacob Carlborg

"Trass3r" <un known.com> writes:

 I don't think dmd comes with a COFF64 linker now, users are 
 just told to install Visual Studio or the Windows SDK for a 
 linker.  No reason you can't do the same with COFF32.  Optlink 
 can stick around with OMF for a couple releases.  I suspect 
 nobody would use it when given the choice of COFF32 support.

Anybody seriously programming on Windows uses VS anyway.
But at least for 32 bit http://lld.llvm.org/windows_support.html 
could be packaged.

"Kagamin" <spam here.lot> writes:

On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole 
wrote:
 If we obsoleted the OMF format output we would need to have a 
 free and distributed with PE-COFF linker.

GNU binutils should do.

"Trass3r" <un known.com> writes:

 What makes it craziest is that there's a COFF32 branch lying 
 around that nobody merges:

 http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com

Same procedure as every year.