www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Optlink Contribution

reply "Jonathan Marler" <johnnymarler gmail.com> writes:
I'm attempting to fix 
https://issues.dlang.org/show_bug.cgi?id=4831.  I've been 
debugging the optlink assembly and getting familiar with the 
code.  I have a couple questions though:

1. If I have any questions in the future about optlink who and 
where do I ask these questions? For now I'm posting to this forum 
because I'm not sure where else to send them.

2. Why are the assembly labels in this format: L<number>$?  I ask 
because I changed the labels for some functions so when I stepped 
through the assembly I had a better idea of what the code was 
doing.  So I'm not sure if I should leave the new labels or go 
back to the old L<number>$ labels.

3. Is there any documentation for developers who would like to 
contribute to optlink?  I'd like to make sure I'm following any 
style rules or guidelines.

Any information about developing for optlink would be helpful.  I 
haven't found much information online.  Thanks for any help, I'd 
like to start contributing to the D tools, its a crime to let all 
these bugs live on for so long when they are making adopting this 
language a hindrence for others.
Jul 29 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/29/2014 10:53 PM, Jonathan Marler wrote:
 I'm attempting to fix https://issues.dlang.org/show_bug.cgi?id=4831.  I've been
 debugging the optlink assembly and getting familiar with the code.  I have a
 couple questions though:

 1. If I have any questions in the future about optlink who and where do I ask
 these questions? For now I'm posting to this forum because I'm not sure where
 else to send them.
I'm really the only person, except a couple others who have submitted PRs against optlink.
 2. Why are the assembly labels in this format: L<number>$?  I ask because I
 changed the labels for some functions so when I stepped through the assembly I
 had a better idea of what the code was doing.  So I'm not sure if I should
leave
 the new labels or go back to the old L<number>$ labels.
No known reason.
 3. Is there any documentation for developers who would like to contribute to
 optlink?  I'd like to make sure I'm following any style rules or guidelines.
The fundamental problem with fixing optlink is there is essentially no test suite. This means that any fixes to it need to be surgical - as little code modified as practical, and pretty great care in doing it. Wholesale refactoring "just because" or for "cleanup" are out of the question. Wholesale reformatting is also out of the question. Once what a function does is figured out, and its inputs and outputs identified, adding comments to that effect is appreciated. Most functions in Optlink do not identify what registers are used as input, what registers are used for output, and which registers must be preserved. Some functions even return results in flags. There is no consistency. To identify these, one must look at every call of that function.
 Any information about developing for optlink would be helpful.  I haven't found
 much information online.  Thanks for any help, I'd like to start contributing
to
 the D tools, its a crime to let all these bugs live on for so long when they
are
 making adopting this language a hindrence for others.
I appreciate you stepping up to make a difference on this. What I've done to fix bugs in Optlink is to first identify where in the code things go wrong. Then, I convert that section of code to C. The C code, line by line, mirrors the assembler. Then I fix the C code. This makes things much easier because I can use printf, etc., in the C code.
Jul 29 2014
next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Wednesday, 30 July 2014 at 06:54:52 UTC, Walter Bright wrote:
 The fundamental problem with fixing optlink is there is 
 essentially no test suite. This means that any fixes to it need 
 to be surgical - as little code modified as practical, and 
 pretty great care in doing it. Wholesale refactoring "just 
 because" or for "cleanup" are out of the question. Wholesale 
 reformatting is also out of the question.
The solution there is to write a comprehensive test suite. Then it can be refactored. I'm not saying it's an easy solution, it's probably a major pain, but that's what it is.
Jul 30 2014
prev sibling parent "Jonathan Marler" <johnnymarler gmail.com> writes:
It looks like the easiest way to fix the bug is to change the 
get_filename function to support non HPFS characters.  I'm 
guessing this was originally written to run exclusively on HPFS 
systems?  Is this tool still suppose to support HPFS file 
systems?  If so maybe we could add a runtime check to see what 
kind of fs the program is being run on (or a compile time check 
if we are guaranteed that each system with a different file 
system will use a different build, I'm guessing this is not the 
case.)  If HPFS does not need to be supported then the fix is 
really easy.

I also found a buffer overrun bug in the get_filename function 
(if your filename does not have an ending quote).
Jul 30 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
Making dmd generate coff would make more sense.
Jul 30 2014
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.
+1 Most of the code should already be present in dmd, which makes it far crazier not to.
Jul 30 2014
parent reply "Joakim" <dlang joakim.airpost.net> writes:
On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole 
wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.
+1 Most of the code should already be present in dmd, which makes it far crazier not to.
What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether.
Jul 30 2014
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 30/07/2014 8:58 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.
+1 Most of the code should already be present in dmd, which makes it far crazier not to.
What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether.
If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. If we can do this, I think Walter might go along with it. Unless of course we could convince Microsoft to have a download just for the linker. We could download that in e.g. the installer. Would be better than a full install.
Jul 30 2014
next sibling parent reply "Joakim" <dlang joakim.airpost.net> writes:
On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole 
wrote:
 On 30/07/2014 8:58 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole 
 wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.
+1 Most of the code should already be present in dmd, which makes it far crazier not to.
What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether.
If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. If we can do this, I think Walter might go along with it. Unless of course we could convince Microsoft to have a download just for the linker. We could download that in e.g. the installer. Would be better than a full install.
I don't think dmd comes with a COFF64 linker now, users are just told to install Visual Studio or the Windows SDK for a linker. No reason you can't do the same with COFF32. Optlink can stick around with OMF for a couple releases. I suspect nobody would use it when given the choice of COFF32 support.
Jul 30 2014
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 30/07/2014 9:17 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole wrote:
 On 30/07/2014 8:58 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.
+1 Most of the code should already be present in dmd, which makes it far crazier not to.
What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether.
If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. If we can do this, I think Walter might go along with it. Unless of course we could convince Microsoft to have a download just for the linker. We could download that in e.g. the installer. Would be better than a full install.
I don't think dmd comes with a COFF64 linker now, users are just told to install Visual Studio or the Windows SDK for a linker. No reason you can't do the same with COFF32. Optlink can stick around with OMF for a couple releases. I suspect nobody would use it when given the choice of COFF32 support.
I'm in agreement with you about just getting people to install it. But the problem is, its not out of the box enough for Walters liking and I'm partially agreeing with him on this. At the end of the day, we don't really want to tell new people you have to also install x. Also not to forget that we could also remove OUR implib versions of Windows Dlls. That would solve a lot of issues for people I bet.
Jul 30 2014
prev sibling next sibling parent reply "w0rp" <devw0rp gmail.com> writes:
On Wednesday, 30 July 2014 at 09:17:05 UTC, Joakim wrote:
 On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole 
 wrote:
 On 30/07/2014 8:58 p.m., Joakim wrote:
 On Wednesday, 30 July 2014 at 08:12:17 UTC, Rikki Cattermole 
 wrote:
 On 30/07/2014 7:03 p.m., Kagamin wrote:
 Making dmd generate coff would make more sense.
+1 Most of the code should already be present in dmd, which makes it far crazier not to.
What makes it craziest is that there's a COFF32 branch lying around that nobody merges: http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com It would be a far better use of Jonathan's time to get COFF32 merged and obsolete Optlink altogether.
If we obsoleted the OMF format output we would need to have a free and distributed with PE-COFF linker. If we can do this, I think Walter might go along with it. Unless of course we could convince Microsoft to have a download just for the linker. We could download that in e.g. the installer. Would be better than a full install.
I don't think dmd comes with a COFF64 linker now, users are just told to install Visual Studio or the Windows SDK for a linker. No reason you can't do the same with COFF32. Optlink can stick around with OMF for a couple releases. I suspect nobody would use it when given the choice of COFF32 support.
I think it's important to ship with a linker without requiring any further installation. One of the things that helped me to learn D was being able to download DMD and run RDMD on Windows without installing anything else. It's not obvious to very new users who don't come from a Windows C++ background where you can get a gratis Microsoft compiler or how to configure it. If we were going to switch to COFF32, it would be a big bonus to ship a COFF32 linker in the installer.
Jul 30 2014
next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"w0rp"  wrote in message news:sinwmhzuvhmevqtunxms forum.dlang.org...

 I think it's important to ship with a linker without requiring any further 
 installation. One of the things that helped me to learn D was being able 
 to download DMD and run RDMD on Windows without installing anything else. 
 It's not obvious to very new users who don't come from a Windows C++ 
 background where you can get a gratis Microsoft compiler or how to 
 configure it. If we were going to switch to COFF32, it would be a big 
 bonus to ship a COFF32 linker in the installer.
FWIW this was the plan when I started ylink. I wanted to make a linker that could link omf and coff together, so I could tackle the object-format and runtime-format transitions independently. It does support mscoff32 hello world now, but not much more. Until it has debug info support it's not a viable replacement for optlink unfortunately.
Jul 30 2014
parent reply "w0rp" <devw0rp gmail.com> writes:
On Wednesday, 30 July 2014 at 13:03:30 UTC, Daniel Murphy wrote:
 "w0rp"  wrote in message 
 news:sinwmhzuvhmevqtunxms forum.dlang.org...

 I think it's important to ship with a linker without requiring 
 any further installation. One of the things that helped me to 
 learn D was being able to download DMD and run RDMD on Windows 
 without installing anything else. It's not obvious to very new 
 users who don't come from a Windows C++ background where you 
 can get a gratis Microsoft compiler or how to configure it. If 
 we were going to switch to COFF32, it would be a big bonus to 
 ship a COFF32 linker in the installer.
FWIW this was the plan when I started ylink. I wanted to make a linker that could link omf and coff together, so I could tackle the object-format and runtime-format transitions independently. It does support mscoff32 hello world now, but not much more. Until it has debug info support it's not a viable replacement for optlink unfortunately.
I didn't know about ylink. The prospect of having a free software linker for D on Win32 written in D does sound attractive. I assume it would be a lot of work to make it acceptable for usage.
Jul 30 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"w0rp"  wrote in message news:vnaffnibgvtmqeuhzczx forum.dlang.org...

 I didn't know about ylink. The prospect of having a free software linker 
 for D on Win32 written in D does sound attractive. I assume it would be a 
 lot of work to make it acceptable for usage.
I would estimate it's a smaller effort than DDMD. (win32 + omf + cv4) We really don't need to match the performance and memory usage of optlink. The best part is we can then potentially re-use parts of it inside dmd, currently dmd has its own lib generator for each format.
Jul 30 2014
parent reply "Jonathan Marler" <johnnymarler gmail.com> writes:
I like the discussion.  I do want to remind everyone that OPTLINK 
is very fast and switching to a different linker will likely 
result performance hit.  There are advantages to using COFF as it 
seems more compilers use that format and D would be more 
interoperable with other compilers and languages.  I think every 
peice of software has it's own goals which determine what 
tradeoffs to chose.  In my view, the linker should 1) always work 
and 2) be as fast as possible.  OPTLINK passes number 2 with 
flying colors but is does have bugs.  This is obviously due to 
the fact that it is written in assembly but keep in mind that the 
performance of the linker affects everyone who uses D.  Any 
performance hit on the linker will be seen in every single build 
of any D program. Adding 1 second to link time will add 1 second 
to build time since this process cannot be parallelized (as far 
as I know).

That being said, if it were up to me I wouldn't abandon OPTLINK 
so quickly.  I would spend time creating a test suite for 
OPTLINK, and try to comment the code a little better to encourage 
other developers to contribute.  Then I would consider adding 
support in DMD to generate COFF output files for those who want 
to link D object files with another linker. I wouldn't make this 
a huge priority though as one can use an OMF to COFF converter if 
they really needed it.
Jul 30 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 30 July 2014 at 19:50:35 UTC, Jonathan Marler wrote:
 I like the discussion.  I do want to remind everyone that 
 OPTLINK is very fast and switching to a different linker will 
 likely result performance hit.
Wouldn't it be easier to optimize a linker written in D than tinker with optlink?
Jul 30 2014
parent reply "Jonathan Marler" <johnnymarler gmail.com> writes:
On Thursday, 31 July 2014 at 06:46:19 UTC, Kagamin wrote:
 On Wednesday, 30 July 2014 at 19:50:35 UTC, Jonathan Marler 
 wrote:
 I like the discussion.  I do want to remind everyone that 
 OPTLINK is very fast and switching to a different linker will 
 likely result performance hit.
Wouldn't it be easier to optimize a linker written in D than tinker with optlink?
No matter how much you optimize your D code, you will only ever be able to use a subset of what assembly can do. That being said, writing perfect assembly code is impossible, so we rely on the compiler to take our higher level concepts and have it decide on the best way to optimize. The beauty of higher level languages is you can make high level changes resulting an a cascade of optimizations. But the downside is sometimes changes can have cascading adverse affects as well. One of the things I like about D is that it does a much better job of allowing the programmer to tell the compiler the "right" information it needs to know how to optimize. I would like to see a linker written in D and see how it compares to optlink. But I would reserve making a decision on moving to another linker until I got some solid performance data. Some performance data for a D linker compiled with all three D compilers as well.
Jul 31 2014
parent "Kagamin" <spam here.lot> writes:
On Thursday, 31 July 2014 at 07:07:27 UTC, Jonathan Marler wrote:
 No matter how much you optimize your D code, you will only ever 
 be able to use a subset of what assembly can do.
Such micro-optimizations give only marginal contribution to performance, which is mostly determined by choice of algorithms. The approach to write everything in assembly with manual optimizations is obsolete for many years, at least for desktop platforms. This is also one of the reasons, why optlink is translated to C.
 I would like to see a linker written in D and see how it 
 compares to optlink.  But I would reserve making a decision on 
 moving to another linker until I got some solid performance 
 data.  Some performance data for a D linker compiled with all 
 three D compilers as well.
I only wanted to say, no matter what performance data we get, the linker can be optimized by choosing faster algorithms without tinkering with assembler.
Jul 31 2014
prev sibling parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jonathan Marler"  wrote in message 
news:xrpdzjuaxtdhyfhpsgcu forum.dlang.org...

 I like the discussion.  I do want to remind everyone that OPTLINK is very 
 fast and switching to a different linker will likely result performance 
 hit.  There are advantages to using COFF as it seems more compilers use 
 that format and D would be more interoperable with other compilers and 
 languages.  I think every peice of software has it's own goals which 
 determine what tradeoffs to chose.  In my view, the linker should 1) 
 always work and 2) be as fast as possible.  OPTLINK passes number 2 with 
 flying colors but is does have bugs.  This is obviously due to the fact 
 that it is written in assembly but keep in mind that the performance of 
 the linker affects everyone who uses D.  Any performance hit on the linker 
 will be seen in every single build of any D program. Adding 1 second to 
 link time will add 1 second to build time since this process cannot be 
 parallelized (as far as I know).
Optlink simply doesn't do enough. It doesn't support modern debug info formats. It doesn't support mixing omf and coff object files (this is probably the most requested feature, because it would allow using the stock import libraries). Worst of all, optlink is hostile to change. win32 is dying, and optlink will certainly die with it. Having a more accessible linker will let us adapt much faster.
Jul 31 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-07-30 14:51, w0rp wrote:

 I think it's important to ship with a linker without requiring any
 further installation.
Just for the record. Currently, Windows in the only platform where this is true. -- /Jacob Carlborg
Aug 06 2014
prev sibling parent "Trass3r" <un known.com> writes:
 I don't think dmd comes with a COFF64 linker now, users are 
 just told to install Visual Studio or the Windows SDK for a 
 linker.  No reason you can't do the same with COFF32.  Optlink 
 can stick around with OMF for a couple releases.  I suspect 
 nobody would use it when given the choice of COFF32 support.
Anybody seriously programming on Windows uses VS anyway. But at least for 32 bit http://lld.llvm.org/windows_support.html could be packaged.
Jul 30 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
On Wednesday, 30 July 2014 at 09:06:11 UTC, Rikki Cattermole 
wrote:
 If we obsoleted the OMF format output we would need to have a 
 free and distributed with PE-COFF linker.
GNU binutils should do.
Jul 30 2014
prev sibling parent "Trass3r" <un known.com> writes:
 What makes it craziest is that there's a COFF32 branch lying 
 around that nobody merges:

 http://forum.dlang.org/thread/mailman.1560.1323886804.24802.digitalmars-d puremagic.com?page=9#post-llldfc:242q6p:241:40digitalmars.com
Same procedure as every year.
Jul 30 2014