www.digitalmars.com         C & C++   DMDScript  

D - Why is the linker necessary?

reply "Giancarlo Bellido" <vicentico1 hotmail.com> writes:
I don't know that much about compilers, and I've been wondering why is
the linker necessary. Is it really necessary to first compile the code into
an
.obj file and then link it with the .lib files and other .obj files in the
program? I believe that the source code of a program has the necessary
information to build an executable, except for DLLs and other Libraries but
that can be implemented like Borland did in Delphi.

what about compiled modules instead of .obj files, imported with the
"import" keyword.

library function calls can be implemented like this...
extern(Windows, "dllname.dll") name(pars);

it would make the compilation process extremely easy. you could just type:

dmc main.d (-flags...) to compile a program with multiple modules and
library calls instead of
dmc module1.d module2.d module3.d main.d library1.lib library2.lib
library3.lib .. etc.

these are just some ideas.
Aug 21 2003
next sibling parent "Sean L. Palmer" <palmer.sean verizon.net> writes:
Good idea.  Internally there will be a link step, otherwise separate
compilation and incremental compilation would never work.  But we shouldn't
have to deal with it.

Along with that comes some other stuff:  A module has to be a file, the file
has to be in the right place, someplace very predictable so it can find it.
Maybe modules could "incorporate" some other module, which would then extend
it into multiple files.  This is like public imports.

This could slow compilation down somewhat.  Obviously it would be more than
made up for by not having to parse text headers.  But in order to discover
that in order to build module A, it must first build module B, but by the
time it finds out it needs B, it has already started compiling A.  It would
not want to discard the useful work it has done on A, so it chains out and
compiles be and comes back and finishes compiling A.  But that means you
could run low on memory as the compiler crawls thru potentially hundreds of
files figuring out what to do.  Once it's finished it should definitely
write out dependency files so it knows whether anything needs recompiled
without having to do the whole thing again.

Anyway I would love to be spared the details of the linking.  ;)

Sean

"Giancarlo Bellido" <vicentico1 hotmail.com> wrote in message
news:bi41d2$26ip$1 digitaldaemon.com...

 I don't know that much about compilers, and I've been wondering why is
 the linker necessary. Is it really necessary to first compile the code


into

 an
 .obj file and then link it with the .lib files and other .obj files in the
 program? I believe that the source code of a program has the necessary
 information to build an executable, except for DLLs and other Libraries


but

 that can be implemented like Borland did in Delphi.

 what about compiled modules instead of .obj files, imported with the
 "import" keyword.

 library function calls can be implemented like this...
 extern(Windows, "dllname.dll") name(pars);

 it would make the compilation process extremely easy. you could just type:

 dmc main.d (-flags...) to compile a program with multiple modules and
 library calls instead of
 dmc module1.d module2.d module3.d main.d library1.lib library2.lib
 library3.lib .. etc.

 these are just some ideas.
Aug 22 2003
prev sibling next sibling parent reply Richard Krehbiel <rich kastle.com> writes:
Giancarlo Bellido wrote:

 I don't know that much about compilers, and I've been wondering why is
 the linker necessary. Is it really necessary to first compile the code into
 an
 .obj file and then link it with the .lib files and other .obj files in the
 program? I believe that the source code of a program has the necessary
 information to build an executable, except for DLLs and other Libraries but
 that can be implemented like Borland did in Delphi.
 
 what about compiled modules instead of .obj files, imported with the
 "import" keyword.
 
 library function calls can be implemented like this...
 extern(Windows, "dllname.dll") name(pars);
 
 it would make the compilation process extremely easy. you could just type:
 
 dmc main.d (-flags...) to compile a program with multiple modules and
 library calls instead of
 dmc module1.d module2.d module3.d main.d library1.lib library2.lib
 library3.lib .. etc.
 
 these are just some ideas.


Well, once upon a time, compiling was a very lengthy procedure.  It 
could take HOURS to compile your program AND the source for all the 
library functions it uses.  If the library functions were precompiled, 
and then (say) LINKED with your program, then youd only have to compile 
your source, which is much faster (only 30 minutes or so).

Of course, computers have gotten tens of thousands of times faster since 
then.  It might not be so impractical to build everything from source 
each time.

But there's still one concern, that the libraries you use might not be 
your own.  In this case, the owner of the library might not want you to 
be able to see the source, but they're happy to sell you a pre-compiled 
library.

Still... there's dynamic linking these days, too.  Maybe all the 
pre-compiled libraries would be .DLL (or .so) libraries, and everything 
else would need to be compiled from source.  Each and every time you 
change one line in one file of 70, all 70 get recompiled.  Or all 170. 
Or all 700.

Nah.  I think we should keep the linker, for a while anyway.
Aug 22 2003
parent "Samuel Barber" <opendtv yahoo.com> writes:
"Richard Krehbiel" <rich kastle.com> wrote in message
news:bi4u6j$gld$1 digitaldaemon.com...

 Well, once upon a time, compiling was a very lengthy procedure.  It
 could take HOURS to compile your program AND the source for all the
 library functions it uses.  If the library functions were precompiled,
 and then (say) LINKED with your program, then youd only have to compile
 your source, which is much faster (only 30 minutes or so).

 Of course, computers have gotten tens of thousands of times faster since
 then.  It might not be so impractical to build everything from source
 each time.

 But there's still one concern, that the libraries you use might not be
 your own.  In this case, the owner of the library might not want you to
 be able to see the source, but they're happy to sell you a pre-compiled
 library.


A more basic point is that object modules can be created by different
compilers/assemblers. It's no help to see the source if it isn't buildable
with your (single) tool.

Sam
Aug 25 2003
prev sibling next sibling parent Bill Cox <bill viasic.com> writes:
Giancarlo Bellido wrote:

 I don't know that much about compilers, and I've been wondering why is
 the linker necessary. Is it really necessary


...

I like to say (just because it's fun) that "The C linker is the Root of 
All Evil."  Being compatible with it leads to hacked C++ template 
support.  Most C++ programmers know about name mangling.  Optimization 
across files still generally sucks (there are some fairly new compilers 
that globally optimize).

The reason for the linker is mainly historical.  Back in high school, I 
had a job programming a PDP-11/45.  It had a whopping 128K of core 
memory (actual magnetic cores used for each bit).  We shared it between 
8 users at a time.  My max memory allocation was 16K bytes.

The compilers had to run off disk, in little pieces.  We had "4 pass 
compilers", which were slower than "3 pass compilers."  That meant that 
the source code had to be read 3 times instead of 4, which was the 
bottleneck in compilation speed.

After creating all the obect files, the linker ran.  That was always 
fun.  We had these washing machine sized hard disks, and the linker 
sensitized an oscillation mode.  Whenever we linked, one of these huge 
things would rock back and forth making a terrible niose, and then start 
walking around the room!

I guess I haven't been fond of linkers ever since.

Bill
Aug 22 2003
prev sibling next sibling parent "Mike Wynn" <mike.wynn l8night.co.uk> writes:
"Giancarlo Bellido" <vicentico1 hotmail.com> wrote in message
news:bi41d2$26ip$1 digitaldaemon.com...

 I don't know that much about compilers, and I've been wondering why is
 the linker necessary. Is it really necessary to first compile the code


into

 an
 .obj file and then link it with the .lib files and other .obj files in the
 program? I believe that the source code of a program has the necessary
 information to build an executable, except for DLLs and other Libraries


but

 that can be implemented like Borland did in Delphi.


if you have a big project incremental compilation saves a lot of time when
making changes
i.e. a make file that compiles d->obj then make libs then link the libs +
program

also there was a limit on command line length so you can't pass the whole
project as one command. (Some OS's limit command line length anyway)


 what about compiled modules instead of .obj files, imported with the
 "import" keyword.

 library function calls can be implemented like this...
 extern(Windows, "dllname.dll") name(pars);


I did write and post a templated lib that allow almost this behaviour.
(explicit loading rather than implicit though)


 it would make the compilation process extremely easy. you could just type:

 dmc main.d (-flags...) to compile a program with multiple modules and
 library calls instead of
 dmc module1.d module2.d module3.d main.d library1.lib library2.lib
 library3.lib .. etc.

 these are just some ideas.


I guess ppl are all thinking a long similar lines ....

I would also like to be able to do `dmd -build -Imylib -I/shared/otherlib -o
myapp.exe main.d` and have the d compiler find and compile all files into
myapp.exe

I believe that the D compiler should use obj files in the same way C
compiler use precompiled headers, so there should be a tempary dir (defaults
to c:\dmd\inter\<mangled-path> or similar) which contains the precompiled
Objs the D compiler just has to check that <path>file.d is older than
c:\dmd\inter\<mangled-path>file.obj so work out if it need to read in the d
file or the obj
there is an issue with templates the easiest solution is that the obj file
gets marked in someway to say its incomplete and its original source file
will be read in every time.

the required options are build me an exe, build a shared object (.so/.dll),
build me a lib or for backward compatibility `-c` => build me an object
file.

this removes the requirement to have make files that contain the dependancy
info, or tools that scan the D files looking for imports (he says wound up
again 'cos adding a new member function to a class requires all the objects
to be rebuild that use it or the prog crashes ... (obviously). and tracking
dependacies is a pain).

it would also seem in these days of IDE's that the D compiler should export
an interface to allow it to be intergrated into an IDE
again why write a tool to workout what the D compiler imports, when the D
compiler is doing all that work anyway.
dmd should just be a front end that loads dcompiler.dll (.so) and passes it
the start file etc.

the required interface (C or COM) should allow the following
send a file to D to build in to an exe, .dll/.so /lib or obj
the D compiler then calls back to request source or object (flags to say
either or source only)
and callbacks to store intermedite and final data.
I would invisage that searching and intermediate storage are all processed
in the frontend, so the D compiler does not have to bother with where things
came from just that they are the right thing.

this could be taken a step further (as D is intended to be useable as a
backend lang) to perform jit like (well just ahead of time) compilation. and
if an added callback (DGetProcAddress or sililar was added then a script
could be run and link to the exe that started the dcompiler up).

my vision ....
extern (C) bool DGetDCompiler( out IDCompiler dc ); // in dcompiler.dll
interface IDCompiler : IUnknown {
       enum OutputFormat { RAW, EXE, SHARED, LIB }
        bool beginCompilation( wchar[] filename, int tag, OutputFormat
format, IDCompilerHelper helper );
}
interface IDCompilerHelper : IUnknown {
        enum FileType { ANY = 0xF, SOURCE=1, OBJECT=2, LIB=3 }
        bool getFileItem( wchar[] name, int tag, FileType type, IDCompiler
from, out IDInputStream src );
        bool setFileItem( wchar[] name, int tag, FileType type, IDCompiler
from, IDOutputStream src );
}
// tag values >0 are helper side identifiers, values <0 are compiler
generated tags for items
// this allows you to send beginCompilation( "internal", 1, .... )
// then when item with tag 1 is requested you return generated D rather than
a file.

interface IDStream : IUnknown {
       void close();
        bit valid();
}
interface IDInputStream : IDStream {
       int read( byte[] buf, int max_len );
// returns the number of bytes read <0 == error
}
interface IDOutputStream : IDStream {
       int write( byte[] buf, int len );
  // returns the number of bytes written <0 == error can write less than len
}
Aug 22 2003
prev sibling parent "Carlos Santander B." <carlos8294 msn.com> writes:
Like Walter has said, he's trying to reuse the few resources he has at hand.
That's why he uses the linker.
However, I also think there should be a way for not having those really long
lines when linking. Maybe, in some way, the compiler could look for all the
needed .obj or .lib and pass directly to the linker. I mean, I have 2
versions of the same program: a console mode one and a graphics mode, but
both use the same core. Why should I be compiling each time that core? I
know I can pass just the .obj, but I'd rather just do
dmd -c core.d
dmd textmode.d
dmd guimode.d
And both compilations would automagically link with core.obj too. (I don't
use digc much because I don't know what the effect of that .res file is. I
only use it for compiling dig programs.)
And that idea of having some sort of a central repository could be really
useful too.

—————————————————————————
Carlos Santander


---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.512 / Virus Database: 309 - Release Date: 2003-08-19
Aug 22 2003