www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Where should D programs look for .so files on Linux (by default)?

reply "Chad Joan" <chadjoan gmail.com> writes:
--- The Problem ---

I was using Derelict to play around with SDL/OpenGL in D for a 
bit, and I was initially unable to do so because DerelictSDL2's 
search paths for .so files could not find libSDL2_ttf.so on my 
Linux distribution (Gentoo).

This seems quite solvable these days, so I entered an issue on 
DerelictSDL2's issue tracker.  As I typed away, it became 
apparent that this is probably something that the entire D 
community should care about: intelligent lookup of a system's .so 
files (or lack thereof) could have a profound impact on user 
experience for anyone on Linux using programs written in D.  Mike 
(aldacron) responded with a request to gather input from the D 
community at large, so here it is ;)

The original issue tracker is here:
https://github.com/DerelictOrg/DerelictSDL2/issues/28

--- One Solution ---

As an initial proposal, how about this:
1. Look in ./
2. Look in $LD_LIBRARY_PATH
3. Look in all directories listed in /etc/ld.so.conf
4. Look in a baked-in list of last-ditch fallback options. 
Although I'm sure better could be done, my current suggestion 
would look like this:
4a. ~/lib
4b. /usr/local/lib
4c. /usr/lib

This logic should probably end up in Phobos or somewhere suitably 
common, and revised as needed, so that all D programs can benefit 
from the community's research and experience on the subject.

As a data point: on my Gentoo box, the truly important step 
(usually) is to look in '/etc/ld.so.conf'.  My machine has a 
fairly comprehensive list in that file, and LD_LIBRARY_PATH 
doesn't have squat.

It may be desirable to first search folders or specific paths 
that are explicitly provided by the caller, and possibly skip the 
default search paths entirely if the caller insists.  I'm not 
sure how that would look, since the explicit path would probably 
be passed to a (non-Phobos) library-specific loading function 
(ex: DerelictSDL2.load(...)) before making its way to any 
community ordained logic.

In retrospect, I also have some reservations about looking in 
'./'.  I am not sure what the security implications are.  If 
there are no security concerns, then I, as both a developer and a 
user, would intuitively want D programs to look in ./ first 
(unless, perhaps, the program has an explicit search path 
defined, such as a packaged folder full of .so files, then 
/maybe/ those should go first).  As a developer, I might want to 
drop my program's .so/.dll files into the same directory as the 
executable, and I'd expect it to "just work". This is important 
whenever package managers can't be relied upon to deliver a 
correct version of the .so (ex: for obscure libraries), or can't 
be relied upon to release the program at all.  As a user, I 
sometimes want to replace .so/.dll files with different versions 
(ex: if the developer's version links to symbols or is compiled 
with features that don't exist on my system and are not strictly 
needed by the program).



Is this a reasonable approach, or is there a more appropriate way 
to handle this search?
Sep 16 2014
next sibling parent Jordi Sayol via Digitalmars-d <digitalmars-d puremagic.com> writes:
El 17/09/14 a les 07:52, Chad Joan via Digitalmars-d ha escrit:

 I was using Derelict to play around with SDL/OpenGL in D for a bit, and I was
initially unable to do so because DerelictSDL2's search paths for .so files
could not find libSDL2_ttf.so on my Linux distribution (Gentoo).


.so files are shared libraries on Linux. These libraries should be available on
specific directories listed in "/etc/ld.so.conf" and "/etc/ld.so.conf.d/*"
files. Place your shared libraries in some of these paths and then run
"ldconfig" command to create symlinks to the latest libraries version and to
update the "/etc/ld.so.cache" cache file. This is not specific for D but for
all Linux programs.

Regards,
-- 
Jordi Sayol
Sep 16 2014
prev sibling next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Wednesday, 17 September 2014 at 05:52:29 UTC, Chad Joan wrote:

 --- The Problem ---

 I was using Derelict to play around with SDL/OpenGL in D for a 
 bit, and I was initially unable to do so because DerelictSDL2's 
 search paths for .so files could not find libSDL2_ttf.so on my 
 Linux distribution (Gentoo).


If you're not using the slotted ebuild from:
https://bugs.gentoo.org/show_bug.cgi?id=481788
...you won't have that so file; only the version that links with 
SDL 1.2.  And if we look:

wyatt Yue ~ $ ls -F -w66 /usr/lib/libSDL* |grep ttf
/usr/lib/libSDL_ttf-2.0.so.0 
/usr/lib/libSDL_ttf-2.0.so.0.10.1*
/usr/lib/libSDL_ttf.so 

Sure enough!  I'd suggest pinging that bug because SDL2 isn't 
going away any time soon.


 intelligent lookup of a system's .so files (or lack thereof) 
 could have a profound impact on user experience for anyone on 
 Linux using programs written in D.  Mike (aldacron) responded 
 with a request to gather input from the D community at large, 
 so here it is ;)


While I certainly couldn't blame you for harshing on the dynamic 
linker most days, this (surprisingly) isn't one of them.  It's 
innocent.  Shocking, I know.


 As an initial proposal, how about this:
 1. Look in ./
 2. Look in $LD_LIBRARY_PATH
 3. Look in all directories listed in /etc/ld.so.conf
 4. Look in a baked-in list of last-ditch fallback options. 
 Although I'm sure better could be done, my current suggestion 
 would look like this:
 4a. ~/lib
 4b. /usr/local/lib
 4c. /usr/lib


As it happens, this is almost exactly what actually happens:
http://unix.stackexchange.com/a/22999


 This logic should probably end up in Phobos or somewhere 
 suitably common, and revised as needed, so that all D programs 
 can benefit from the community's research and experience on the 
 subject.


On Linux (and OSX?), at least, E_NOTOURPROBLEM.  I possibly 
recall there was some amount of hilarity related to this on 
Windows, but that's an area where I Just Don't Care™.

-Wyatt
Sep 17 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
I presume you mean dynamically loaded .so libraries and not just 
dynamically linked? Because latter use the same library path 
resolution rules as in C world (ld takes care of that)

As for dynamically loaded ones - it is somewhat tough question. 
Loading non-local plugin .so libraries is quite unusual, Derelict 
is the only software I know that does it. I am not sure it is 
even a good idea in general.

Are there any examples how this is solved in other languages / 
platforms?
Sep 17 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Sep 17, 2014 at 04:43:04PM +0000, Dicebot via Digitalmars-d wrote:

 I presume you mean dynamically loaded .so libraries and not just
 dynamically linked? Because latter use the same library path
 resolution rules as in C world (ld takes care of that)
 
 As for dynamically loaded ones - it is somewhat tough question.
 Loading non-local plugin .so libraries is quite unusual, Derelict is
 the only software I know that does it. I am not sure it is even a good
 idea in general.
 
 Are there any examples how this is solved in other languages /
 platforms?


In C / Posix, you can load any library dynamically using dlopen(),
dlsym(), et al, manually "link" your program by extracting function
pointers and data pointers. Some applications use this to implement a
plugin mechanism -- some Linux media players do this, I believe, and
browsers probably also use the same mechanism for browser plugins.

In any case, dlopen() uses the OS's dynamic linker to find libraries, so
it's really a matter of setting up your OS environment to be able to
find the libraries. It's not really within the scope of the D runtime to
attempt to configure this. Furthermore, some application plugin
frameworks may do things like fiddle with LD_LIBRARY_PATH to get it to
look for libraries where the dynamic linker wouldn't usually look; in
that case, you're really at the mercy of the plugin framework author(s)
as to how to configure such lookups. Again, it's not really within the D
runtime's mandate to control such things.

As for how it works on Windows, I have no idea at all. It's probably
completely different from Posix, which is more reason to leave it up to
plugin framework implementors to implement, rather than hard-coding an
incomplete / inconsistent implementation in druntime.


T

-- 
One disk to rule them all, One disk to find them. One disk to bring them all
and in the darkness grind them. In the Land of Redmond where the shadows lie.
-- The Silicon Valley Tarot
Sep 17 2014
parent "Cliff" <cliff.s.hudson gmail.com> writes:
On Wednesday, 17 September 2014 at 17:13:07 UTC, H. S. Teoh via
Digitalmars-d wrote:


 As for how it works on Windows, I have no idea at all. It's 
 probably
 completely different from Posix, which is more reason to leave 
 it up to
 plugin framework implementors to implement, rather than 
 hard-coding an
 incomplete / inconsistent implementation in druntime.


 T


FYI: Windows rules are here:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586(v=vs.85).aspx
Sep 17 2014