www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - version() abuse! Note of library writers.

reply Travis Boucher <boucher.travis gmail.com> writes:
The use of version(...) in D has the potential for some very elegant 
portable code design.  However, from most of the libraries I have seen, 
it is abused and misused turning it into a portability nightmare.

http://dsource.org/projects/dmd/browser/trunk/src/mars.c#L313 defines 
the following versions: Windows, Posix, linux, OSX, darwin, FreeBSD and 
Solaris.

http://dgcc.svn.sourceforge.net/viewvc/dgcc/trunk/d/target-ver
syms.sh?view=markup 
defines aix, unix, cygwin, darwin, freebsd, Win32, skyos, solaris, 
freebsd (and others).

The problem I run into is the assumption that linux == unix/posix.  This 
assumption is not correct.

The use of version(linux) should be limited to code that:

1. Declares externals from sys/*.h
2. Accesses /proc or /sys (or other Linux specific pseudo filesystems)
3. Things that interface directly with the dynamic linker (eg. linking 
against libdl)
4. Other things that are linux specific....

Anything that accesses standard libc functions, standard unix semantics 
(eg. signals, shm, etc) should use version(Posix) or version(unix).

Build systems and scripts that are designed to run on unix machines 
should not assume the locations of libraries and binaries, and refer to 
posix standards for their locations.  For example, bash in /bin/bash or 
the assumption of the existence of bash at all.  If you need a shell 
script, try writing it with plain bourne syntax without all of the bash 
extensions to the shell, and use /bin/sh.  Also avoid using the GNU 
extensions to standard tools (sed and awk for example).  If you really 
want to do something fancy, do it in D and use the appropriate {g,l}dmd 
-run command.

A few things to keep in mind about linux systems vs. pretty much all 
other unix systems:

Linux is a kernel.  The userspace stuff you use (your shell, your libc, 
etc) is (typically) GNU, and not Linux.  On other unix systems, the 
kernel is tightly linked to the libc and other standard libraries, and 
the other base applications (shell, login, sed, awk, etc) are often part 
of the base system and not GNU (this is not always true, as some systems 
use GNU tools as part of the base system as well).

If you are writing a wrapper around something, and it is a library in 
Linux, it most likely is also a library in other unix machines.  This 
includes opengl, image libraries, X11 libraries, sound & media 
libraries, etc.  If you require an external library, please state as 
much in the documentation and don't hide it in a version(linux) 
statement because you just abused the reason for version(...) to exist 
in the first place.

Other tips:
  - Don't use the /proc filesystem unless you really must.  If you do, 
abstract the interface and implement a Linux specific interface.  This 
will ease porting (I'd be happy to come in and do FreeBSD stuff where 
possible).
  - If you are unsure, check http://www.freebsd.org/cgi/man.cgi This 
interface can look up the manual pages for multiple OSes, and the 
FreeBSD versions of the manuals are very consistent.  Some of the other 
ones will also give hints on bugs and subtle differences for different 
implementations.
  - If you want to make something work under FreeBSD or other OSes, post 
on the NG (or for FreeBSD, bug me directly).
  - Linux programs need to be linked against libdl for access to 
dlopen() and family, most unix OSes have access to dlopen() from libc (I 
think this is partially due to the tight libc/kernel coupling in most 
unix OSes).
  - Darwin/OSX & FreeBSD all share alot of similar kernel interfaces, 
specifically most of the process model, network stack and virtual 
filesystem layers.  OSX even has kqueue! (although slightly different 
then FreeBSD).
  - FreeBSD & Solaris share some common ancestry.  Although the 
similarities are not very important to application developers, internal 
kernel interfaces and designs are similar.
  - The GNU tools typically conform to standards, but add extra 
extensions everywhere.  If you write your scripts for a standard bourne 
shell rather then bash, bash will still be able to run it most of the 
time.  Personally the GNU extensions to everything feels like a move 
Microsoft would do, just breaking enough compatibility to create vendor 
lock in.

Thanks,
Travis
Nov 17 2009
next sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
Travis Boucher wrote:

...

May I suggest to put these notes in Wiki4D so they don't get lost in the 
flood of postings?
Nov 18 2009
prev sibling next sibling parent reply =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Travis Boucher wrote:

 The use of version(...) in D has the potential for some very elegant 
 portable code design.  However, from most of the libraries I have seen, 
 it is abused and misused turning it into a portability nightmare.


It has done this for years, so it's already turned that way.
Usually it's "version(Win32) /*Windows*/; else /*linux*/;"...


 Anything that accesses standard libc functions, standard unix semantics 
 (eg. signals, shm, etc) should use version(Posix) or version(unix).


Nice rant, but it's "version(Unix)" in GCC and we're probably
stuck with the horrible version(linux) and version(OSX) forever.


 Build systems and scripts that are designed to run on unix machines 
 should not assume the locations of libraries and binaries, and refer to 
 posix standards for their locations.  For example, bash in /bin/bash or 
 the assumption of the existence of bash at all.  If you need a shell 
 script, try writing it with plain bourne syntax without all of the bash 
 extensions to the shell, and use /bin/sh.  Also avoid using the GNU 
 extensions to standard tools (sed and awk for example).  If you really 
 want to do something fancy, do it in D and use the appropriate {g,l}dmd 
 -run command.


I rewrote my shell scripts in C++ for wxD, to work on Windows.
Tried to use D (mostly for DSSS), but it wasn't working right.


 A few things to keep in mind about linux systems vs. pretty much all
 other unix systems:


Nice list, you should put it on a web page somewhere (Wiki4D ?)
Usually one also ends up using runtime checks or even autoconf.

--anders


PS. Some people even think that /usr/bin/python exists. :-)
     Guess they were confusing it with standard /usr/bin/perl
Nov 18 2009
next sibling parent Travis Boucher <boucher.travis gmail.com> writes:
Anders F Björklund wrote:

 Travis Boucher wrote:

 The use of version(...) in D has the potential for some very elegant 
 portable code design.  However, from most of the libraries I have 
 seen, it is abused and misused turning it into a portability nightmare.


 
 It has done this for years, so it's already turned that way.
 Usually it's "version(Win32) /*Windows*/; else /*linux*/;"...


I'm fairly new to D, and one thing I really love about it is the removal 
of the preprocessor in favor of specific conditional compilation 
(version, debug, unittest, static if, CTFE, etc).  Nothing was worse 
then trying to decode a massive #ifdef tree supporting different 
features from different OSes.

I don't expect things to change right now, but I think that there should 
be some standard version() statements that are not only implementation 
defined.  I'd also like people to start thinking about the OS 
hierarchies with version statements.

Windows
   Win32
   Win64
   WinCE  (as an example...)
Posix (or Unix, I don't care which one)
   BSD
     FreeBSD
     OpenBSD
     NetBSD
     Darwin
   Linux
   Solaris

The problem with "version(Win32) /*Windows*/; else /*linux*/;" is fairly 
subtle, but I have run into it alot with bindings to C libraries that 
use the dlopen() family and try to link against libdl.


 Anything that accesses standard libc functions, standard unix 
 semantics (eg. signals, shm, etc) should use version(Posix) or 
 version(unix).


 
 Nice rant, but it's "version(Unix)" in GCC and we're probably
 stuck with the horrible version(linux) and version(OSX) forever.


On my install (FreeBSD) version(Unix) and version(Posix) are both defined.


 Build systems and scripts that are designed to run on unix machines 
 should not assume the locations of libraries and binaries, and refer 
 to posix standards for their locations.  For example, bash in 
 /bin/bash or the assumption of the existence of bash at all.  If you 
 need a shell script, try writing it with plain bourne syntax without 
 all of the bash extensions to the shell, and use /bin/sh.  Also avoid 
 using the GNU extensions to standard tools (sed and awk for example).  
 If you really want to do something fancy, do it in D and use the 
 appropriate {g,l}dmd -run command.


 
 I rewrote my shell scripts in C++ for wxD, to work on Windows.
 Tried to use D (mostly for DSSS), but it wasn't working right.


Yeah, I can understand in some cases using D itself could be a major 
bootstrapping hassle.  This issue isn't D specific, and exists in alot 
of packages.  I've even gotten to the point to expect most third party 
packages won't work with FreeBSD's make, and always make sure GNU make 
is available.


 A few things to keep in mind about linux systems vs. pretty much all
 other unix systems:


 
 Nice list, you should put it on a web page somewhere (Wiki4D ?)
 Usually one also ends up using runtime checks or even autoconf.


I haven't registered in Wiki4D yet, I might soon once I take the time to 
clean up this ranty post into something a little more useful.


 
 PS. Some people even think that /usr/bin/python exists. :-)
     Guess they were confusing it with standard /usr/bin/perl


I won't even go into my feelings about python.  Sadly perl is slowly 
becoming more extinct.  It would be nice for people to remember that 
perl started as a replacement for sed & awk, and still works well for 
that purpose.  At least people don't assume ruby exists.

The bad thing is when a build system breaks because of something 
non-critical failing.  A good example of this is the gtkd demoselect.sh 
script.  It use to assume /bin/bash, which would trigger a full build 
failure.  Since it was changed to /bin/sh, it doesn't work correctly on 
FreeBSD (due to I think some GNU extensions used in sed), but it doesn't 
cause a build failure.  It just means the default demos are built.
Nov 17 2009
prev sibling parent reply grauzone <none example.net> writes:
Anders F Björklund wrote:

 PS. Some people even think that /usr/bin/python exists. :-)
     Guess they were confusing it with standard /usr/bin/perl


What else should you use for Python scripts?
Nov 18 2009
parent =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
grauzone wrote:

 PS. Some people even think that /usr/bin/python exists. :-)
     Guess they were confusing it with standard /usr/bin/perl


 
 What else should you use for Python scripts?



be installed in /usr/local/bin/python or something.

--anders
Nov 18 2009
prev sibling next sibling parent reply Travis Boucher <boucher.travis gmail.com> writes:
Another note, something I see in tango and I don't know why I didn't 
think about it before.

If you want to require bash, use:



instead of
Nov 17 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Travis Boucher wrote:

 Another note, something I see in tango and I don't know why I didn't 
 think about it before.
 
 If you want to require bash, use:
 

 
 instead of
 




Sadly, env may be in /bin/ or /usr/bin/.

Andrei
Nov 18 2009
prev sibling next sibling parent "Nick Sabalausky" <a a.a> writes:
"Travis Boucher" <boucher.travis gmail.com> wrote in message 
news:hdvoke$1qs3$1 digitalmars.com...

 [posix/unix/linux/bsd version info and tips]


As a mainly-windows person who just knows enough *nix to get by and to 
maintain ports of their own software, this is great stuff to know! Some of 
it I was already aware of, but there was a lot I didn't. So much of that is 
the kind of information that can be very difficult for a *nix non-expert (to 
say nothing of novices) to find, or even think to look for. Consider your 
post much appreciated :)

Looking forward to a wiki4d version. And I'll be making sure to check 
through my d tools/libs for any such misuses. Next time I do linux builds of 
my Goldie and SemiTwistDTools projects (I normally do most of my dev on Win, 
so the linux bnaries get old, and I may occasionally break linux without 
knowing), I just may take you up on that offer to do a FreeBSD check on it.
Nov 18 2009
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Travis Boucher Wrote:


 
 The problem I run into is the assumption that linux == unix/posix.  This 
 assumption is not correct.
 
 The use of version(linux) should be limited to code that:
 
 1. Declares externals from sys/*.h
 2. Accesses /proc or /sys (or other Linux specific pseudo filesystems)
 3. Things that interface directly with the dynamic linker (eg. linking 
 against libdl)
 4. Other things that are linux specific....
 
 Anything that accesses standard libc functions, standard unix semantics 
 (eg. signals, shm, etc) should use version(Posix) or version(unix).


It may help to think about weird configurations like targeting Cygwin on
Windows (where Windows, Win32, and Posix may theoretically be defined) or WINE
on Linux (where linux and Win32 may be defined).  I haven't really considered
the latter situation, but the former is handled correctly in D2.  As you've
said, the best idea is to version for the API you're targeting rather than the
OS you're on.  For the most part, a version also corresponds to a package in
core.sys in D2.  core.sys.posix contains pretty much everything *nix folks are
used to, with core.sys.osx and core.sys.linux containing kernel APIs, etc.
Nov 18 2009
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Sean Kelly Wrote:


 or WINE on Linux (where linux and Win32 may be defined)


Oops, add Posix to that list.
Nov 18 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Sean Kelly wrote:

 Sean Kelly Wrote:
 

 or WINE on Linux (where linux and Win32 may be defined)


 
 Oops, add Posix to that list.


FWIW, Wine is one of the platforms used for testing Phobos.

Andrei
Nov 18 2009
prev sibling parent reply =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Sean Kelly wrote:

 It may help to think about weird configurations like targeting Cygwin
 on Windows (where Windows, Win32, and Posix may theoretically be
 defined) or WINE on Linux (where linux and Win32 may be defined).


Something is broken if "Windows" is declared in Cygwin,
or if "linux" is declared when running under Wine...

--anders
Nov 18 2009
parent reply Sean Kelly <sean invisibleduck.org> writes:
Anders F Björklund Wrote:


 Sean Kelly wrote:

 It may help to think about weird configurations like targeting Cygwin
 on Windows (where Windows, Win32, and Posix may theoretically be
 defined) or WINE on Linux (where linux and Win32 may be defined).


 
 Something is broken if "Windows" is declared in Cygwin,
 or if "linux" is declared when running under Wine...


Why?  That's the OS the app is being compiled on/for.  Let's consider another
example.  Windows Services for Unix Applications (I think that's what it's
called now) is a POSIX subsystem built into Vista and Windows 7.  There, both
Windows and Posix would definitely be defined, and I *think* Win32 would be
defined as well.  I suppose one could argue that the OS version should be
Interix instead of Windows, but it amounts to the same thing.
Nov 18 2009
parent =?windows-1252?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Sean Kelly wrote:


 It may help to think about weird configurations like targeting Cygwin
 on Windows (where Windows, Win32, and Posix may theoretically be
 defined) or WINE on Linux (where linux and Win32 may be defined).


 Something is broken if "Windows" is declared in Cygwin,
 or if "linux" is declared when running under Wine...


 
 Why?  That's the OS the app is being compiled on/for.  Let's consider
 another example.  Windows Services for Unix Applications (I think
 that's what it's called now) is a POSIX subsystem built into Vista
 and Windows 7.  There, both Windows and Posix would definitely be
 defined, and I *think* Win32 would be defined as well.  I suppose one
 could argue that the OS version should be Interix instead of Windows,
 but it amounts to the same thing.


I guess it'll get "fixed" when it moves to API over OS...

Just that I had been surprised if D1 was changed like that.

--anders
Nov 18 2009