www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Problem parsing IPv4/IPv6 addresses with std.socket.parseAddress

reply Dsciple <a.bianciotto gmail.com> writes:
Hi there!

I wrote a small utility library to read configuration parameters 
from both command-line arguments (using std.getopt) and SDLang 
files (using sdlang-d package).
The main library defines a struct ConfigParams whose fields are 
themselves structs defined in sub-libraries (set as 
dependencies), each responsible for reading one kind of parameter 
(BindAddress, BindPort, etc) from both command-line and SDL 
files. Each sub-library has its own unit tests which run 
successfully in isolation.
Then, when putting everything together in the main struct 
CofigParams I get the compile-time error:

/usr/include/dlang/dmd/std/socket.d(1195,9): Error: static 
variable getaddrinfoPointer cannot be read at compile time
called from here: parseAddress(addressString, null)

while using std.socket.parseAddress to validate IPv4/IPv6 bind 
addresses.
I'm using dmd v2.071.2 and dub v1.0.0.
The relevant piece of code where this happens is the following:

struct BindAddress {

   import std.socket: parseAddress, SocketException;

   // Private members
   private string addressString = "0.0.0.0";

   // Construct from address string
   this(string addressString) {
     try {
       auto address = parseAddress(addressString);
     } catch(SocketException ex) {
       throw new BindAddressException("Invalid bind address " ~ 
addressString);
     }
     this.addressString = addressString;
   }

}

As said, this works fine when tested in isolation, and the 
compiler only complains when using BindAddress as a member of 
ConfigParams.
Any idea what the problem may be?
Or is there maybe a ready to use, high-level library for parsing 
parameters from command-line arguments and config files of some 
kind?
Sep 27 2016
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Tuesday, 27 September 2016 at 09:04:53 UTC, Dsciple wrote:

 As said, this works fine when tested in isolation, and the 
 compiler only complains when using BindAddress as a member of 
 ConfigParams.
 Any idea what the problem may be?
 Or is there maybe a ready to use, high-level library for 
 parsing parameters from command-line arguments and config files 
 of some kind?


I assume your ConfigParams variable is global or static? Can you 
show how you initialize it, and how it's declared?

You're probably using it in a way that requires it to be 
evaluated at compile time. That's the case for initializers of 
global/static variables, as well as default values of struct 
members.
Sep 27 2016
parent reply Dsciple <a.bianciotto gmail.com> writes:
On Tuesday, 27 September 2016 at 14:02:25 UTC, Marc Schütz wrote:

 On Tuesday, 27 September 2016 at 09:04:53 UTC, Dsciple wrote:

 As said, this works fine when tested in isolation, and the 
 compiler only complains when using BindAddress as a member of 
 ConfigParams.
 Any idea what the problem may be?
 Or is there maybe a ready to use, high-level library for 
 parsing parameters from command-line arguments and config 
 files of some kind?


 I assume your ConfigParams variable is global or static? Can 
 you show how you initialize it, and how it's declared?

 You're probably using it in a way that requires it to be 
 evaluated at compile time. That's the case for initializers of 
 global/static variables, as well as default values of struct 
 members.


Yes I think so.
I use static default values for all members of ConfigParams and I 
instantiate ConfigParams in my unit tests, so I assume that the 
variable would be global there.
The code looks like:

unittest {

   string[] args = [
     "binaryFileName",
     "--bindAddresses=0.1.2.3;4.5.6.7",
     "--bindHTTPPort=80",
     "--bindHTTPSPort=443",
     "--configFile=testfiles/test.conf.sdl",
     "--verbosityLevel=detailed",
   ];

   ConfigParams configParams;  // default values for parameters
   configParams.readFromAll(args); // values read from 
command-line arguents

   // assertion checks here
}

What do you suggest? Should I move all default initializations to 
a constructor?
Thank you for your response.
Sep 27 2016
parent reply Dsciple <a.bianciotto gmail.com> writes:
On Tuesday, 27 September 2016 at 14:39:10 UTC, Dsciple wrote:

 On Tuesday, 27 September 2016 at 14:02:25 UTC, Marc Schütz 
 wrote:

 On Tuesday, 27 September 2016 at 09:04:53 UTC, Dsciple wrote:

 As said, this works fine when tested in isolation, and the 
 compiler only complains when using BindAddress as a member of 
 ConfigParams.
 Any idea what the problem may be?
 Or is there maybe a ready to use, high-level library for 
 parsing parameters from command-line arguments and config 
 files of some kind?


 I assume your ConfigParams variable is global or static? Can 
 you show how you initialize it, and how it's declared?

 You're probably using it in a way that requires it to be 
 evaluated at compile time. That's the case for initializers of 
 global/static variables, as well as default values of struct 
 members.


 Yes I think so.
 I use static default values for all members of ConfigParams and 
 I instantiate ConfigParams in my unit tests, so I assume that 
 the variable would be global there.
 The code looks like:

 unittest {

   string[] args = [
     "binaryFileName",
     "--bindAddresses=0.1.2.3;4.5.6.7",
     "--bindHTTPPort=80",
     "--bindHTTPSPort=443",
     "--configFile=testfiles/test.conf.sdl",
     "--verbosityLevel=detailed",
   ];

   ConfigParams configParams;  // default values for parameters
   configParams.readFromAll(args); // values read from 
 command-line arguents

   // assertion checks here
 }

 What do you suggest? Should I move all default initializations 
 to a constructor?
 Thank you for your response.


I forgot to show declaration. Here it is:

struct ConfigParams {

   import ConfigParamsCLAMixin: ConfigParamsCLA;
   import ConfigParamsSDLMixin: ConfigParamsSDL;
   import ConfigParamsAllMixin: ConfigParamsAll;
   import BindAddress: BindAddress, BindAddressException;
   import BindAddresses: BindAddresses, BindAddressesException;
   import BindPort: BindPort, BindPortException;
   import PosixPath: PosixPath, PosixPathException;
   import VerbosityLevel: VerbosityLevel, VerbosityLevelException;

   // Define configuration parameters' static default fields
   static immutable BindAddresses defaultBindAddresses = 
BindAddresses([
     BindAddress("192.168.2.10")
   ]);
   static immutable BindPort defaultBindHTTPPort = BindPort(8080);
   static immutable BindPort defaultBindHTTPSPort = BindPort(4430);
   static immutable PosixPath defaultConfigFile =
     PosixPath("/etc/ras/ras.conf.sdl");
   static immutable VerbosityLevel defaultVerbosityLevel =
     VerbosityLevel("quiet");

   // Define configuration parameters fields with default values
   BindAddresses bindAddresses = defaultBindAddresses;
   BindPort bindHTTPPort = defaultBindHTTPPort;
   BindPort bindHTTPSPort = defaultBindHTTPSPort;
   PosixPath configFile = defaultConfigFile;
   VerbosityLevel verbosityLevel = defaultVerbosityLevel;

}

I don't understand why the unit tests of all individual members 
of ConfigParams (BindAddresses, BindAddress and so on) would work 
in isolation using the same kind of initialization, whereas some 
of them would fail as members of ConfigParams.
I suspect this may also be a problem with settings in dub.json 
for ConfigParams:

{
   ...
   "sourcePaths": [
     "sources"
   ],
   "importPaths": [
     "sources"
   ],
   "configurations": [
     {
       "name": "library",
       "targetName": "ConfigParams",
       "targetType": "library"
     }
   ],
   "dependencies": {
     "bind-address": {
       "version": "*",
       "path": "./libraries/BindAddress"
     },
     "bind-addresses": {
       "version": "*",
       "path": "./libraries/BindAddresses"
     },
     "bind-port": {
       "version": "*",
       "path": "./libraries/BindPort"
     },
     "posix-name": {
       "version": "*",
       "path": "./libraries/PosixName"
     },
     "posix-path": {
       "version": "*",
       "path": "./libraries/PosixPath"
     },
     "verbosity-level": {
       "version": "*",
       "path": "./libraries/VerbosityLevel"
     }
   }

if I include bind-address as a dependency (as above), dub refuses 
to run with:

Sub package bind-address: doesn't exist.

although the package is there and used by bind-addresses and 
confi params too...
Sep 27 2016
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Tuesday, 27 September 2016 at 14:57:26 UTC, Dsciple wrote:

 struct ConfigParams {
   // ...
   // Define configuration parameters' static default fields
   static immutable BindAddresses defaultBindAddresses = 
 BindAddresses([
     BindAddress("192.168.2.10")
   ]);
   // ...
 }


Yepp, that's definitely the culprit.


 I don't understand why the unit tests of all individual members 
 of ConfigParams (BindAddresses, BindAddress and so on) would 
 work in isolation using the same kind of initialization, 
 whereas some of them would fail as members of ConfigParams.


Because in your unit tests, your BindAddress variables are 
probably normal, local variables. These are initial at runtime as 
you would expect. Only in places where a static initialization is 
required, the compiler will evaluate the initializer at compile 
time. Examples:

     // global, but no initializer => ok
     BindAddress global1;
     // global with initializer => needs to be evaluated at 
compile time
     BindAddress global2 = BindAddress("127.0.0.1");
     struct Config {
         // default value for members => evaluate at compile time
         BindAddress bindAddress = BindAddress("0.0.0.0");
     }

     void main() {
         // assignment to globals at runtime works (of course)
         global1 = BindAddress("192.168.1.2");
         // local with initializer => works at runtime
         BindAddress local1 = BindAddress("10.0.0.1");
         // static => same as global, evaluated at compile time
         static BindAddress static1 = BindAddress("127.0.0.1");
     }

You could solve the problem as you suggested by moving the 
initializations into a constructor, but if your default values 
are only ever IPs (i.e., no hostname resolution necessary), you 
could also add an additional function that parse them without 
calling into the C library, e.g.

     struct BindAddress {
         // ...
         static BindAddress fromIP(string addr) {
             // ...
         }
     }

     struct Config {
         // ...
         BindAddress bindAddress = BindAddress.fromIP("127.0.0.1");
     }

You can try whether InternetAddress.parse() [1] works at compile 
time, otherwise you would have to implement it by hand.

[1] 
https://dlang.org/phobos/std_socket.html#.InternetAddress.parse
Sep 28 2016
parent reply Dsciple <a.bianciotto gmail.com> writes:
On Wednesday, 28 September 2016 at 09:56:06 UTC, Marc Schütz 
wrote:

 You could solve the problem as you suggested by moving the 
 initializations into a constructor, but if your default values 
 are only ever IPs (i.e., no hostname resolution necessary), you 
 could also add an additional function that parse them without 
 calling into the C library, e.g.

     struct BindAddress {
         // ...
         static BindAddress fromIP(string addr) {
             // ...
         }
     }

     struct Config {
         // ...
         BindAddress bindAddress = 
 BindAddress.fromIP("127.0.0.1");
     }

 You can try whether InternetAddress.parse() [1] works at 
 compile time, otherwise you would have to implement it by hand.

 [1] 
 https://dlang.org/phobos/std_socket.html#.InternetAddress.parse


Indeed, getting rid of all static initializations in ConfigParams 
did finally work!
Now I'm left with parameters' definitions without default values, 
and an ugly constructor taking a long list of initialization 
values as arguments and storing them in their respective private 
members :/ .

struct ConfigParams {
   BindAddresses bindAddresses;
   BindPort bindHTTPPort;
   BindPort bindHTTPSPort;
   PosixPath configFile;
   VerbosityLevel verbosityLevel;

   this(BindAddresses bindAddresses,
       BindPort bindHTTPPort = BindPort(8080),
       BindPort bindHTTPSPort = BindPort(4430),
       PosixPath configFile = PosixPath("/etc/ras/ras.conf.sdl"),
       VerbosityLevel verbosityLevel = VerbosityLevel("quiet")) {
     this.bindAddresses = bindAddresses;
     this.bindHTTPPort = bindHTTPPort;
     this.bindHTTPSPort = bindHTTPSPort;
     this.configFile = configFile;
     this.verbosityLevel = verbosityLevel;
   }
}

Luckily I can set default values in constructor's signature 
(except for the very first argument, otherwise I get a 
deprecation warning for defaulting all arguments which does not 
make sense), so I save some typing when calling it.
Any idea about how to improve this?

In the end everything boils down to not being able to use static 
initializations with std.socket.parseAddress and friends. I don't 
understand what prevents such function (in turn calling some 
OS-level C function) from doing its job at compile time too. 
Guess it's a very challanging thing to do at compile-time, or 
should I open an issue with the std.socket lib developers?

By the way, out of despair, I also tried to validate addresses 
myself using regular expressions but:
1) parsing generic IPv6 addresses is not trivial at all!
2) even skipping IPv6, I get similar compiler errors complaining 
again about malloc not being able to compute some value at 
compile time, so std.regex also conflicts with static 
initializations somehow..

All in all, I can live with the current solution and move on.
Thank you a lot for your support Marc!
Sep 28 2016
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 28 September 2016 at 15:34:56 UTC, Dsciple wrote:

 I don't understand what prevents such function (in turn calling 
 some OS-level C function) from doing its job at compile time 
 too. Guess it's a very challanging thing to do at compile-time, 
 or should I open an issue with the std.socket lib developers?


std.socket simply passes the request on to the operating system. 
Enabling parsing IP addresses at compile time would necessitate 
duplicating the logic already present in the operating system 
libraries, which would not be future-proof at best, and buggy at 
worst.

It's not possible to implement by invoking C functions like at 
runtime, since that is forbidden during compile-time for a number 
of reasons, mainly safety.
Sep 28 2016
parent Dsciple <a.bianciotto gmail.com> writes:
On Wednesday, 28 September 2016 at 16:49:54 UTC, Vladimir 
Panteleev wrote:

 On Wednesday, 28 September 2016 at 15:34:56 UTC, Dsciple wrote:

 I don't understand what prevents such function (in turn 
 calling some OS-level C function) from doing its job at 
 compile time too. Guess it's a very challanging thing to do at 
 compile-time, or should I open an issue with the std.socket 
 lib developers?


 std.socket simply passes the request on to the operating 
 system. Enabling parsing IP addresses at compile time would 
 necessitate duplicating the logic already present in the 
 operating system libraries, which would not be future-proof at 
 best, and buggy at worst.

 It's not possible to implement by invoking C functions like at 
 runtime, since that is forbidden during compile-time for a 
 number of reasons, mainly safety.


I suspected there must have been a very good reason for this not 
being possible at compile time and this definitely looks like 
one: thanks Vladimir for tracking this discussion and clarifying 
it to me!
So I guess this answers all my questions and. as a beginner, I 
can also say I learned something in the process: thank you again!
Sep 28 2016