www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - getOpt with shared

reply Danny Arends <Danny.Arends gmail.com> writes:
Hey all,

I have been working on creating a multi-threaded application, so 
I have a shared configuration object which hold several command 
line parameters (which I fill using getopt).

The problem is that I get deprecation warnings when trying to set 
numerical values:

/usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation: 
read-modify-write operations are not allowed for shared 
variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.

using getopt with a shared object and boolean values seems 
completely broken:

/usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation 
not allowed on bool *receiver += 1
/usr/include/dmd/phobos/std/getopt.d(751,46): Error: template 
instance `std.getopt.handleOption!(shared(bool)*)` error 
instantiating
/usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive 
instantiations from here: getoptImpl!(string, shared(string)*, 
string, shared(string)*, string, shared(VSync)*, string, 
shared(ulong)*, string, shared(bool)*, string, shared(Verbose)*)

Is getopt not supposed to be used with shared structs ?
May 11 2018
next sibling parent Danny Arends <Danny.Arends gmail.com> writes:
On Friday, 11 May 2018 at 17:25:44 UTC, Danny Arends wrote:

 Hey all,

 I have been working on creating a multi-threaded application, 
 so I have a shared configuration object which hold several 
 command line parameters (which I fill using getopt).

 The problem is that I get deprecation warnings when trying to 
 set numerical values:

 /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation: 
 read-modify-write operations are not allowed for shared 
 variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.

 using getopt with a shared object and boolean values seems 
 completely broken:

 /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation 
 not allowed on bool *receiver += 1
 /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template 
 instance `std.getopt.handleOption!(shared(bool)*)` error 
 instantiating
 /usr/include/dmd/phobos/std/getopt.d(435,15):        6 
 recursive instantiations from here: getoptImpl!(string, 
 shared(string)*, string, shared(string)*, string, 
 shared(VSync)*, string, shared(ulong)*, string, shared(bool)*, 
 string, shared(Verbose)*)

 Is getopt not supposed to be used with shared structs ?


small example:

import std.getopt : getopt;

struct Settings {
   size_t myvalue = 0;
}

shared(Settings) config;

int main(string[] args) {
   getopt(args, "m|myvalue", &(config.myvalue));
   return(0);
}

Changing size_t to bool, results in the compilation error 
mentioned in the previous post
May 11 2018
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, May 11, 2018 17:25:44 Danny Arends via Digitalmars-d-learn wrote:

 Hey all,

 I have been working on creating a multi-threaded application, so
 I have a shared configuration object which hold several command
 line parameters (which I fill using getopt).

 The problem is that I get deprecation warnings when trying to set
 numerical values:

 /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
 read-modify-write operations are not allowed for shared
 variables. Use core.atomic.atomicOp!"+="(*receiver, 1) instead.

 using getopt with a shared object and boolean values seems
 completely broken:

 /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation
 not allowed on bool *receiver += 1
 /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template
 instance `std.getopt.handleOption!(shared(bool)*)` error
 instantiating
 /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive
 instantiations from here: getoptImpl!(string, shared(string)*,
 string, shared(string)*, string, shared(VSync)*, string,
 shared(ulong)*, string, shared(bool)*, string, shared(Verbose)*)

 Is getopt not supposed to be used with shared structs ?


getopt is designed to be single-threaded. The keyword shared is not used a
single type in that module. If you want to use shared with anything in D,
you have three options:

1. Use core.atomic to atomically do stuff like increment a integer.

2. Use a mutex (or synchronized block) to protect access to a shared object.
You lock the mutex before accessing the object (and before _all_ accesses to
that object). Within that section of code, you cast away shared and operate
on the object as thread-local. Then at the end of that section, before
releasing the lock, you make sure that no thread-local references to the
shared object remain, and then free the lock. e.g. something like

synchronized(mutex)
{
    auto tls = cast(MyObject)mySharedObject.

    //... do stuff...

} // at this point, no thread-local references to mySharedObject
  // should // exist

In this scenario, the types in question are not designed to be used with
shared at all. They're designed to be thread-local. So, if they're marked as
shared, they're basically useless except when you protect them with a mutex
and correctly cast away shared in order to operate on the object while it's
protected by the mutex and therefore is thread-safe.

3. An object is designed to be used as shared. In this case, it can have
shared member functions, and they can be called on shared objects, but then
internally, the object has to deal with properly managing shared. It either

it's done internally by the object rather than the programmer using the
object.

synchronized classes are supposed to help with this particular case, but
unfortunately, they're not fully implemented, so they don't currently help.
But either way, the concept is still the same. You have on object that is
designed to work with shared and deals with all of the appropriate
protections internally.

So, those are your three options. In the case of getopt, if you want to use

like an int or pointer, so atomics aren't going to work, and it's not
designed to work with shared. So, if you want to do anything with getopt and
shared, you're going to have to protect it with a mutex.

That being said, I have to say that getopt seems like a really weird choice
for wanting to do anything with shared. You normally call it immediately at
the beginning of the program before doing anything with threads. The results
might then be passed on to other threads via std.concurrency or through
shared variables, but I wouldn't think that it would make a lot of sense to
try and used getopt from more than one thread.

- Jonathan M Davis
May 11 2018
parent reply Danny Arends <Danny.Arends gmail.com> writes:
On Friday, 11 May 2018 at 17:49:17 UTC, Jonathan M Davis wrote:

 On Friday, May 11, 2018 17:25:44 Danny Arends via 
 Digitalmars-d-learn wrote:

 [...]


 getopt is designed to be single-threaded. The keyword shared is 
 not used a single type in that module. If you want to use 
 shared with anything in D, you have three options:

 [...]


Hey Jonathan,

Thanks for the long and insightful answer.
The object is indeed constructed from the main thread, but 
afterwards multiple threads need to read the values given via the 
command line. since everything in the object is read only I was 
hoping to get away with making it shared.

I should just define tls variables to use with getopt and then 
set the corresponding variables in the shared object.

Danny
May 11 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, May 11, 2018 18:01:18 Danny Arends via Digitalmars-d-learn wrote:

 On Friday, 11 May 2018 at 17:49:17 UTC, Jonathan M Davis wrote:

 On Friday, May 11, 2018 17:25:44 Danny Arends via

 Digitalmars-d-learn wrote:

 [...]


 getopt is designed to be single-threaded. The keyword shared is
 not used a single type in that module. If you want to use
 shared with anything in D, you have three options:

 [...]


 Hey Jonathan,

 Thanks for the long and insightful answer.
 The object is indeed constructed from the main thread, but
 afterwards multiple threads need to read the values given via the
 command line. since everything in the object is read only I was
 hoping to get away with making it shared.

 I should just define tls variables to use with getopt and then
 set the corresponding variables in the shared object.


If you want to operate on that data as shared, then yes. But if you're
really just looking for each thread to have its own copy, I'd suggest that
you either give each thread its own copy on thread creation or pass it using
std.concurrency rather than trying to deal with a shared variable -
especially since the type system has no way to know that you're just
planning to read from the shared variable after that, and it will scream at
you for various operations - and once shared is fully locked down, it will
probably scream if you do much of _anything_ with shared, since at that
point, the compiler would only allow operations that were either marked with
shared or where it could guarantee that they were thread-safe.

- Jonathan M Davis
May 11 2018
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 5/11/18 1:25 PM, Danny Arends wrote:

 Hey all,
 
 I have been working on creating a multi-threaded application, so I have 
 a shared configuration object which hold several command line parameters 
 (which I fill using getopt).
 
 The problem is that I get deprecation warnings when trying to set 
 numerical values:
 
 /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation: 
 read-modify-write operations are not allowed for shared variables. Use 
 core.atomic.atomicOp!"+="(*receiver, 1) instead.
 
 using getopt with a shared object and boolean values seems completely 
 broken:
 
 /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not 
 allowed on bool *receiver += 1
 /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance 
 `std.getopt.handleOption!(shared(bool)*)` error instantiating
 /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive 
 instantiations from here: getoptImpl!(string, shared(string)*, string, 
 shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string, 
 shared(bool)*, string, shared(Verbose)*)
 
 Is getopt not supposed to be used with shared structs ?
 


No, just fill in a local copy, and then copy it to the shared location.

In the case where all you want is read-only access to the data, build it 
once and then cast to immutable:

immutable Config config;

void main(string[] args)
{
    Config localConfig;
    getopt(...); // fill in localConfig
    // VERY IMPORTANT, only do this ONCE, and don't use it before this line
    *(cast()&config) = localConfig;
}

It should now be accessible from all threads.

Normally, this is not encouraged (casting away immutable), but the 
compiler gets that an immutable global is allowed to be initialized once 
(in fact, you can do this without casting inside shared static 
constructors).

-Steve
May 11 2018
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, May 11, 2018 14:31:17 Steven Schveighoffer via Digitalmars-d-
learn wrote:

 On 5/11/18 1:25 PM, Danny Arends wrote:

 Hey all,

 I have been working on creating a multi-threaded application, so I have
 a shared configuration object which hold several command line parameters
 (which I fill using getopt).

 The problem is that I get deprecation warnings when trying to set
 numerical values:

 /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
 read-modify-write operations are not allowed for shared variables. Use
 core.atomic.atomicOp!"+="(*receiver, 1) instead.

 using getopt with a shared object and boolean values seems completely
 broken:

 /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not
 allowed on bool *receiver += 1
 /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance
 `std.getopt.handleOption!(shared(bool)*)` error instantiating
 /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive
 instantiations from here: getoptImpl!(string, shared(string)*, string,
 shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string,
 shared(bool)*, string, shared(Verbose)*)

 Is getopt not supposed to be used with shared structs ?


 No, just fill in a local copy, and then copy it to the shared location.

 In the case where all you want is read-only access to the data, build it
 once and then cast to immutable:

 immutable Config config;

 void main(string[] args)
 {
     Config localConfig;
     getopt(...); // fill in localConfig
     // VERY IMPORTANT, only do this ONCE, and don't use it before this
 line *(cast()&config) = localConfig;
 }

 It should now be accessible from all threads.

 Normally, this is not encouraged (casting away immutable), but the
 compiler gets that an immutable global is allowed to be initialized once
 (in fact, you can do this without casting inside shared static
 constructors).


Except that because this is done outside of a shared static constructor,
you're technically mutating immutable data. This _should_ work, but it is
violating the type system, and technically, the compiler is allowed to
assume that config is Config.init everywhere. In practice, I wouldn't expect
a problem, but because it's violating the type system, all bets are off.

- Jonathan M Davis
May 11 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 5/11/18 2:49 PM, Jonathan M Davis wrote:

 On Friday, May 11, 2018 14:31:17 Steven Schveighoffer via Digitalmars-d-
 learn wrote:

 On 5/11/18 1:25 PM, Danny Arends wrote:

 Hey all,

 I have been working on creating a multi-threaded application, so I have
 a shared configuration object which hold several command line parameters
 (which I fill using getopt).

 The problem is that I get deprecation warnings when trying to set
 numerical values:

 /usr/include/dmd/phobos/std/getopt.d(895,36): Deprecation:
 read-modify-write operations are not allowed for shared variables. Use
 core.atomic.atomicOp!"+="(*receiver, 1) instead.

 using getopt with a shared object and boolean values seems completely
 broken:

 /usr/include/dmd/phobos/std/getopt.d(895,34): Error: operation not
 allowed on bool *receiver += 1
 /usr/include/dmd/phobos/std/getopt.d(751,46): Error: template instance
 `std.getopt.handleOption!(shared(bool)*)` error instantiating
 /usr/include/dmd/phobos/std/getopt.d(435,15):        6 recursive
 instantiations from here: getoptImpl!(string, shared(string)*, string,
 shared(string)*, string, shared(VSync)*, string, shared(ulong)*, string,
 shared(bool)*, string, shared(Verbose)*)

 Is getopt not supposed to be used with shared structs ?


 No, just fill in a local copy, and then copy it to the shared location.

 In the case where all you want is read-only access to the data, build it
 once and then cast to immutable:

 immutable Config config;

 void main(string[] args)
 {
      Config localConfig;
      getopt(...); // fill in localConfig
      // VERY IMPORTANT, only do this ONCE, and don't use it before this
 line *(cast()&config) = localConfig;
 }

 It should now be accessible from all threads.

 Normally, this is not encouraged (casting away immutable), but the
 compiler gets that an immutable global is allowed to be initialized once
 (in fact, you can do this without casting inside shared static
 constructors).


 
 Except that because this is done outside of a shared static constructor,
 you're technically mutating immutable data. This _should_ work, but it is
 violating the type system, and technically, the compiler is allowed to
 assume that config is Config.init everywhere. In practice, I wouldn't expect
 a problem, but because it's violating the type system, all bets are off.


It's not a problem. The compiler cannot assume the value is .init 
because it doesn't have access to the whole codebase to see if some 
static ctor has changed it. Effectively, you ARE changing it in a static 
ctor, but just after all the other static ctors have run.

Crucially noted as well: don't use this value inside a static ctor!

-Steve
May 11 2018
prev sibling parent Basile B. <b2.temp gmx.com> writes:
On Friday, 11 May 2018 at 17:25:44 UTC, Danny Arends wrote:

 Hey all,


 Is getopt not supposed to be used with shared structs ?


I don't know but if you are opened to alternative i have just 
tested my getopt-like function at it works with shared, i don't 
know why... The design is different tho, it's not based on 
pointer, it rather generates the code to write the data without 
indirection.

links:
- 
https://github.com/BBasile/iz/commit/ac8b2e23214ec8c2d3a1000895b374f9c73f8ca7
- http://bbasile.github.io/iz/iz/options.html#handleArguments

Baz.
May 12 2018