www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - An Issue I Wish To Raise Awareness On

reply Jack Stouffer <jack jackstouffer.com> writes:
TL;DR: Issue 17658 [1] makes using shared very 
annoying/practically impossible.

Over the weekend, I attempted to fix Issue 15768 [2], which is an 
underlying problem in the std.stdio.File struct that stops it 
from closing properly when shared across threads. This is a big 
problem for users because stdin/out/err are all __gshared File, 
and are therefore unsafe. This makes for a really annoying 
situation where writeln("a") is  safe but stderr.writeln("a") 
isn't.

The obvious solution is to make stdin/out/err shared(File) and 
modify File to have shared overloads which either lock or use 
atomic ops safely. When I tried to write said functions, I ran 
into compilation issues that I couldn't diagnose until I ran 
across this thread by Atila [3]. The problem is, unlike a 
constructor, destructors and post-blits can't be overloaded with 
shared variants. Consider:

```
struct A
{
     this(string a) {}
     this(string a) shared {}

     ~this() {}
     ~this() shared {}

     this(this) {}
     this(this) shared {}
}

void main()
{
     shared f = A("");
}
```

Error: destructor f152.A.~this conflicts with destructor 
f152.A.~this at /d422/f152.d(6)
Error: function f152.A.__postblit conflicts with function 
f152.A.__postblit at /d422/f152.d(9)

This is further compounded with this

```
struct A
{
	~this() {}
}

void main()
{
	auto a = A();
	shared b = A();
}
```

Error: non-shared method f585.A.~this is not callable using a 
shared object

The only way to work around this is to create a new type that is 
defined as shared struct and copy over all of the code from the 
original type. This really hobbles shared in any real world 
context.

I ask that someone who knows the DMD code base could please take 
a look at this and see if this is something that can be fixed 
easily and without breakage.

[1] https://issues.dlang.org/show_bug.cgi?id=17658
[2] https://issues.dlang.org/show_bug.cgi?id=15768
[3] 
https://forum.dlang.org/post/sqazguejrcdtjimtjxtz forum.dlang.org
Jul 17 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 17 July 2017 at 14:26:19 UTC, Jack Stouffer wrote:

 TL;DR: Issue 17658 [1] makes using shared very 
 annoying/practically impossible.

 [...]


I fixed this already, should be in the next release.

Atila
Jul 17 2017
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 17 July 2017 at 17:41:58 UTC, Atila Neves wrote:

 On Monday, 17 July 2017 at 14:26:19 UTC, Jack Stouffer wrote:

 TL;DR: Issue 17658 [1] makes using shared very 
 annoying/practically impossible.

 [...]


 I fixed this already, should be in the next release.

 Atila


Are you sure? Because DMD nightly still errors:

https://run.dlang.io?compiler=dmd-nightly&source=struct%20A%0A%7B%0A%20%20%20%20this(string%20a)%20%7B%7D%0A%20%20%20%20this(string%20a)%20shared%20%7B%7D%0A%0A%20%20%20%20~this()%20%7B%7D%0A%20%20%20%20~this()%20shared%20%7B%7D%0A%0A%20%20%20%20this(this)%20%7B%7D%0A%20%20%20%20this(this)%20shared%20%7B%7D%0A%7D%0A%0Avoid%20main()%0A%7B%0A%20%20%20%20shared%20f%20%3D%20A(%22%22)%3B%0A%7D


(maybe we should remove the ban on URL shorteners for our own 
sites)
Jul 17 2017
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 17 July 2017 at 19:30:53 UTC, Jack Stouffer wrote:

 On Monday, 17 July 2017 at 17:41:58 UTC, Atila Neves wrote:

 On Monday, 17 July 2017 at 14:26:19 UTC, Jack Stouffer wrote:

 TL;DR: Issue 17658 [1] makes using shared very 
 annoying/practically impossible.

 [...]


 I fixed this already, should be in the next release.

 Atila


 Are you sure? Because DMD nightly still errors:

 https://run.dlang.io?compiler=dmd-nightly&source=struct%20A%0A%7B%0A%20%20%20%20this(string%20a)%20%7B%7D%0A%20%20%20%20this(string%20a)%20shared%20%7B%7D%0A%0A%20%20%20%20~this()%20%7B%7D%0A%20%20%20%20~this()%20shared%20%7B%7D%0A%0A%20%20%20%20this(this)%20%7B%7D%0A%20%20%20%20this(this)%20shared%20%7B%7D%0A%7D%0A%0Avoid%20main()%0A%7B%0A%20%20%20%20shared%20f%20%3D%20A(%22%22)%3B%0A%7D


 (maybe we should remove the ban on URL shorteners for our own 
 sites)


I think Atila was talking about this one:
struct A
{
	~this() {}
}

void main()
{
	auto a = A();
	shared b = A();
}

https://is.gd/kOYlWY
Jul 18 2017
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 18 July 2017 at 11:47:37 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Monday, 17 July 2017 at 19:30:53 UTC, Jack Stouffer wrote:

 On Monday, 17 July 2017 at 17:41:58 UTC, Atila Neves wrote:

 On Monday, 17 July 2017 at 14:26:19 UTC, Jack Stouffer wrote:

 TL;DR: Issue 17658 [1] makes using shared very 
 annoying/practically impossible.

 [...]


 I fixed this already, should be in the next release.

 Atila


 Are you sure? Because DMD nightly still errors:

 https://run.dlang.io?compiler=dmd-nightly&source=struct%20A%0A%7B%0A%20%20%20%20this(string%20a)%20%7B%7D%0A%20%20%20%20this(string%20a)%20shared%20%7B%7D%0A%0A%20%20%20%20~this()%20%7B%7D%0A%20%20%20%20~this()%20shared%20%7B%7D%0A%0A%20%20%20%20this(this)%20%7B%7D%0A%20%20%20%20this(this)%20shared%20%7B%7D%0A%7D%0A%0Avoid%20main()%0A%7B%0A%20%20%20%20shared%20f%20%3D%20A(%22%22)%3B%0A%7D


 (maybe we should remove the ban on URL shorteners for our own 
 sites)


 I think Atila was talking about this one:
 struct A
 {
 	~this() {}
 }

 void main()
 {
 	auto a = A();
 	shared b = A();
 }

 https://is.gd/kOYlWY


https://github.com/dlang/dmd/pull/6752
Jul 18 2017
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 18 July 2017 at 11:47:37 UTC, Petar Kirov 
[ZombineDev] wrote:

 I think Atila was talking about this one:
 struct A
 {
 	~this() {}
 }

 void main()
 {
 	auto a = A();
 	shared b = A();
 }


This is strange. There's nothing that suggests that struct A and 
its destructor is thread-safe, yet compiler assumes it is.
Jul 18 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 18 July 2017 at 15:03:07 UTC, Kagamin wrote:

 On Tuesday, 18 July 2017 at 11:47:37 UTC, Petar Kirov 
 [ZombineDev] wrote:

 I think Atila was talking about this one:
 struct A
 {
 	~this() {}
 }

 void main()
 {
 	auto a = A();
 	shared b = A();
 }


 This is strange. There's nothing that suggests that struct A 
 and its destructor is thread-safe, yet compiler assumes it is.


Except for a programmer explicitly and manually calling the 
destructor (in which case, don't), the destructor is only ever 
called by one thread.

Atila
Jul 18 2017
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, July 18, 2017 18:06:56 Atila Neves via Digitalmars-d wrote:

 On Tuesday, 18 July 2017 at 15:03:07 UTC, Kagamin wrote:

 On Tuesday, 18 July 2017 at 11:47:37 UTC, Petar Kirov

 [ZombineDev] wrote:

 I think Atila was talking about this one:
 struct A
 {

    ~this() {}

 }

 void main()
 {

    auto a = A();
    shared b = A();

 }


 This is strange. There's nothing that suggests that struct A
 and its destructor is thread-safe, yet compiler assumes it is.


 Except for a programmer explicitly and manually calling the
 destructor (in which case, don't), the destructor is only ever
 called by one thread.


It could still be a problem if the struct has a member variable that is a
reference type, because then something else could refer to that object, and
if it's shared, then you would need to protect it, and the operations that
shared prevents should still be prevented. For full-on value types, it
should be a non-issue though.

- Jonathan M Davis
Jul 18 2017
next sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 18 July 2017 at 19:24:18 UTC, Jonathan M Davis wrote:

 For full-on value types, it should be a non-issue though.


Not quite. Value types include resource identifiers, which may 
have threading requirements, e.g. GUI widget handles and OpenGL 
handles, assuming they are thread-safe and making them implicitly 
shared would be incorrect.
Jul 19 2017
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 19 Jul 2017 08:50:11 +0000
schrieb Kagamin <spam here.lot>:


 On Tuesday, 18 July 2017 at 19:24:18 UTC, Jonathan M Davis wrote:

 For full-on value types, it should be a non-issue though.  


 
 Not quite. Value types include resource identifiers, which may 
 have threading requirements, e.g. GUI widget handles and OpenGL 
 handles, assuming they are thread-safe and making them implicitly 
 shared would be incorrect.


That's exactly what I was opposing in the other post. These
handles are opaque and never change their value. Within the
Dlang language barrier they can be immutable and as such,
implicitly shared.
Your thinking is less technical, trying to find a best fit
between type system and foreign API, so that only handles with
a thread-safe API may become `shared`. I like the idea, but it
is impractical. It sometimes depends on whether a library was
compiled with multi-threading support or not and a value type
can be copied from and to shared anyways, rendering the safety
argument void:

	int x;
	shared int y = x;
	int z = y;

-- 
Marco
Jul 19 2017
parent reply Kagamin <spam here.lot> writes:
On Wednesday, 19 July 2017 at 12:56:38 UTC, Marco Leise wrote:

 That's exactly what I was opposing in the other post. These
 handles are opaque and never change their value. Within the
 Dlang language barrier they can be immutable and as such,
 implicitly shared.


Given transitivity of immutability the handle should have the 
same immutability as the resource it represents.


 It sometimes depends on whether a library was
 compiled with multi-threading support or not


Then you can communicate multithreading support with type system.


 and a value type can be copied from and to shared anyways, 
 rendering the safety
 argument void:

 	int x;
 	shared int y = x;
 	int z = y;


If it's overlooked, it doesn't mean D can't have proper sharing.
Jul 20 2017
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 20 Jul 2017 08:56:57 +0000
schrieb Kagamin <spam here.lot>:


 On Wednesday, 19 July 2017 at 12:56:38 UTC, Marco Leise wrote:

 That's exactly what I was opposing in the other post. These
 handles are opaque and never change their value. Within the
 Dlang language barrier they can be immutable and as such,
 implicitly shared.  


 
 Given transitivity of immutability the handle should have the 
 same immutability as the resource it represents.


I understand that you apply D keywords to C types in a
best fit fashion, to get some errors from the compiler when
you use them in the wrong context. It should work alright in
some APIs (maybe you can show me an example).

But since C does not have these qualifiers, one function may
be used for shared and unshared resources or the library may
even be compiled with or without thread-safety enabled and you
have to query that at *runtime*, where you have no help from
the type-system. So I believe, relying on the pattern will be
frustrating at times as not general enough.

What I seek to achieve by slapping immutable on things like
file descriptors and opaque types is the use as hash table
keys. As the hashed part of hash table keys must not change,
this approach enables us to use these types as keys and
statically verify immutability, too.

Because opaque structs and integer handles are used in C
APIs to *hide* the implementation details, the compiler is
deliberately left blind. That FILE* could as well be an
integer as far as transitivity of `immutable` goes. Nothing
the compiler *can see of it* will ever change. And that's all
the type system will ever care about really! Even if part of
that hidden structure is actually returned mutably by some
function, it is just an implementation detail. Whether you
slap `immutable` on anything there is mostly cosmetic.

Now what does that mean for type checks in practice?

1) Looking at POSIX' C `fgetc()` function, the stream is a
   plain FILE*: int fgetc (FILE *stream).
   Since I/O is thread-safe[1], it should ideally be `shared`
   the way you put it. And the way I look at it, it should be
   `immutable` on *our* side of the language barrier. (I.e.
   Dlang wont be able to change its contents or see the
   contents change.)

2) You can look up items by file descriptors or FILE* in hash
   tables implementations with immutable keys.

So we can take this away:
* Making a struct opaque, is implicitly making it immutable,
  because it's contents cannot be modified nor read directly - 
  the compiler cannot reason about its contents.
* Now we also have a layman's head-const, making resource
  pointers usable as immutable hash table keys.

As you can see my thinking revolves around the idea that hash
table keys must be immutable and that stems from the idea that
once hashed and sorted into a table, the hashed data must not
change. There are other approaches and druntime's AAs simply
allow mutable keys:

	void main() {
		struct S { int* i; }
		int a = 1, b = 2;
		uint[S] aa;
		aa[S(&a)] = 42;
		foreach(ref s_key; aa.byKey())
			// Allows "innocent" changes to the keys
			s_key.i = &b;
		foreach(ref s_key; aa.byKey())
			// AA doesn't find the key it just returned! Range violation.
			uint u = aa[s_key];
	}

-- 
Marco
Jul 25 2017
next sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 25 July 2017 at 19:22:11 UTC, Marco Leise wrote:

 I understand that you apply D keywords to C types in a
 best fit fashion, to get some errors from the compiler when
 you use them in the wrong context. It should work alright in
 some APIs (maybe you can show me an example).


I currently only use value types, and they unfortunately 
implicitly convert to all qualifiers.


 But since C does not have these qualifiers, one function may be 
 used for shared and unshared resources or the library may even 
 be compiled with or without thread-safety enabled and you have 
 to query that at *runtime*, where you have no help from the 
 type-system. So I believe, relying on the pattern will be 
 frustrating at times as not general enough.


This means you use custom multithreading approach to work with 
the resources, shared qualifier would reflect it.


 What I seek to achieve by slapping immutable on things like 
 file descriptors and opaque types is the use as hash table keys.


Reference types could benefit from this too, I use it regularly 

table design here. If this use case is useful, you shouldn't 
fight with the library.


 As the hashed part of hash table keys must not change, this 
 approach enables us to use these types as keys and statically 
 verify immutability, too.
 Because opaque structs and integer handles are used in C APIs 
 to *hide* the implementation details, the compiler is 
 deliberately left blind.


Blind compiler can't verify anything: it's blind. And ftell and 
lseek clearly indicate that file stream is not immutable.


    Since I/O is thread-safe[1], it should ideally be `shared`
    the way you put it.


Indeed, but only C11 specifies this.
Jul 28 2017
prev sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 25 July 2017 at 19:22:11 UTC, Marco Leise wrote:

    Since I/O is thread-safe[1], it should ideally be `shared`
    the way you put it.


Also it sounds like a bad decision.

void f()
{
   printf("hello ");
   printf("world\n");
}

If you have shared stdout and this function runs in 2 threads, 
you will get a character soup
---
hello hello world
world
---

If you have a separate line buffered stdout for each thread, you 
will get whole lines in the output
---
hello world
hello world
---
And incur no locking.
Jul 29 2017
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 18 July 2017 at 19:24:18 UTC, Jonathan M Davis wrote:

 On Tuesday, July 18, 2017 18:06:56 Atila Neves via 
 Digitalmars-d wrote:

 On Tuesday, 18 July 2017 at 15:03:07 UTC, Kagamin wrote:

 On Tuesday, 18 July 2017 at 11:47:37 UTC, Petar Kirov

 [ZombineDev] wrote:

 I think Atila was talking about this one:
 struct A
 {

    ~this() {}

 }

 void main()
 {

    auto a = A();
    shared b = A();

 }


 This is strange. There's nothing that suggests that struct A 
 and its destructor is thread-safe, yet compiler assumes it 
 is.


 Except for a programmer explicitly and manually calling the 
 destructor (in which case, don't), the destructor is only ever 
 called by one thread.


 It could still be a problem if the struct has a member variable 
 that is a reference type, because then something else could 
 refer to that object, and if it's shared, then you would need 
 to protect it, and the operations that shared prevents should 
 still be prevented. For full-on value types, it should be a 
 non-issue though.

 - Jonathan M Davis


Mmm, I guess so. As Marco pointed out, it's a similar problem 
with immutable (because the compiler casts it away before calling 
the destructor).

Although I dare say that anybody writing code that depends on 
such locking and destruction when shared is unlikely to get it 
right in the first place.

Atila
Jul 19 2017
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wednesday, July 19, 2017 2:29:04 PM MDT Atila Neves via Digitalmars-d 
wrote:

 On Tuesday, 18 July 2017 at 19:24:18 UTC, Jonathan M Davis wrote:

 On Tuesday, July 18, 2017 18:06:56 Atila Neves via

 Except for a programmer explicitly and manually calling the
 destructor (in which case, don't), the destructor is only ever
 called by one thread.


 It could still be a problem if the struct has a member variable
 that is a reference type, because then something else could
 refer to that object, and if it's shared, then you would need
 to protect it, and the operations that shared prevents should
 still be prevented. For full-on value types, it should be a
 non-issue though.

 - Jonathan M Davis


 Mmm, I guess so. As Marco pointed out, it's a similar problem
 with immutable (because the compiler casts it away before calling
 the destructor).

 Although I dare say that anybody writing code that depends on
 such locking and destruction when shared is unlikely to get it
 right in the first place.


Well, consider that something like a reference counted type would have to
get this right. Now, a reference counted type that worled with shared would
need to be written that way - simply slapping shared on such a smart pointer
type isn't going to work - but it would then be really bad for the
destructor to be treated as thread-local.

It really looks to me like having a thread-local dstructor for shared data
is just begging for problems. It may work in the simple cases, but it'll
fall apart in the more complicated ones, and I don't see how that's
acceptable.

Only really basic types are going to work as shared without being
specifically designed for it, so I'm inclined to think that it would be far
better for the language to be changed so that it supports having both a
shared and non-shared destructor rather than having shared objects work with
non-shared destructors.

- Jonathan M Davis
Jul 19 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 19 July 2017 at 20:23:18 UTC, Jonathan M Davis 
wrote:

 On Wednesday, July 19, 2017 2:29:04 PM MDT Atila Neves via 
 Digitalmars-d wrote:

 On Tuesday, 18 July 2017 at 19:24:18 UTC, Jonathan M Davis 
 wrote:

 On Tuesday, July 18, 2017 18:06:56 Atila Neves via

 Except for a programmer explicitly and manually calling the 
 destructor (in which case, don't), the destructor is only 
 ever called by one thread.


 It could still be a problem if the struct has a member 
 variable that is a reference type, because then something 
 else could refer to that object, and if it's shared, then 
 you would need to protect it, and the operations that shared 
 prevents should still be prevented. For full-on value types, 
 it should be a non-issue though.

 - Jonathan M Davis


 Mmm, I guess so. As Marco pointed out, it's a similar problem 
 with immutable (because the compiler casts it away before 
 calling the destructor).

 Although I dare say that anybody writing code that depends on 
 such locking and destruction when shared is unlikely to get it 
 right in the first place.


 Well, consider that something like a reference counted type 
 would have to get this right. Now, a reference counted type 
 that worled with shared would need to be written that way - 
 simply slapping shared on such a smart pointer type isn't going 
 to work - but it would then be really bad for the destructor to 
 be treated as thread-local.


Not necessarily - the reference counted smart pointer doesn't 
have to be `shared` itself to have a `shared` payload. I'm not 
even entirely sure what the advantage of it being `shared` would 
be, or even what that would really mean.

The way `automem.RefCounted` works right now is by doing atomic 
reference count manipulations by using reflection to know if the 
payload is `shared`.


 It really looks to me like having a thread-local dstructor for 
 shared data is just begging for problems. It may work in the 
 simple cases, but it'll fall apart in the more complicated 
 ones, and I don't see how that's acceptable.


You've definitely made me wonder about complicated cases, but I'd 
argue that they'd be rare. Destructors are (bar manually calling 
them) run in one thread. I'm having trouble imagining a situation 
where two threads have references to a `shared` object/value that 
is going to be destroyed deterministically.


 Only really basic types are going to work as shared without 
 being specifically designed for it, so I'm inclined to think 
 that it would be far better for the language to be changed so 
 that it supports having both a shared and non-shared destructor 
 rather than having shared objects work with non-shared 
 destructors.


Perhaps.

Atila
Jul 19 2017
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Wednesday, 19 July 2017 at 20:59:03 UTC, Atila Neves wrote:

 On Wednesday, 19 July 2017 at 20:23:18 UTC, Jonathan M Davis 
 wrote:

 On Wednesday, July 19, 2017 2:29:04 PM MDT Atila Neves via 
 Digitalmars-d wrote:

 On Tuesday, 18 July 2017 at 19:24:18 UTC, Jonathan M Davis 
 wrote:

 On Tuesday, July 18, 2017 18:06:56 Atila Neves via

 Except for a programmer explicitly and manually calling 
 the destructor (in which case, don't), the destructor is 
 only ever called by one thread.


 It could still be a problem if the struct has a member 
 variable that is a reference type, because then something 
 else could refer to that object, and if it's shared, then 
 you would need to protect it, and the operations that 
 shared prevents should still be prevented. For full-on 
 value types, it should be a non-issue though.

 - Jonathan M Davis


 Mmm, I guess so. As Marco pointed out, it's a similar problem 
 with immutable (because the compiler casts it away before 
 calling the destructor).

 Although I dare say that anybody writing code that depends on 
 such locking and destruction when shared is unlikely to get 
 it right in the first place.


 Well, consider that something like a reference counted type 
 would have to get this right. Now, a reference counted type 
 that worled with shared would need to be written that way - 
 simply slapping shared on such a smart pointer type isn't 
 going to work - but it would then be really bad for the 
 destructor to be treated as thread-local.


 Not necessarily - the reference counted smart pointer doesn't 
 have to be `shared` itself to have a `shared` payload.


Agreed. I'm exploring doing the same for std.stdio.File.


 I'm not even entirely sure what the advantage of it being 
 `shared` would be, or even what that would really mean.


There are plenty of cases where you need the smart pointer to be 
shared itself -
e.g. member composition and global variables. See also: [0]

Note that this doesn't play well with regular [1] value types 
becuase e.g. you don't have control over the synthesized bit-blit 
for this(this) and so you can't assume that structs with a single 
pointer member are updated atomically, even if would write the 
opAssign that way. In C++17 atomic_shared_ptr has it's 
copy-constructor and assign operator deleted. You can only do 
atomic<T> like ops with it and derive a plain shared_ptr<T> from 
it, kind-of like core.atomic's HeadUnshared(T).

[0]: 
https://www.justsoftwaresolutions.co.uk/threading/why-do-we-need-atomic_shared_ptr.html
[1]: http://stepanovpapers.com/DeSt98.pdf (definition of regular 
types).
Jul 19 2017
parent reply Kagamin <spam here.lot> writes:
On Wednesday, 19 July 2017 at 21:50:32 UTC, Petar Kirov 
[ZombineDev] wrote:

 Note that this doesn't play well with regular [1] value types 
 becuase e.g. you don't have control over the synthesized 
 bit-blit for this(this) and so you can't assume that structs 
 with a single pointer member are updated atomically, even if 
 would write the opAssign that way. In C++17 atomic_shared_ptr 
 has it's copy-constructor and assign operator deleted. You can 
 only do atomic<T> like ops with it and derive a plain 
 shared_ptr<T> from it, kind-of like core.atomic's 
 HeadUnshared(T).


Huh? Why opAssign can't just do what atomic<T> does?
Jul 20 2017
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 20 July 2017 at 10:16:21 UTC, Kagamin wrote:

 On Wednesday, 19 July 2017 at 21:50:32 UTC, Petar Kirov 
 [ZombineDev] wrote:

 Note that this doesn't play well with regular [1] value types 
 becuase e.g. you don't have control over the synthesized 
 bit-blit for this(this) and so you can't assume that structs 
 with a single pointer member are updated atomically, even if 
 would write the opAssign that way. In C++17 atomic_shared_ptr 
 has it's copy-constructor and assign operator deleted. You can 
 only do atomic<T> like ops with it and derive a plain 
 shared_ptr<T> from it, kind-of like core.atomic's 
 HeadUnshared(T).


 Huh? Why opAssign can't just do what atomic<T> does?


opAssign is fine, the problem is with the this(this).
Jul 20 2017
prev sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 20 July 2017 at 10:16:21 UTC, Kagamin wrote:

 On Wednesday, 19 July 2017 at 21:50:32 UTC, Petar Kirov 
 [ZombineDev] wrote:

 Note that this doesn't play well with regular [1] value types 
 becuase e.g. you don't have control over the synthesized 
 bit-blit for this(this) and so you can't assume that structs 
 with a single pointer member are updated atomically, even if 
 would write the opAssign that way. In C++17 atomic_shared_ptr 
 has it's copy-constructor and assign operator deleted. You can 
 only do atomic<T> like ops with it and derive a plain 
 shared_ptr<T> from it, kind-of like core.atomic's 
 HeadUnshared(T).


 Huh? Why opAssign can't just do what atomic<T> does?


Also note that atomic<T> doesn't have neither copy constructor 
nor assignment operator:

http://en.cppreference.com/w/cpp/atomic/atomic/atomic


 atomic( const atomic& ) = delete;


http://en.cppreference.com/w/cpp/atomic/atomic/operator%3D


 atomic& operator=( const atomic& ) = delete;
 atomic& operator=( const atomic& ) volatile = delete;
Jul 20 2017
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wednesday, July 19, 2017 8:59:03 PM MDT Atila Neves via Digitalmars-d 
wrote:

 On Wednesday, 19 July 2017 at 20:23:18 UTC, Jonathan M Davis

 wrote:

 On Wednesday, July 19, 2017 2:29:04 PM MDT Atila Neves via

 Digitalmars-d wrote:

 On Tuesday, 18 July 2017 at 19:24:18 UTC, Jonathan M Davis

 wrote:

 On Tuesday, July 18, 2017 18:06:56 Atila Neves via


 Except for a programmer explicitly and manually calling the
 destructor (in which case, don't), the destructor is only
 ever called by one thread.


 It could still be a problem if the struct has a member
 variable that is a reference type, because then something
 else could refer to that object, and if it's shared, then
 you would need to protect it, and the operations that shared
 prevents should still be prevented. For full-on value types,
 it should be a non-issue though.

 - Jonathan M Davis


 Mmm, I guess so. As Marco pointed out, it's a similar problem
 with immutable (because the compiler casts it away before
 calling the destructor).

 Although I dare say that anybody writing code that depends on
 such locking and destruction when shared is unlikely to get it
 right in the first place.


 Well, consider that something like a reference counted type
 would have to get this right. Now, a reference counted type
 that worled with shared would need to be written that way -
 simply slapping shared on such a smart pointer type isn't going
 to work - but it would then be really bad for the destructor to
 be treated as thread-local.


 Not necessarily - the reference counted smart pointer doesn't
 have to be `shared` itself to have a `shared` payload. I'm not
 even entirely sure what the advantage of it being `shared` would
 be, or even what that would really mean.

 The way `automem.RefCounted` works right now is by doing atomic
 reference count manipulations by using reflection to know if the
 payload is `shared`.


Okay, but my point is that it's perfectly legal to have a shared object that
refers to other shared objects that it does not own, and casting away shared
in the destructor means that the compiler can no longer enforce shared like
it's supposed to.


 It really looks to me like having a thread-local dstructor for
 shared data is just begging for problems. It may work in the
 simple cases, but it'll fall apart in the more complicated
 ones, and I don't see how that's acceptable.


 You've definitely made me wonder about complicated cases, but I'd
 argue that they'd be rare. Destructors are (bar manually calling
 them) run in one thread. I'm having trouble imagining a situation
 where two threads have references to a `shared` object/value that
 is going to be destroyed deterministically.


The issue isn't the object being destroyed. It's what it refers to via its
member variables. For instance, what if an object were to remove itself from
a shared list when it's destroyed (e.g. because it's an observer in the
observer pattern). The object has a reference to the list, but it doesn't
own it. So, the fact that the destructor is only run in a single thread
doesn't help any. You could have ten of these objects all being destroyed
and accessing the same list at the same time from different threads, and if
the destructor treats the objects as thread-local - and thus treats all of
the member variables as thread-local, then you won't get the compiler errors
that you're supposed to get when you do something non-atomic with shared.

Sure, the list could be designed in such a way that it protects itself
against threading issues, but it's perfectly legal to have it be shared and
require that anyone using it lock a mutex, and destructors need to take that
into account just like any other function would.

Basically, what these changes have done is act like all destructors are part
of a synchronized class (as described in TDPL) where shared is stripped off
of the member variables - except that _all_ of shared has been stripped off
instead of just the outer layer, but these aren't synchronized classes, and
they don't provide any guarantees about references not escaping or locking
occurring, and not even synchronized classes can safely cast away shared
except for the outer layer. As such, from what I can tell, these changes are
very broken. Yes, we need a solution which allows us to deal with shared
destructors properly, but casting away shared without any guarantees beyond
the fact that no other threads are calling any member functions on that
object at that time simply does not guarantee that shared objects have been
properly protected. To do that, you'd essentially need synchronized classes,
and even they can only strip off the outer layer of shared, not the whole
thing. Yes, destructors have fewer problems with shared than most functions,
but there's nothing about then which makes them immune to issues with
shared.

- Jonathan M Davis
Jul 19 2017
parent reply Dominikus Dittes Scherkl <Dominikus.Scherkl continental-corporation.com> writes:
On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis 
wrote:

 The issue isn't the object being destroyed. It's what it refers 
 to via its member variables. For instance, what if an object 
 were to remove itself from a shared list when it's destroyed 
 (e.g. because it's an observer in the observer pattern). The 
 object has a reference to the list, but it doesn't own it.


So, even a thread-local object that has references to a shared 
list
has to handle those as shared, even in its non-shared destructor.
I can't follow your argument.
Jul 20 2017
next sibling parent reply Kagamin <spam here.lot> writes:
On Thursday, 20 July 2017 at 07:40:35 UTC, Dominikus Dittes 
Scherkl wrote:

 On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis 
 wrote:

 The issue isn't the object being destroyed. It's what it 
 refers to via its member variables. For instance, what if an 
 object were to remove itself from a shared list when it's 
 destroyed (e.g. because it's an observer in the observer 
 pattern). The object has a reference to the list, but it 
 doesn't own it.


 So, even a thread-local object that has references to a shared 
 list
 has to handle those as shared, even in its non-shared 
 destructor.
 I can't follow your argument.


Thread local object can't be contained in a shared list, the list 
is referred as unqualified, and thread local object will be 
contained in a thread local list, and shared object will be 
contained in a shared list because of transitivity of the shared 
qualifier.
Jul 20 2017
parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Thursday, 20 July 2017 at 10:19:30 UTC, Kagamin wrote:

 On Thursday, 20 July 2017 at 07:40:35 UTC, Dominikus Dittes 
 Scherkl wrote:

 On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis 
 wrote:

 The issue isn't the object being destroyed. It's what it 
 refers to via its member variables. For instance, what if an 
 object were to remove itself from a shared list when it's 
 destroyed (e.g. because it's an observer in the observer 
 pattern). The object has a reference to the list, but it 
 doesn't own it.


 So, even a thread-local object that has references to a shared 
 list
 has to handle those as shared, even in its non-shared 
 destructor.
 I can't follow your argument.


 Thread local object can't be contained in a shared list, the 
 list is referred as unqualified, and thread local object will 
 be contained in a thread local list, and shared object will be 
 contained in a shared list because of transitivity of the 
 shared qualifier.


It's the other way around:

ThreadLocal tl;

struct ThreadLocal
{
     shared(ListNode*)* listHead;
     shared(ListNode)*  listNode;

     ~this()
     {
         listHead.removeNodeFromList(listNode);
     }
}
Jul 20 2017
prev sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thursday, July 20, 2017 07:40:35 Dominikus Dittes Scherkl via 
Digitalmars-d wrote:

 On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis

 wrote:

 The issue isn't the object being destroyed. It's what it refers
 to via its member variables. For instance, what if an object
 were to remove itself from a shared list when it's destroyed
 (e.g. because it's an observer in the observer pattern). The
 object has a reference to the list, but it doesn't own it.


 So, even a thread-local object that has references to a shared
 list
 has to handle those as shared, even in its non-shared destructor.
 I can't follow your argument.


You can't just strip off shared. To do so defeats the purpose of shared. If
you have something like

struct S
{
     shared List<Foo> _list;

     ~this()
     {
        ...
     }
}

then inside of the destructor, _list is not treated as shared, meaning that
none of the compiler protections for shared are in place, no locking has
occurred, and the compiler is free to make optimizations based on the wrong
assumption that all of the member variables are thread-local. If nothing
else has access to that list, then it'll work, but if anything else does -
and if it's a reference type, that's perfectly possible - then you have a
threading problem, because shared has been violated.

Except in cases where the member variables are all value types and thus no
other references to them should exist when the destructor is called,
stripping away shared from them means that the compiler can no longer
properly enforce shared, and it's going to make the wrong assumptions about
whether the data can be treated as thread-local or not.

If we go with the assumption that nothing has pointers to the member
variables (since doing so would be  system, and they're only valid so long
as the struct isn't moved anyway), you can probably strip off the outer
layer of shared safely in the destructor, but if you're dealing with a
reference type, anything it points to needs to still be treated as shared.

- Jonathan M Davis
Jul 20 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 20 July 2017 at 21:20:46 UTC, Jonathan M Davis wrote:

 On Thursday, July 20, 2017 07:40:35 Dominikus Dittes Scherkl 
 via Digitalmars-d wrote:

 On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis

 wrote:

 The issue isn't the object being destroyed. It's what it 
 refers to via its member variables. For instance, what if an 
 object were to remove itself from a shared list when it's 
 destroyed (e.g. because it's an observer in the observer 
 pattern). The object has a reference to the list, but it 
 doesn't own it.


 So, even a thread-local object that has references to a shared
 list
 has to handle those as shared, even in its non-shared 
 destructor.
 I can't follow your argument.


 You can't just strip off shared. To do so defeats the purpose 
 of shared. If you have something like

 struct S
 {
      shared List<Foo> _list;

      ~this()
      {
         ...
      }
 }


This is fine. What dmd does now is strip shared off of the `this` 
pointer, not the member variables. There's only a problem if the 
sharedness of the member variable(s) depends on sharedness of the 
enclosing object.


 then inside of the destructor, _list is not treated as shared,


It is.

Atila
Jul 21 2017
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, July 21, 2017 08:37:51 Atila Neves via Digitalmars-d wrote:

 On Thursday, 20 July 2017 at 21:20:46 UTC, Jonathan M Davis wrote:

 On Thursday, July 20, 2017 07:40:35 Dominikus Dittes Scherkl

 via Digitalmars-d wrote:

 On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis

 wrote:

 The issue isn't the object being destroyed. It's what it
 refers to via its member variables. For instance, what if an
 object were to remove itself from a shared list when it's
 destroyed (e.g. because it's an observer in the observer
 pattern). The object has a reference to the list, but it
 doesn't own it.


 So, even a thread-local object that has references to a shared
 list
 has to handle those as shared, even in its non-shared
 destructor.
 I can't follow your argument.


 You can't just strip off shared. To do so defeats the purpose
 of shared. If you have something like

 struct S
 {

      shared List<Foo> _list;

      ~this()
      {

         ...

      }

 }




Wow. I've been doing too much C++ lately apparently, since I used <>. :|


 This is fine. What dmd does now is strip shared off of the `this`
 pointer, not the member variables. There's only a problem if the
 sharedness of the member variable(s) depends on sharedness of the
 enclosing object.


What happens with something like

struct S
{
    Foo* _foo;

    ~this() {...}
}

shared S s;

Inside the destructor, is what _foo points to still treated as shared:
shared(Foo)*? i.e. is the outer layer of shared the only layer being made
thread-local - like what would supposedly happen with synchronized classes?
If so, then that largely solves the problem. The only issue is pointers to
the member variables, which would not be  safe but would still technically
be possible, in which case something outside the struct could still
reference the member variables from another thread. But given that that's
going to blow up in your face soon thereafter anyway, since the object is
being destroyed (and thus you screwed up making sure that your  system code
was  safe), that's probably fine.

However, if _foo is treated as Foo* instead of shared(Foo)* in the
destructor, then there definitely is a problem. The fact that when the
member variable is explicitly shared, it continues to be treated as shared
definitely reduces the problem, but it doesn't fully close the hole. The
parts of the member variables not directly in the object still need to be
treated as shared, because they aren't necessarily owned or controlled by
the object and could legally and  safely be manipulated from other threads
even while the destructor is running. So, are they still treated as shared,
or are they treated as fully thread-local? I would have thought that they'd
still be treated as thread-local given that the shared part is then only
known to the variable that was marked as shared and not the destructor
itself, since it's the same destructor for thread-local and shared objects.

And if the destructor treats the member variables as completely thread-local
(rather than just the outer layer as thread local) even when the object
itself was shared, then I don't think that this is a viable solution. It
would either need to be made illegal to make an object shared if it has
indirections and a destructor, or it needs to have a shared destructor.

- Jonathan M Davis
Jul 21 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 21 July 2017 at 22:02:41 UTC, Jonathan M Davis wrote:

 On Friday, July 21, 2017 08:37:51 Atila Neves via Digitalmars-d 
 wrote:

 On Thursday, 20 July 2017 at 21:20:46 UTC, Jonathan M Davis 
 wrote:

 On Thursday, July 20, 2017 07:40:35 Dominikus Dittes Scherkl

 via Digitalmars-d wrote:

 On Wednesday, 19 July 2017 at 22:35:43 UTC, Jonathan M Davis

 wrote:

 The issue isn't the object being destroyed. It's what it 
 refers to via its member variables. For instance, what if 
 an object were to remove itself from a shared list when 
 it's destroyed (e.g. because it's an observer in the 
 observer pattern). The object has a reference to the 
 list, but it doesn't own it.


 So, even a thread-local object that has references to a 
 shared
 list
 has to handle those as shared, even in its non-shared
 destructor.
 I can't follow your argument.


 You can't just strip off shared. To do so defeats the 
 purpose of shared. If you have something like

 struct S
 {

      shared List<Foo> _list;

      ~this()
      {

         ...

      }

 }




 Wow. I've been doing too much C++ lately apparently, since I 
 used <>. :|


I noticed, but I wasn't going to say anything ;)


 This is fine. What dmd does now is strip shared off of the 
 `this` pointer, not the member variables. There's only a 
 problem if the sharedness of the member variable(s) depends on 
 sharedness of the enclosing object.


 What happens with something like

 struct S
 {
     Foo* _foo;

     ~this() {...}
 }

 shared S s;

 Inside the destructor, is what _foo points to still treated as 
 shared: shared(Foo)*?


No. This is what I meant by the sharedness depening on the 
enclosing object. However, there's a workaround:

struct Foo { }


struct S {

     Foo* _foo;
     bool _isShared;

     this(this T, U)(U foo) if(is(T == shared) && is(U == 
shared(Foo)*) || !is(T == shared) && is(U == Foo*)) {
         static if(is(T == shared)) _isShared = true;
         _foo = foo;
     }

     ~this() {
         import std.stdio: writeln;
         _isShared ? writeln("shared dtor") : writeln("non-shared 
dtor");
     }
}

void main() {
     auto f = Foo();
     auto sf = shared Foo();
     auto s = S(&f);
     auto ss = shared S(&sf);
}


It's annoying to use that bool up memory-wise, but I assume it's 
not a big deal for most applications.

In any case, that example wouldn't have worked anyway before my 
change to dmd - even creating the S struct would've been a 
compiler error.

Atila
Jul 24 2017
next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Monday, July 24, 2017 2:30:01 PM MDT Atila Neves via Digitalmars-d wrote:

 This is fine. What dmd does now is strip shared off of the
 `this` pointer, not the member variables. There's only a
 problem if the sharedness of the member variable(s) depends on
 sharedness of the enclosing object.


 What happens with something like

 struct S
 {

     Foo* _foo;

     ~this() {...}

 }

 shared S s;

 Inside the destructor, is what _foo points to still treated as
 shared: shared(Foo)*?


 No. This is what I meant by the sharedness depening on the
 enclosing object. However, there's a workaround:

 struct Foo { }


 struct S {

      Foo* _foo;
      bool _isShared;

      this(this T, U)(U foo) if(is(T == shared) && is(U ==
 shared(Foo)*) || !is(T == shared) && is(U == Foo*)) {
          static if(is(T == shared)) _isShared = true;
          _foo = foo;
      }

      ~this() {
          import std.stdio: writeln;
          _isShared ? writeln("shared dtor") : writeln("non-shared
 dtor");
      }
 }

 void main() {
      auto f = Foo();
      auto sf = shared Foo();
      auto s = S(&f);
      auto ss = shared S(&sf);
 }


 It's annoying to use that bool up memory-wise, but I assume it's
 not a big deal for most applications.

 In any case, that example wouldn't have worked anyway before my
 change to dmd - even creating the S struct would've been a
 compiler error.


The problem with this is that this means that shared is not being properly
enforced by the compiler. Your workaround is a way for the programmer to
figure out if the object is shared and do something differently based on
that, but for the compiler to do what it's supposed to be doing with shared
(e.g. prevent non-atomic operations), any indirections in the member
variables must continue to be typed as shared inside the destructor, and
that's clearly not happening right now, which is a serious problem IMHO. The
situation may be better thanks to your changes in that some stuff is now
possible that should be possible and was not before, but it's not completely
sound as far as the type system goes, and we really should be fixing it so
that shared is properly enforced rather than just blindly stripped off.

- Jonathan M Davis
Jul 24 2017
parent Atila Neves <atila.neves gmail.com> writes:
On Monday, 24 July 2017 at 20:30:25 UTC, Jonathan M Davis wrote:

 On Monday, July 24, 2017 2:30:01 PM MDT Atila Neves via 
 Digitalmars-d wrote:

 This is fine. What dmd does now is strip shared off of the 
 `this` pointer, not the member variables. There's only a 
 problem if the sharedness of the member variable(s) depends 
 on sharedness of the enclosing object.


 What happens with something like

 struct S
 {

     Foo* _foo;

     ~this() {...}

 }

 shared S s;

 Inside the destructor, is what _foo points to still treated 
 as
 shared: shared(Foo)*?


 No. This is what I meant by the sharedness depening on the 
 enclosing object. However, there's a workaround:

 struct Foo { }


 struct S {

      Foo* _foo;
      bool _isShared;

      this(this T, U)(U foo) if(is(T == shared) && is(U ==
 shared(Foo)*) || !is(T == shared) && is(U == Foo*)) {
          static if(is(T == shared)) _isShared = true;
          _foo = foo;
      }

      ~this() {
          import std.stdio: writeln;
          _isShared ? writeln("shared dtor") : 
 writeln("non-shared
 dtor");
      }
 }

 void main() {
      auto f = Foo();
      auto sf = shared Foo();
      auto s = S(&f);
      auto ss = shared S(&sf);
 }


 It's annoying to use that bool up memory-wise, but I assume 
 it's not a big deal for most applications.

 In any case, that example wouldn't have worked anyway before 
 my change to dmd - even creating the S struct would've been a 
 compiler error.


 The problem with this is that this means that shared is not 
 being properly enforced by the compiler. Your workaround is a 
 way for the programmer to figure out if the object is shared 
 and do something differently based on that, but for the 
 compiler to do what it's supposed to be doing with shared (e.g. 
 prevent non-atomic operations), any indirections in the member 
 variables must continue to be typed as shared inside the 
 destructor, and that's clearly not happening right now, which 
 is a serious problem IMHO. The situation may be better thanks 
 to your changes in that some stuff is now possible that should 
 be possible and was not before, but it's not completely sound 
 as far as the type system goes, and we really should be fixing 
 it so that shared is properly enforced rather than just blindly 
 stripped off.

 - Jonathan M Davis


I agree that this could be a problem, and that the proper 
solution is probably to allow the user to define more than one 
destructor. The problem isn't just with shared - immutable is 
similar, since you'd be able to invoke undefined behaviour from 
the destructor since immutable would be cast away and the 
compiler wouldn't even warn you. And that was already the 
behaviour in dmd. I think the situation isn't ideal but better 
than before.

I also think that while the problem exists, I don't think it'll 
be common. This would only affect structs that can be shared 
_and_ non-shared (or immutable).

This will require a DIP, methinks.

Atila

Atila
Jul 25 2017
prev sibling parent Kagamin <spam here.lot> writes:
On Monday, 24 July 2017 at 14:30:01 UTC, Atila Neves wrote:

 struct Foo { }


 struct S {

     Foo* _foo;
     bool _isShared;

     this(this T, U)(U foo) if(is(T == shared) && is(U == 
 shared(Foo)*) || !is(T == shared) && is(U == Foo*)) {
         static if(is(T == shared)) _isShared = true;
         _foo = foo;
     }

     ~this() {
         import std.stdio: writeln;
         _isShared ? writeln("shared dtor") : 
 writeln("non-shared dtor");
     }
 }

 void main() {
     auto f = Foo();
     auto sf = shared Foo();
     auto s = S(&f);
     auto ss = shared S(&sf);
 }


Exactly this. You must design struct to support shared type, in 
which case it's better and more straightforward to just write 
shared destructor rather than work it around. Same for immutable.
Jul 25 2017
prev sibling parent reply Kagamin <spam here.lot> writes:
On Wednesday, 19 July 2017 at 20:59:03 UTC, Atila Neves wrote:

 Not necessarily - the reference counted smart pointer doesn't 
 have to be `shared` itself to have a `shared` payload.


Yes, but it can be done either way. It's actually what Jack is 
trying to do: make stdout shared and reference counted: 
https://issues.dlang.org/show_bug.cgi?id=15768#c7


 I'm not even entirely sure what the advantage of it being 
 `shared` would be, or even what that would really mean.


It will be thread safe and its lifetime will be automatically 
managed.


 You've definitely made me wonder about complicated cases, but 
 I'd argue that they'd be rare. Destructors are (bar manually 
 calling them) run in one thread. I'm having trouble imagining a 
 situation where two threads have references to a `shared` 
 object/value that is going to be destroyed deterministically.


A mutex, a file, a socket, any shareable resource. Though I agree 
that reference counting of shared resources should be optimized 
by thread local counters.
Jul 20 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 20 July 2017 at 10:15:26 UTC, Kagamin wrote:

 On Wednesday, 19 July 2017 at 20:59:03 UTC, Atila Neves wrote:

 Not necessarily - the reference counted smart pointer doesn't 
 have to be `shared` itself to have a `shared` payload.


 Yes, but it can be done either way. It's actually what Jack is 
 trying to do: make stdout shared and reference counted: 
 https://issues.dlang.org/show_bug.cgi?id=15768#c7


 I'm not even entirely sure what the advantage of it being 
 `shared` would be, or even what that would really mean.


 It will be thread safe and its lifetime will be automatically 
 managed.


 You've definitely made me wonder about complicated cases, but 
 I'd argue that they'd be rare. Destructors are (bar manually 
 calling them) run in one thread. I'm having trouble imagining 
 a situation where two threads have references to a `shared` 
 object/value that is going to be destroyed deterministically.


 A mutex, a file, a socket, any shareable resource. Though I 
 agree that reference counting of shared resources should be 
 optimized by thread local counters.


Mutexes and sockets are classes, so not destroyed 
deterministically.

Anything that is `shared` is likely to be a reference (pointer, 
class...), or a global. Either way the compiler-generated 
destructor call isn't going to exist, which means it's probably 
ok to cast away shared when the compiler inserts the automatic 
call to a destructor at the end of scope.

Atila
Jul 21 2017
parent reply Kagamin <spam here.lot> writes:
On Friday, 21 July 2017 at 08:51:27 UTC, Atila Neves wrote:

 Mutexes and sockets are classes, so not destroyed 
 deterministically.


They should, like any unmanaged resources, e.g. by wrapping in a 
smart pointer. Imagine 10000 lingering tcp connections 
accumulated over time due to poor timing of destruction, it 
becomes a stress test for the server.


 Anything that is `shared` is likely to be a reference (pointer, 
 class...), or a global. Either way the compiler-generated 
 destructor call isn't going to exist, which means it's probably 
 ok to cast away shared when the compiler inserts the automatic 
 call to a destructor at the end of scope.


These are contradictory. Does the automatic destructor call exist 
or not?
Jul 21 2017
next sibling parent Kagamin <spam here.lot> writes:
Hmm, if proper implementation of a shared smart pointer is 
impossible, it probably means that such smart pointer should be 
typed unshared when passed around. But then it doesn't make sense 
to call unshared destructor on shared smart pointer anyway, 
because it's not designed to be typed shared. In this case 
absence of shared destructor will indicate that the object 
doesn't support being shared and the compiler should reject the 
code.
Jul 21 2017
prev sibling next sibling parent Moritz Maxeiner <moritz ucworks.org> writes:
On Friday, 21 July 2017 at 11:57:11 UTC, Kagamin wrote:

 On Friday, 21 July 2017 at 08:51:27 UTC, Atila Neves wrote:

 Mutexes and sockets are classes, so not destroyed 
 deterministically.


 They should, like any unmanaged resources


I tend to agree, although


 e.g. by wrapping in a smart pointer.


objects that manage their own lifetime limit the design space 
unnecessarily.
It's better to use normal structs to wrap resources (RAII) and 
then build whatever object lifetime management scheme one wants 
(including reference counting) on top of that.
Jul 21 2017
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Friday, 21 July 2017 at 11:57:11 UTC, Kagamin wrote:

 On Friday, 21 July 2017 at 08:51:27 UTC, Atila Neves wrote:

 Mutexes and sockets are classes, so not destroyed 
 deterministically.


 They should, like any unmanaged resources, e.g. by wrapping in 
 a smart pointer. Imagine 10000 lingering tcp connections 
 accumulated over time due to poor timing of destruction, it 
 becomes a stress test for the server.


 Anything that is `shared` is likely to be a reference 
 (pointer, class...), or a global. Either way the 
 compiler-generated destructor call isn't going to exist, which 
 means it's probably ok to cast away shared when the compiler 
 inserts the automatic call to a destructor at the end of scope.


 These are contradictory. Does the automatic destructor call 
 exist or not?


What I'm trying to say is that `shared` values will usually be 
references or globals and therefore there won't be a 
compiler-generated call to the destructor at the end of scope.

When the compiler _does_ generate a call to the destructor, the 
value is unlikely to be shared.

Since then I've thought that sending a value to another thread 
does indeed create a shared value with defined scope. Or, for 
that matter, calling any function that takes shared values (but 
those are rare, so it'll usually be `send`).

I think I've not done a good job of explaining the destructor 
fix: what it changes is the code the compiler writes for you when 
variables go out of scope, i.e. the automatic destructor call 
casts away shared. It already did the same thing for immutable, 
otherwise you wouldn't be able to put immutable values on the 
stack.

Atila
Jul 21 2017
prev sibling next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 18 July 2017 at 11:47:37 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Monday, 17 July 2017 at 19:30:53 UTC, Jack Stouffer wrote:

 [...]


 I think Atila was talking about this one:
 struct A
 {
 	~this() {}
 }

 void main()
 {
 	auto a = A();
 	shared b = A();
 }

 https://is.gd/kOYlWY


That's what I meant. I was on a train and only skimmed through.

Atila
Jul 18 2017
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
On Tuesday, 18 July 2017 at 11:47:37 UTC, Petar Kirov 
[ZombineDev] wrote:

 I think Atila was talking about this one:
 struct A
 {
 	~this() {}
 }

 void main()
 {
 	auto a = A();
 	shared b = A();
 }


Shouldn't it be :
struct A
{
     ~this() shared {}
}

void main()
{
     auto a = A();
     shared b = A();
}
?

Because handling theard-local data as shared is safe as far as I 
remember, but not the other way round.

And if you want a destructor which works with both immutable and 
normal, shouldn't it be a const destructor?
Jul 19 2017
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Wednesday, 19 July 2017 at 14:56:58 UTC, Dukc wrote:

 Shouldn't it be :
 struct A
 {
     ~this() shared {}
 }

 void main()
 {
     auto a = A();
     shared b = A();
 }
 ?

 Because handling theard-local data as shared is safe as far as 
 I remember, but not the other way round.


Non-shared structs/classes can't call shared methods.

Unless you're saying that the above should work even though it 
currently doesn't. Even then, I don't know about that. If your 
type is complex enough to need a shared dtor then the dtor 
probably needs to do some locking or extra checks. You don't want 
to impose that cost on a struct instance which isn't shared.
Jul 19 2017
parent Dukc <ajieskola gmail.com> writes:
On Wednesday, 19 July 2017 at 15:38:08 UTC, Jack Stouffer wrote:

 Unless you're saying that the above should work even though it 
 currently doesn't. Even then, I don't know about that. If your 
 type is complex enough to need a shared dtor then the dtor 
 probably needs to do some locking or extra checks. You don't 
 want to impose that cost on a struct instance which isn't 
 shared.


Yes, just what I meant. And perfectly explained why we need 
something better.
Jul 19 2017
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 17 July 2017 at 19:30:53 UTC, Jack Stouffer wrote:

 On Monday, 17 July 2017 at 17:41:58 UTC, Atila Neves wrote:

 On Monday, 17 July 2017 at 14:26:19 UTC, Jack Stouffer wrote:

 TL;DR: Issue 17658 [1] makes using shared very 
 annoying/practically impossible.

 [...]


 I fixed this already, should be in the next release.

 Atila


 Are you sure? Because DMD nightly still errors:

 https://run.dlang.io?compiler=dmd-nightly&source=struct%20A%0A%7B%0A%20%20%20%20this(string%20a)%20%7B%7D%0A%20%20%20%20this(string%20a)%20shared%20%7B%7D%0A%0A%20%20%20%20~this()%20%7B%7D%0A%20%20%20%20~this()%20shared%20%7B%7D%0A%0A%20%20%20%20this(this)%20%7B%7D%0A%20%20%20%20this(this)%20shared%20%7B%7D%0A%7D%0A%0Avoid%20main()%0A%7B%0A%20%20%20%20shared%20f%20%3D%20A(%22%22)%3B%0A%7D


 (maybe we should remove the ban on URL shorteners for our own 
 sites)


Now I've read your post properly: there is only one destructor. 
With the fix I mentioned, just don't defined the `shared` 
version, there's no need to. Postblit is still a problem, however.

Atila
Jul 18 2017
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 18 Jul 2017 18:10:58 +0000
schrieb Atila Neves <atila.neves gmail.com>:

 Now I've read your post properly: there is only one destructor. 
 With the fix I mentioned, just don't defined the `shared` 
 version, there's no need to. Postblit is still a problem, however.
 
 Atila


The issue is wider than just `shared` by the way:
https://issues.dlang.org/show_bug.cgi?id=13628
Some may jump to say that an immutable struct can't be
destructed, but my perspective here is that immutable only
applies to what the compiler can introspect. A file descriptor
or an opaque struct pointer from a C API are just flat values
and escape the compiler. They can be stored in an immutable
struct and still need `close()` called on them.
Layman's head-const :p

-- 
Marco
Jul 18 2017
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 17 July 2017 at 19:30:53 UTC, Jack Stouffer wrote:

 On Monday, 17 July 2017 at 17:41:58 UTC, Atila Neves wrote:

 On Monday, 17 July 2017 at 14:26:19 UTC, Jack Stouffer wrote:

 TL;DR: Issue 17658 [1] makes using shared very 
 annoying/practically impossible.

 [...]


 I fixed this already, should be in the next release.

 Atila


 Are you sure? Because DMD nightly still errors:

 https://run.dlang.io?compiler=dmd-nightly&source=struct%20A%0A%7B%0A%20%20%20%20this(string%20a)%20%7B%7D%0A%20%20%20%20this(string%20a)%20shared%20%7B%7D%0A%0A%20%20%20%20~this()%20%7B%7D%0A%20%20%20%20~this()%20shared%20%7B%7D%0A%0A%20%20%20%20this(this)%20%7B%7D%0A%20%20%20%20this(this)%20shared%20%7B%7D%0A%7D%0A%0Avoid%20main()%0A%7B%0A%20%20%20%20shared%20f%20%3D%20A(%22%22)%3B%0A%7D


 (maybe we should remove the ban on URL shorteners for our own 
 sites)


This works fine in dmd 2.075:

struct A
{
     this(string a) {}
     this(string a) shared {}

     ~this() {}

     this(this T)(this) {} // you can reflect to find out if shared
}

void main()
{
     auto nonShared = A("");
     auto shared_ = shared A("");
     auto nonSharedCopy = nonShared;
     auto sharedCopy = shared_;
}


Atila
Jul 21 2017
parent reply Arek <arychlinski gmail.com> writes:
On Friday, 21 July 2017 at 08:53:44 UTC, Atila Neves wrote:


 This works fine in dmd 2.075:

 struct A
 {
     this(string a) {}
     this(string a) shared {}

     ~this() {}

     this(this T)(this) {} // you can reflect to find out if 
 shared
 }

 void main()
 {
     auto nonShared = A("");
     auto shared_ = shared A("");
     auto nonSharedCopy = nonShared;
     auto sharedCopy = shared_;
 }


 Atila


This look interesting: this(this T)(this) {}
What is it? Postblit?

It compiles but doesn't work for me.

this(this T)(this) {writeln("postblit"); }   // doesn't print 
anything

Arek
Aug 12 2017
parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 12 August 2017 at 19:34:35 UTC, Arek wrote:

 On Friday, 21 July 2017 at 08:53:44 UTC, Atila Neves wrote:


 This works fine in dmd 2.075:

 struct A
 {
     this(string a) {}
     this(string a) shared {}

     ~this() {}

     this(this T)(this) {} // you can reflect to find out if 
 shared
 }

 void main()
 {
     auto nonShared = A("");
     auto shared_ = shared A("");
     auto nonSharedCopy = nonShared;
     auto sharedCopy = shared_;
 }


 Atila


 This look interesting: this(this T)(this) {}
 What is it? Postblit?

 It compiles but doesn't work for me.

 this(this T)(this) {writeln("postblit"); }   // doesn't print 
 anything

 Arek


It's a template postblit constructor - it'd get instantiated 
differently depending on the type of the implicit `this` 
parameter and would be able to fix things up taking into account 
whether or not `this` was shared (or immutable).

Atila
Aug 14 2017
next sibling parent reply Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 16:49:22 UTC, Atila Neves wrote:

 On Saturday, 12 August 2017 at 19:34:35 UTC, Arek wrote:

 On Friday, 21 July 2017 at 08:53:44 UTC, Atila Neves wrote:




... /cut/...

 It's a template postblit constructor - it'd get instantiated 
 differently depending on the type of the implicit `this` 
 parameter and would be able to fix things up taking into 
 account whether or not `this` was shared (or immutable).

 Atila


I've tested this code on dmd 2.075.0 and it doesn't behave like 
postblit.

It's not executed in these statements:
     auto nonSharedCopy = nonShared;
     auto sharedCopy = shared_;

To get it executed, you have to change the statements into
     auto nonSharedCopy = A(nonShared);
     auto sharedCopy = A(shared_);

this(this T)(this) is compiled into

0000000000000000 <shared(ref shared(b.A) 
function(immutable(char)[])) b.A.__ctor>:
    0:	55                   	push   %rbp
    1:	48 8b ec             	mov    %rsp,%rbp
    4:	48 89 f8             	mov    %rdi,%rax
    7:	5d                   	pop    %rbp
    8:	c3                   	retq
    9:	00 00                	add    %al,(%rax)
	...
what is exacly the same as the constructor:

0000000000000000 <ref b.A b.A.__ctor(immutable(char)[])>:
    0:	55                   	push   %rbp
    1:	48 8b ec             	mov    %rsp,%rbp
    4:	48 89 f8             	mov    %rdi,%rax
    7:	5d                   	pop    %rbp
    8:	c3                   	retq
    9:	00 00                	add    %al,(%rax)
	...

The real postblit looks like this:
Disassembly of section .text.void b.A.__postblit():

0000000000000000 <void b.A.__postblit()>:
    0:	55                   	push   %rbp
    1:	48 8b ec             	mov    %rsp,%rbp
    4:	48 83 ec 10          	sub    $0x10,%rsp
    8:	48 89 7d f8          	mov    %rdi,-0x8(%rbp)
    c:	b8 73 00 00 00       	mov    $0x73,%eax
   11:	b9 0a 00 00 00       	mov    $0xa,%ecx
   16:	99                   	cltd
   17:	f7 f9                	idiv   %ecx
   19:	48 83 7d f8 00       	cmpq   $0x0,-0x8(%rbp)
   1e:	75 2e                	jne    4e <void b.A.__postblit()+0x4e>
   20:	49 89 c8             	mov    %rcx,%r8

<void b.A.__postblit()+0x2a>
   2a:	b8 03 00 00 00       	mov    $0x3,%eax
   2f:	48 89 c2             	mov    %rax,%rdx
   32:	48 89 55 f0          	mov    %rdx,-0x10(%rbp)

<void b.A.__postblit()+0x3d>
   3d:	bf 09 00 00 00       	mov    $0x9,%edi
   42:	48 89 d6             	mov    %rdx,%rsi
   45:	48 8b 55 f0          	mov    -0x10(%rbp),%rdx
   49:	e8 00 00 00 00       	callq  4e <void b.A.__postblit()+0x4e>
   4e:	c9                   	leaveq
   4f:	c3                   	retq

So, in my opinion, your example compiles (but doesn't work) 
because it doesn't have the user defined postblit.

It's a pity, because it's looked promising.

Arek
Aug 14 2017
parent Atila Neves <atila.neves gmail.com> writes:
On Monday, 14 August 2017 at 18:59:17 UTC, Arek wrote:

 On Monday, 14 August 2017 at 16:49:22 UTC, Atila Neves wrote:

 [...]




 ... /cut/...

 [...]


 I've tested this code on dmd 2.075.0 and it doesn't behave like 
 postblit.

 [...]


I'd have to double check, but this seems like a bug to me.

Atila
Aug 15 2017
prev sibling parent Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 16:49:22 UTC, Atila Neves wrote:

 On Saturday, 12 August 2017 at 19:34:35 UTC, Arek wrote:

 On Friday, 21 July 2017 at 08:53:44 UTC, Atila Neves wrote:





 It's a template postblit constructor - it'd get instantiated 
 differently depending on the type of the implicit `this` 
 parameter and would be able to fix things up taking into 
 account whether or not `this` was shared (or immutable).

 Atila


Sorry - my mistake.

This fancy template compiles into

Disassembly of section .text.ref b.A b.A.__ctor!(b.A).__ctor(b.A):

0000000000000000 <ref b.A b.A.__ctor!(b.A).__ctor(b.A)>:
    0:	55                   	push   %rbp
    1:	48 8b ec             	mov    %rsp,%rbp
    4:	48 83 ec 30          	sub    $0x30,%rsp
    8:	48 89 7d f8          	mov    %rdi,-0x8(%rbp)

<ref b.A b.A.__ctor!(b.A).__ctor(b.A)+0x13>
   13:	b8 0a 00 00 00       	mov    $0xa,%eax
... cut...

But it's still not the postblit and doesn't works like postblit.

Arek
Aug 14 2017