www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP 1024--Shared Atomics--Community Review Round 1

reply Mike Parker <aldacron gmail.com> writes:
This is the feedback thread for the first round of Community 
Review for DIP 1024, "Shared Atomics":

https://github.com/dlang/DIPs/blob/0b892dd99aba74b9631572ad3a53000f5975b7c2/DIPs/DIP1024.md

All review-related feedback on and discussion of the DIP should 
occur in this thread. The review period will end at 11:59 PM ET 
on October 15, or when I make a post declaring it complete.

At the end of Round 1, if further review is deemed necessary, the 
DIP will be scheduled for another round of Community Review. 
Otherwise, it will be queued for the Final Review and Formal 
Assessment.

Anyone intending to post feedback in this thread is expected to 
be familiar with the reviewer guidelines:

https://github.com/dlang/DIPs/blob/master/docs/guidelines-reviewers.md

**Thanks in advance for keeping all discussion on topic.**
Oct 01
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
This DIP will need to detail how it does (and doesn't) interact with 
methods with the shared attribute on it, along with examples of the 
shared qualifier on a struct/class.

As it stands, if I was to implement it, I could interpret the DIP to 
only affect globals.
Oct 01
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 4:21 AM, rikki cattermole wrote:
 This DIP will need to detail how it does (and doesn't) interact with methods 
 with the shared attribute on it, along with examples of the shared qualifier
on 
 a struct/class.
The DIP does not change how `shared` is part of the type system, and does not change how `shared` affects the types of structs or classes or instances of them.
 As it stands, if I was to implement it, I could interpret the DIP to only
affect 
 globals.
`shared` isn't a storage class, it's a type constructor, so I don't see how you could infer that.
Oct 01
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 02/10/2019 9:31 AM, Walter Bright wrote:
 On 10/1/2019 4:21 AM, rikki cattermole wrote:
 This DIP will need to detail how it does (and doesn't) interact with 
 methods with the shared attribute on it, along with examples of the 
 shared qualifier on a struct/class.
The DIP does not change how `shared` is part of the type system, and does not change how `shared` affects the types of structs or classes or instances of them.
Okay, if we get a statement to that affect in the DIP, maybe with an example then nobody can interpret it wrong :)
 As it stands, if I was to implement it, I could interpret the DIP to 
 only affect globals.
`shared` isn't a storage class, it's a type constructor, so I don't see how you could infer that.
Okay in that case, the spec needs updating. Because right now it is listed as part of `StorageClass` along with `const`.
Oct 01
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 1:54 PM, rikki cattermole wrote:
 On 02/10/2019 9:31 AM, Walter Bright wrote:
 On 10/1/2019 4:21 AM, rikki cattermole wrote:
 This DIP will need to detail how it does (and doesn't) interact with methods 
 with the shared attribute on it, along with examples of the shared qualifier 
 on a struct/class.
The DIP does not change how `shared` is part of the type system, and does not change how `shared` affects the types of structs or classes or instances of them.
Okay, if we get a statement to that affect in the DIP, maybe with an example then nobody can interpret it wrong :)
Ok.
 As it stands, if I was to implement it, I could interpret the DIP to only 
 affect globals.
`shared` isn't a storage class, it's a type constructor, so I don't see how you could infer that.
Okay in that case, the spec needs updating. Because right now it is listed as part of `StorageClass` along with `const`.
While const appears in the grammar as a storage class, mainly for programmer convenience, it is a type constructor.
Oct 03
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:
 [snip]
In terms of other languages that might take a similar approach, the DIP only mentions C's volatile. The Task Parallelism and Synchronization section of the Chapel Language specification [1] might be of interest. [1] https://chapel-lang.org/docs/_downloads/chapelLanguageSpec.pdf
Oct 01
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1024, "Shared Atomics":
Section "Alternatives" should be properly referenced as it makes unsubstantiated claims.(Also the references are not linked to from the main document). The linked implementation (which is already merged(!)) appears to bear no correlation to the DIP, which appears to suffer a severe identity crisis, and as such completely fails the vengeful-ex principle:
 This change will require using core.atomic or equivalent 
 functions to read and write to shared memory objects. It will 
 prevent unintended, inadvertent non-use of atomic access.
 ...
 By prohibiting direct access to shared data, the user will be 
 required to use core.atomic and to consider the correctness of 
 their code.
 [The DIP is] just replacing calls to core.atomic with language 
 support...
 This is not pioneering new ground. The only innovation is 
 support by the language type system rather than library calls.
 x=1; // atomic store, release, happens after everything above
So which one is it? If it is the first, all the references to apparent automatic rewriting should be dropped. If it is the second, I fail to see why this is semantically any different to what C++ does with atomic<int>, which the DIP makes the claim is controversial.
Oct 01
prev sibling next sibling parent reply IGotD- <nise nise.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:
"This change will require using core.atomic or equivalent 
functions to read and >write to shared memory objects. It will 
prevent unintended, inadvertent non-use of >atomic access."
What if the type is complex (like a struct) outside what the ISA can handle? Usually this only applies for integers 8 - 128 bits depending on architecture, anything beyond that is out of reach unless you want to dig into Intel TSX which the shared attribute isn't suitable for. Another thing that is left out are the effects of operator overloading. shared int s; // What would this do? Atomically increment s or just atomically load s, add one // and then atomically store s? s = s + 1; // Would this atomically increase s and return the value after increment? s++;
Oct 01
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 7:14 AM, IGotD- wrote:
 On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:
 "This change will require using core.atomic or equivalent functions to read 
 and >write to shared memory objects. It will prevent unintended, inadvertent 
 non-use of >atomic access."
What if the type is complex (like a struct) outside what the ISA can handle? Usually this only applies for integers 8 - 128 bits depending on architecture, anything beyond that is out of reach unless you want to dig into Intel TSX which the shared attribute isn't suitable for. Another thing that is left out are the effects of operator overloading. shared int s; // What would this do? Atomically increment s or just atomically load s, add one // and then atomically store s? s = s + 1; // Would this atomically increase s and return the value after increment? s++;
Actually, shared operations would no longer be generated from the syntax. They'd all be done with library functions, which could be implemented as intrinsics by the compiler. If it's outside what the ISA could handle, then those library functions would not be intrinsics, and it's up to the implementation of those functions to make it work.
Oct 01
parent reply ag0aep6g <anonymous example.com> writes:
On 01.10.19 22:34, Walter Bright wrote:
 Actually, shared operations would no longer be generated from the 
 syntax. They'd all be done with library functions
To clarify, we're going to get this(?): shared int x; void main() { x = 42; /* error */ } You should change the first sentence of the DIP then. It says: "Reads and writes to data typed as shared are made atomic where the target CPU supports atomic operations". From that sentence I would very much expect the code be accepted and generate an atomic write.
Oct 01
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 3:07 PM, ag0aep6g wrote:
 On 01.10.19 22:34, Walter Bright wrote:
 Actually, shared operations would no longer be generated from the syntax. 
 They'd all be done with library functions
To clarify, we're going to get this(?):     shared int x;     void main()     {         x = 42; /* error */     }
Yes.
 You should change the first sentence of the DIP then. It says: "Reads and
writes 
 to data typed as shared are made atomic where the target CPU supports atomic 
 operations".
 
  From that sentence I would very much expect the code be accepted and generate 
 an atomic write.
I agree it's confusing.
Oct 03
prev sibling next sibling parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1024, "Shared Atomics":

 [...]
There is no detail on how is this going to be implemented. -Alex
Oct 01
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 9:08 AM, 12345swordy wrote:
 There is no detail on how is this going to be implemented.
It'll disallow direct translation of operators to code, instead the user will have to call core.atomic library functions to do it, and the compiler may treat those as intrinsics.
Oct 01
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Tuesday, 1 October 2019 at 20:35:51 UTC, Walter Bright wrote:
 On 10/1/2019 9:08 AM, 12345swordy wrote:
 There is no detail on how is this going to be implemented.
It'll disallow direct translation of operators to code, instead the user will have to call core.atomic library functions to do it, and the compiler may treat those as intrinsics.
Does this mean that the user can rollout their own custom implementation? -Alex
Oct 01
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 1:47 PM, 12345swordy wrote:
 On Tuesday, 1 October 2019 at 20:35:51 UTC, Walter Bright wrote:
 On 10/1/2019 9:08 AM, 12345swordy wrote:
 There is no detail on how is this going to be implemented.
It'll disallow direct translation of operators to code, instead the user will have to call core.atomic library functions to do it, and the compiler may treat those as intrinsics.
Does this mean that the user can rollout their own custom implementation?
Yes, as they can today.
Oct 03
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:

[...]

My feedback: What is the proposal?

"By prohibiting direct access to shared data, the user will be 
required to use core.atomic and to consider the correctness of 
their code."

"This change will require using core.atomic or equivalent 
functions to read and write to shared memory objects. It will 
prevent unintended, inadvertent non-use of atomic access."

"Since it's just replacing calls to core.atomic with language 
support, and atomics are well understood (at least by experts) in 
multiple languages, there should be little risk. This is not 
pioneering new ground. The only innovation is support by the 
language type system rather than library calls."

It's possible that I just don't understand this topic well 
enought, but in my opinion it's more of a sketch of things that 
should be discussed in a DIP rather than a DIP. An example would 
go a long way.
Oct 01
parent reply ag0aep6g <anonymous example.com> writes:
On Tuesday, 1 October 2019 at 16:22:36 UTC, bachmeier wrote:
 My feedback: What is the proposal?
This is the proposal (first paragraph of the DIP / Abstract): "Reads and writes to data typed as shared are made atomic where the target CPU supports atomic operations and become erroneous when they cannot be done atomically." The rest of the DIP sometimes seems to contradict the Abstract (e.g. "prohibiting direct access to shared data", "will require using core.atomic"). It should probably be clarified that using core.atomic will only be necessary when the compiler can't generate the atomic operation itself for whatever reason.
Oct 01
parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 1 October 2019 at 16:49:25 UTC, ag0aep6g wrote:
 [snip]

 This is the proposal (first paragraph of the DIP / Abstract): 
 "Reads and writes to data typed as shared are made atomic where 
 the target CPU supports atomic operations and become erroneous 
 when they cannot be done atomically."

 The rest of the DIP sometimes seems to contradict the Abstract 
 (e.g. "prohibiting direct access to shared data", "will require 
 using core.atomic"). It should probably be clarified that using 
 core.atomic will only be necessary when the compiler can't 
 generate the atomic operation itself for whatever reason.
I assumed bachmeier's point was more an issue with the organization and presentation of the DIP than what the proposal actually is...
Oct 01
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Tue, Oct 1, 2019 at 3:45 AM Mike Parker via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 [...]
At best, this DIP is premature; we need to make shared correctly restrict read/write access before trying to do fancy stuff. That flag is now available, but the implementation has bugs, and it needs to be enabled by default. Assuming that the supporting material I mention above is available and works properly, then I think it might be reasonable to open some discussion of this kind, and on that note... I think this is a really bad idea, and almost the opposite of what you want in virtually every case. I've written a whole lot of lock-free concurrent code, which involves a lot of atomics, and the semantics in this DIP are not anything I have ever seen or wanted (as I've argued here in the past). I think the semantics described are misleading, and would only serve to improperly educate an amateur user how to incorrectly write lock-free algorithms, and increase their confidence that they're doing it right (they're probably not). 1. Most atomic operations are NOT seq-cst. Seq-cst operations are *super rare*. a. I just searched our entire engine codebase; there are 971 calls to atomic functions; there are *four* instances of seq-cst, and I think at least some of them are actually wrong (over-synchronised), possibly all of them, I would need to consider closer. 2. Many atomic operations appear in a series, in which case, it's typical that exactly one operation in the series may be the synchronisation point. 3. In reality, probably at least half of operations are 'relaxed'. 4. xchg and cas operations are fundamental to any atomic code, and there are no syntax or operators for those operations; so, you're undoubtedly using core.atomic aggressively anyway, and I think it's just weird to litter naked arithmetic in between calls to core.atomic functions. I feel that's misleading by making the naked arithmetic look like NOT-atomic operations by contrast to a bunch of atomic library calls. 5. In almost all cases when working with atomics, you tend to do quirky stuff not captured by regular usage of data types, and not conveniently facilitated by this DIP. For instance; this is a common implementation for releasing a ref count: if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed)) { // maybe acquire if necessary // do release logic... } How do I write that code with this DIP? We need to make shared correct before we do any stuff like this, and then when it's correct, we can talk... but I will probably continue to resist this, because the only result I see is that it will train amateurs to write bad atomic algorithms with an improper sense of confidence gained by being sanctioned by the language. This is a bad idea.
Oct 01
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Tue, Oct 1, 2019 at 10:57 PM Manu <turkeyman gmail.com> wrote:
 On Tue, Oct 1, 2019 at 3:45 AM Mike Parker via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 [...]
At best, this DIP is premature; we need to make shared correctly restrict read/write access before trying to do fancy stuff. That flag is now available, but the implementation has bugs, and it needs to be enabled by default. Assuming that the supporting material I mention above is available and works properly, then I think it might be reasonable to open some discussion of this kind, and on that note... I think this is a really bad idea, and almost the opposite of what you want in virtually every case. I've written a whole lot of lock-free concurrent code, which involves a lot of atomics, and the semantics in this DIP are not anything I have ever seen or wanted (as I've argued here in the past). I think the semantics described are misleading, and would only serve to improperly educate an amateur user how to incorrectly write lock-free algorithms, and increase their confidence that they're doing it right (they're probably not). 1. Most atomic operations are NOT seq-cst. Seq-cst operations are *super rare*. a. I just searched our entire engine codebase; there are 971 calls to atomic functions; there are *four* instances of seq-cst, and I think at least some of them are actually wrong (over-synchronised), possibly all of them, I would need to consider closer. 2. Many atomic operations appear in a series, in which case, it's typical that exactly one operation in the series may be the synchronisation point. 3. In reality, probably at least half of operations are 'relaxed'. 4. xchg and cas operations are fundamental to any atomic code, and there are no syntax or operators for those operations; so, you're undoubtedly using core.atomic aggressively anyway, and I think it's just weird to litter naked arithmetic in between calls to core.atomic functions. I feel that's misleading by making the naked arithmetic look like NOT-atomic operations by contrast to a bunch of atomic library calls. 5. In almost all cases when working with atomics, you tend to do quirky stuff not captured by regular usage of data types, and not conveniently facilitated by this DIP. For instance; this is a common implementation for releasing a ref count: if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed)) { // maybe acquire if necessary // do release logic... } How do I write that code with this DIP? We need to make shared correct before we do any stuff like this, and then when it's correct, we can talk... but I will probably continue to resist this, because the only result I see is that it will train amateurs to write bad atomic algorithms with an improper sense of confidence gained by being sanctioned by the language. This is a bad idea.
Sorry, that should have read: if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed) == 1) { // maybe acquire if necessary // do release logic... }
Oct 01
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On Tue, Oct 1, 2019 at 11:01 PM Manu <turkeyman gmail.com> wrote:
 On Tue, Oct 1, 2019 at 10:57 PM Manu <turkeyman gmail.com> wrote:
 On Tue, Oct 1, 2019 at 3:45 AM Mike Parker via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 [...]
At best, this DIP is premature; we need to make shared correctly restrict read/write access before trying to do fancy stuff. That flag is now available, but the implementation has bugs, and it needs to be enabled by default. Assuming that the supporting material I mention above is available and works properly, then I think it might be reasonable to open some discussion of this kind, and on that note... I think this is a really bad idea, and almost the opposite of what you want in virtually every case. I've written a whole lot of lock-free concurrent code, which involves a lot of atomics, and the semantics in this DIP are not anything I have ever seen or wanted (as I've argued here in the past). I think the semantics described are misleading, and would only serve to improperly educate an amateur user how to incorrectly write lock-free algorithms, and increase their confidence that they're doing it right (they're probably not). 1. Most atomic operations are NOT seq-cst. Seq-cst operations are *super rare*. a. I just searched our entire engine codebase; there are 971 calls to atomic functions; there are *four* instances of seq-cst, and I think at least some of them are actually wrong (over-synchronised), possibly all of them, I would need to consider closer. 2. Many atomic operations appear in a series, in which case, it's typical that exactly one operation in the series may be the synchronisation point. 3. In reality, probably at least half of operations are 'relaxed'. 4. xchg and cas operations are fundamental to any atomic code, and there are no syntax or operators for those operations; so, you're undoubtedly using core.atomic aggressively anyway, and I think it's just weird to litter naked arithmetic in between calls to core.atomic functions. I feel that's misleading by making the naked arithmetic look like NOT-atomic operations by contrast to a bunch of atomic library calls. 5. In almost all cases when working with atomics, you tend to do quirky stuff not captured by regular usage of data types, and not conveniently facilitated by this DIP. For instance; this is a common implementation for releasing a ref count: if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed)) { // maybe acquire if necessary // do release logic... } How do I write that code with this DIP? We need to make shared correct before we do any stuff like this, and then when it's correct, we can talk... but I will probably continue to resist this, because the only result I see is that it will train amateurs to write bad atomic algorithms with an improper sense of confidence gained by being sanctioned by the language. This is a bad idea.
Sorry, that should have read: if (atomicLoad(rc, Relaxed) == 1 || atomicSub(rc, 1, Relaxed) == 1) { // maybe acquire if necessary // do release logic... }
Wait up... it's possible that I've misunderstood this DIP... There are 2 paragraphs that seem to contradict each other. First there is A: """ By prohibiting direct access to shared data, the user will be required to use core.atomic and to consider the correctness of their code. This change will require using core.atomic or equivalent functions to read and write to shared memory objects. It will prevent unintended, inadvertent non-use of atomic access. """ I absolutely agree with this text. Then there's B: """ Atomic reads perform an acquire operation, writes perform a release operation, and read-modify-write performs an acquire, then a modification, and then a release. The result is sequentially consistent ordering, and each thread observes the same order of operations (total order). The code generated would be equivalent to that from the core.atomic library for the supported operations. """ And then goes on with examples like this where `x` is clearly accessed without a call to core.atomic: ``` __gshared int g=0; shared int x=0; void thread1(){ g=1; // non-atomic store x=1; // atomic store, release, happens after everything above } void thread2(){ if(x==1){ // atomic load, acquire, happens before everything below assert(g==1); // non-atomic load } } ``` I'm confused, which is it, A or B? I presume B, and my prior commentary applies.
Oct 01
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/1/2019 11:30 PM, Manu wrote:
 I'm confused, which is it, A or B? I presume B, and my prior commentary
applies.
A
Oct 02
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 10/1/2019 11:30 PM, Manu wrote:
 I'm confused, which is it, A or B? I presume B, and my prior commentary
applies.
A
In that case, I don't understand the only code shown in the DIP; you show freely reading from and assigning to a shared int... that should be an error, and there should be atomicLoad/astomicStore in those locations?
Oct 02
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 2 October 2019 at 17:21:59 UTC, Manu wrote:
 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On 10/1/2019 11:30 PM, Manu wrote:
 I'm confused, which is it, A or B? I presume B, and my prior 
 commentary applies.
A
In that case, I don't understand the only code shown in the DIP; you show freely reading from and assigning to a shared int... that should be an error, and there should be atomicLoad/astomicStore in those locations?
My understanding is that the compiler rewrites that code to use core.atomic.
Oct 02
next sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Wednesday, 2 October 2019 at 17:51:09 UTC, Atila Neves wrote:
 On Wednesday, 2 October 2019 at 17:21:59 UTC, Manu wrote:
 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On 10/1/2019 11:30 PM, Manu wrote:
 I'm confused, which is it, A or B? I presume B, and my 
 prior commentary applies.
A
In that case, I don't understand the only code shown in the DIP; you show freely reading from and assigning to a shared int... that should be an error, and there should be atomicLoad/astomicStore in those locations?
My understanding is that the compiler rewrites that code to use core.atomic.
Why can't it import core.atomic automatically then? -Alex
Oct 02
prev sibling parent Manu <turkeyman gmail.com> writes:
On Wed, Oct 2, 2019 at 10:55 AM Atila Neves via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Wednesday, 2 October 2019 at 17:21:59 UTC, Manu wrote:
 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/1/2019 11:30 PM, Manu wrote:
 I'm confused, which is it, A or B? I presume B, and my prior
 commentary applies.
A
In that case, I don't understand the only code shown in the DIP; you show freely reading from and assigning to a shared int... that should be an error, and there should be atomicLoad/astomicStore in those locations?
My understanding is that the compiler rewrites that code to use core.atomic.
I asked Walter a couple posts up whether or not that's what he meant (A/B), and he said A: """ By prohibiting direct access to shared data, the user will be required to use core.atomic and to consider the correctness of their code. This change will require using core.atomic or equivalent functions to read and write to shared memory objects. It will prevent unintended, inadvertent non-use of atomic access. """ ...which is distinctly not that.
Oct 02
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/2/2019 10:21 AM, Manu wrote:
 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/1/2019 11:30 PM, Manu wrote:
 I'm confused, which is it, A or B? I presume B, and my prior commentary
applies.
A
In that case, I don't understand the only code shown in the DIP; you show freely reading from and assigning to a shared int... that should be an error, and there should be atomicLoad/astomicStore in those locations?
Yes. The trouble stems from the DIP being originally written to apply shared semantics to operators. This was changed to be function only, and some vestiges of the original remain. Hence the confusion.
Oct 03
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On Wed, Oct 2, 2019 at 10:21 AM Manu <turkeyman gmail.com> wrote:
 On Wed, Oct 2, 2019 at 2:10 AM Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 10/1/2019 11:30 PM, Manu wrote:
 I'm confused, which is it, A or B? I presume B, and my prior commentary
applies.
A
In that case, I don't understand the only code shown in the DIP; you show freely reading from and assigning to a shared int... that should be an error, and there should be atomicLoad/astomicStore in those locations?
Also the text: "Atomic reads perform an acquire operation, writes perform a release operation, and read-modify-write performs an acquire, then a modification, and then a release." What does that mean? I don't know what that's talking about. Please correct the DIP so we can understand it.
Oct 02
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/2/2019 10:25 AM, Manu wrote:
 Also the text: "Atomic reads perform an acquire operation, writes
 perform a release
 operation, and read-modify-write performs an acquire, then a
 modification, and then a release."
 
 What does that mean? I don't know what that's talking about.
 Please correct the DIP so we can understand it.
That follows the usage pattern in the C++ memory model documentation. I agree it is hard to follow, but it is normal jargon for those familiar with the subtleties of multi-threaded memory models. I have written a spec PR that starts to formally define the memory model for D, but it's been sitting there for a long time and nobody wants to touch it. I don't think a DIP is the right place for it. I see no reason to deviate from the C++ memory model semantics.
Oct 03
parent Manu <turkeyman gmail.com> writes:
On Thu., 3 Oct. 2019, 8:45 pm Walter Bright via Digitalmars-d, <
digitalmars-d puremagic.com> wrote:

 On 10/2/2019 10:25 AM, Manu wrote:
 Also the text: "Atomic reads perform an acquire operation, writes
 perform a release
 operation, and read-modify-write performs an acquire, then a
 modification, and then a release."

 What does that mean? I don't know what that's talking about.
 Please correct the DIP so we can understand it.
That follows the usage pattern in the C++ memory model documentation. I agree it is hard to follow, but it is normal jargon for those familiar with the subtleties of multi-threaded memory models.
I understand the language used, I have no idea how it applies to this DIP, specifically because I'm still not sure what it does, since various paragraphs and the example contradict each other. I look forward to a revision that's easy to follow.
Oct 04
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On Tue, Oct 1, 2019 at 11:30 PM Manu <turkeyman gmail.com> wrote:
 [...]
And then I notice everyone else has pointed out the confusion too. Sorry for the noise! Please clarity the confusion, then I can revoke my criticisms, or not...
Oct 01
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
Access to shared memory should be disallowed only for safe code, 
but not for system code. For safe code it's justified, because 
safety is unavoidable there, for system code there's a concern of 
ease of use of shared data, which is affected by littering and is 
not addressed in this DIP. Stripping shared qualifier off for 
assignment isn't necessarily correct if, say, the data type 
switches between atomic and nonatomic reference counting based on 
its type.
Oct 02
next sibling parent reply Kagamin <spam here.lot> writes:
Also cast() doesn't strip qualifiers

shared int[] a;
pragma(msg,typeof(cast()a).stringof);

prints "shared(int)[]"
Oct 02
parent Meta <jared771 gmail.com> writes:
On Wednesday, 2 October 2019 at 09:00:54 UTC, Kagamin wrote:
 Also cast() doesn't strip qualifiers

 shared int[] a;
 pragma(msg,typeof(cast()a).stringof);

 prints "shared(int)[]"
That's because `shared int[] a` is really `shared(int[] a)`, so cast() only strips the top level shared off.
Oct 02
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:
 Access to shared memory should be disallowed only for safe 
 code, but not for system code.
That definition doesn't play nice with safety inference i.e. templates. It should be that shared memory access is disabled in all contexts, which must be worked around with casts, which makes the function system, which must then be encapsulated with a trusted interface in order to use in safe code.
Oct 02
next sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 2 October 2019 at 10:42:48 UTC, Nicholas Wilson 
wrote:
 On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:
 Access to shared memory should be disallowed only for safe 
 code, but not for system code.
That definition doesn't play nice with safety inference i.e. templates.
I don't think so. How?
Oct 03
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/2/2019 3:42 AM, Nicholas Wilson wrote:
 On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:
 Access to shared memory should be disallowed only for safe code, but not for 
 system code.
That definition doesn't play nice with safety inference i.e. templates. It should be that shared memory access is disabled in all contexts, which must be worked around with casts, which makes the function system, which must then be encapsulated with a trusted interface in order to use in safe code.
Sounds right.
Oct 11
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via Digitalmars-d 
wrote:
 On 10/2/2019 3:42 AM, Nicholas Wilson wrote:
 On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:
 Access to shared memory should be disallowed only for safe code, but
 not for system code.
That definition doesn't play nice with safety inference i.e. templates. It should be that shared memory access is disabled in all contexts, which must be worked around with casts, which makes the function system, which must then be encapsulated with a trusted interface in order to use in safe code.
Sounds right.
This is pretty much what several of us were arguing for during the discussions about shared at dconf this year. - Jonathan M Davis
Oct 11
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 06:31:18 UTC, Jonathan M Davis 
wrote:
 On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via 
 Digitalmars-d wrote:
 On 10/2/2019 3:42 AM, Nicholas Wilson wrote:
 It should be that shared memory access is disabled in all 
 contexts,
 which must be worked around with casts,
 which makes the function  system,
 which must then be encapsulated with a  trusted interface in 
 order to
 use in  safe code.
Sounds right.
This is pretty much what several of us were arguing for during the discussions about shared at dconf this year.
How are you going to prove that safe code does not retain nonshared-references after the lock has been released? How does his work with array elements/slices.
Oct 12
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 3:15:26 AM MDT Ola Fosheim Grstad via 
Digitalmars-d wrote:
 On Saturday, 12 October 2019 at 06:31:18 UTC, Jonathan M Davis

 wrote:
 On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via

 Digitalmars-d wrote:
 On 10/2/2019 3:42 AM, Nicholas Wilson wrote:
 It should be that shared memory access is disabled in all
 contexts,
 which must be worked around with casts,
 which makes the function  system,
 which must then be encapsulated with a  trusted interface in
 order to
 use in  safe code.
Sounds right.
This is pretty much what several of us were arguing for during the discussions about shared at dconf this year.
How are you going to prove that safe code does not retain nonshared-references after the lock has been released? How does his work with array elements/slices.
It's the same as with any trusted code. It's up to the programmer to ensure that what the code is doing is actually safe, and if the code isn't able to provide an API that is safe, then it shouldn't be trusted. Ultimately, if the programmer casts away shared while a mutex is locked, it's up to the programmer to ensure that no thread-local references to the shared data escape (though scope can help with that) so that when the lock is released, the only references left are shared. - Jonathan M Davis
Oct 12
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis 
wrote:
 It's the same as with any  trusted code. It's up to the 
 programmer to ensure that what the code is doing is actually 
  safe, and if the code isn't able to provide an API that is 
  safe, then it shouldn't be  trusted.
But didn't Walter in another thread some time ago say that he would add Rust-like linear typing for memory management? If this is the goal, shouldn't you also use linear typing for allowing more flexible safe/ trusted APIs?
 Ultimately, if the programmer casts away shared while a mutex 
 is locked, it's up to the programmer to ensure that no 
 thread-local references to the shared data escape (though scope 
 can help with that) so that when the lock is released, the only 
 references left are shared.
How does that work with arrays. E.g. processing that has coroutine/ranges style implementations. How can you ensure that it is impossible for the lock to be released and also making sure that all safe single-threaded array processing code can be used? You often want to use "shared" for large sections of arrays. Another thing is that it seems to me that things could go wrong if you have slices and dynamic arrays. E.g.: a. you take a slice of the last element of an unshared dynamic array and hand it to one thread. b. you reduce the length of the dynamic array c. you hand out an unshared version of the dynamic array to another thread which then increase the length of the dynamic array in safe code Now you have two safe unshared references to the last element in two different threads? I don't have much experience with extending dynamic arrays, because I don't like the D semantics of it, but isn't this roughly how things could go?
Oct 12
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 10:22:38 UTC, Ola Fosheim Grøstad 
wrote:
 c. you hand out an unshared version of the dynamic array to 
 another thread which then increase the length of the dynamic 
 array in  safe code
I'm sorry. I mixed up Go and D dynamic array semantics... :-/ I thought they were exactly the same, but apparently D always reallocates while Go tries to avoid reallocation. I dislike the semantics of Go too... that being said. Still, it might be an issue for library types, even though it appears to work out for builtins (?).
Oct 12
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 4:22:38 AM MDT Ola Fosheim Grstad via 
Digitalmars-d wrote:
 On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis

 wrote:
 It's the same as with any  trusted code. It's up to the
 programmer to ensure that what the code is doing is actually
  safe, and if the code isn't able to provide an API that is
  safe, then it shouldn't be  trusted.
But didn't Walter in another thread some time ago say that he would add Rust-like linear typing for memory management? If this is the goal, shouldn't you also use linear typing for allowing more flexible safe/ trusted APIs?
I don't know anything about what you mean by linear types (I really don't know much about Rust). So, I can't say how that would come into play, but in general, with trusted, the programmer must verify that what the code is doing is safe and must provide an API that is safe. If the programmer can't guarantee that what the code is doing is safe, then they shouldn't be marking it is trusted. Any operations that the compiler can guarantee as safe are already considered safe. By definition, once system is involved, it's because the compiler can't guarantee that the code is safe, and trusted is the programmer's tool for telling the compiler that a particular piece of code is actually safe even though the compiler couldn't verify that. In the case of shared, in general, it's not thread-safe to read or write to such a variable without either using atomics or some other form of thread synchronization that is currently beyond the ability of the compiler to make guarantees about and will likely always be beyond the ability of the compiler to make guarantees about except maybe in fairly restricted circumstances. The type system separates thread-local from shared, but it doesn't have any concept of ownership and simply doesn't have the kind of information required to figure out whether something is thread-safe. The closest that anyone has proposed is TDPL's synchronized classes, and even those could only safely remove the outer layer of shared - and even that required locking down synchronized classes pretty thoroughly. That's why it doesn't make sense to allow reading from or writing to shared variables, and it has to be up to the programmer to deal with the synchronization primitives and temporarily casting to thread-local to operate on the shared data while it's protected by those synchronization primitives. The casts automatically make such code system as it should be, and then the programmer is going to have to guarantee the thread-safety just like they'd do in a language like Java or C++. The difference is that D's type system helps the programmer, whereas languages like Java or C++ leave it entirely up to the programmer to get it right. The type system segregates the data that's shared, and the code that does all of the tricky threading stuff is system, requiring the programmer to verify that piece of code and mark it as trusted to then be used by the safe portions of the program. Yes, the casts make the code more verbose than it would be in C++, but other than that, you're basically doing exactly the same thing that you'd be doing in C++. By no means does that mean that any of the threading stuff is easy, but as long as the type system cannot make guarantees about thread-safety, segregating the code that deals with the threading and requiring that the programmer verify its correctness in order to have it work with safe code is the best that we can do. Regardless, even if we're able to later come up with improvements that allow shared to be implicitly removed under some set of circumstances (and thus have such code be safe), shared in general still needs to be locked down in a manner similar to what this DIP is proposing. Any improvements that we later come up with would then just be laid on top of that. - Jonathan M Davis
Oct 12
parent reply Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Saturday, 12 October 2019 at 20:52:45 UTC, Jonathan M Davis 
wrote:
 In the case of shared, in general, it's not thread-safe to read 
 or write to such a variable without either using atomics or 
 some other form of thread synchronization that is currently 
 beyond the ability of the compiler to make guarantees about and 
 will likely always be beyond the ability of the compiler to 
 make guarantees about except maybe in fairly restricted 
 circumstances.
A shared variable may not need any synchronization at all, depending on the algorithm it is used in. There is a class of optimized algorithms that act like gathering operations. You mostly find them on GPUs because they map quite naturally to that hardware architecture, but they can also be implemented on CPUs with multiple threads. The core idea is in every case that you generate a set of output values in parallel in such a way that each value in the output is generated by at most one of the running threads. So there is no need to synchronize memory writes when the underlying hardware architecture provides sufficient cache coherency guarantees. All the threads share the same input, which obviously must not be modified. A compiler cannot possibly be smart enough to prove that synchronization is not required for these kinds of algorithms. And any form of enforced synchronization (explicit or implicit) would significantly affect performance. So how would you express this implicit synchronization by mathematical properties to the compiler? Would the whole implementation have to be marked as safe?
Oct 15
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 15 October 2019 at 10:25:54 UTC, Gregor Mückl wrote:
 generated by at most one of the running threads. So there is no 
 need to synchronize memory writes when the underlying hardware 
 architecture provides sufficient cache coherency guarantees.
So what you basically is saying is that a low level language should be careful about assuming a particular hardware model and leave more to intrinsics and libraries. I think that is reasonable, because hardware does change and concurrent programming strategies change. So D has to figure out whether it is a low level language for many architectures or a specific x86 centric one limited to common contemporary programming patterns. My perception is that D does not need to lock down «shared» but can rather improve on these: 1. A set of basic lowlevel building blocks + intrinsics (which D has mostly). 2. Metaprogramming features to build libraries that support contemporary patterns (which D has mostly). 3. Language features that support clean static analysis tooling (perhaps lacking). But given the history of D, it will probably go with what is demanded based on contemporary x86 patterns.
Oct 15
parent reply Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Tuesday, 15 October 2019 at 11:28:24 UTC, Ola Fosheim Grøstad 
wrote:
 On Tuesday, 15 October 2019 at 10:25:54 UTC, Gregor Mückl wrote:
 generated by at most one of the running threads. So there is 
 no need to synchronize memory writes when the underlying 
 hardware architecture provides sufficient cache coherency 
 guarantees.
So what you basically is saying is that a low level language should be careful about assuming a particular hardware model and leave more to intrinsics and libraries. I think that is reasonable, because hardware does change and concurrent programming strategies change.
I'm not sure if this is quite what I'm saying. I guess I'm fine with the compiler telling me that a piece of code tries to access shared data without any annotation that this is what is desired. In my opinion, it then needs to be up to the developer to deal with this. It is *not* OK for the compiler to go and secretly insert synchronization mechanisms (memory barriers, atomic operations, etc...) behind the developer's back. If such an automatism must exist, it also must be invoked by the developer explicitly ("Computer, generate automatic synchronization!"). And there must be a way to tell the compiler that the developer is a responsible adult who knows what he/she is doing and that the code is OK for reasons unknown to the compiler.
Oct 16
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, October 16, 2019 12:43:30 PM MDT Gregor Mckl via Digitalmars-
d wrote:
 On Tuesday, 15 October 2019 at 11:28:24 UTC, Ola Fosheim Grstad

 wrote:
 On Tuesday, 15 October 2019 at 10:25:54 UTC, Gregor Mckl wrote:
 generated by at most one of the running threads. So there is
 no need to synchronize memory writes when the underlying
 hardware architecture provides sufficient cache coherency
 guarantees.
So what you basically is saying is that a low level language should be careful about assuming a particular hardware model and leave more to intrinsics and libraries. I think that is reasonable, because hardware does change and concurrent programming strategies change.
I'm not sure if this is quite what I'm saying. I guess I'm fine with the compiler telling me that a piece of code tries to access shared data without any annotation that this is what is desired. In my opinion, it then needs to be up to the developer to deal with this. It is *not* OK for the compiler to go and secretly insert synchronization mechanisms (memory barriers, atomic operations, etc...) behind the developer's back. If such an automatism must exist, it also must be invoked by the developer explicitly ("Computer, generate automatic synchronization!"). And there must be a way to tell the compiler that the developer is a responsible adult who knows what he/she is doing and that the code is OK for reasons unknown to the compiler.
The DIP is not clear on the matter and needs to be updated, but Walter has made it clear in his comments that there is no plan to even insert core.atomic calls for you. Reading and writing to shared data is going to be illegal, requiring that the programmer either explicitly use core.atomic or that they cast away shared (rendering that part of the code system, thereby segregating the code that the programmer has to verify for thread safety). It is conceivable that someone will come up with a feature that will allow the compiler to implicitly remove shared under some set of circumstances, because it's able to see that only one thread can access that data at that point (e.g. TDPL's synchronized classes would be able to do this to the outer layer of shared for the member variables of that class), but that would just be an improvement on top of what Walter is proposing, and honestly, I doubt that we'll ever see much along those lines, because making such compiler guarantees is very difficult with D's type system (e.g. TDPL's synchronized classes lock the type down quite a lot and yet are still only able to remove a single layer of shared, making them borderline useless, and AFAIK, no one has yet proposed anything that could do better). The type system would likely need a concept of thread ownership to safely reason about much with regards to shared, and even if the compiler _were_ able to implicitly remove shared under some set of circumstances, there's no way that it's going to understand all of the various threading mechanisms that get used in code. So, at best, you'd be able to use a particular feature to have a piece of code implicitly remove shared, because the compiler is able to do it for that particular idiom. There's no question that the programmer is going to have to cast away shared in many cases in order to actually operate on shared data. And once the programmer casts away shared, that code is then system, requiring the programmer to vet it and certify that it's actually safe by using trusted. So, all of the code that operates on shared data should then be in three camps: 1. It uses core.atomic. 2. It casts away shared and is thus system. 3. It involves shared member functions which then do eith internally. Andrei has talked in the past about inserting magic to make shared "just work" without casting, but I think that it's pretty clear at this point that that isn't going to work. The best that we could do would be to make some operations which are guaranteed to be atomic allowed on shared primitive types, and that would be a questionable choice for a variety of reasons. Either way, it's not currently the plan to do so. - Jonathan M Davis
Oct 16
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis 
wrote:
 It's the same as with any  trusted code. It's up to the 
 programmer to ensure that what the code is doing is actually 
  safe, and if the code isn't able to provide an API that is 
  safe, then it shouldn't be  trusted.
So, I am trying to figure out what that API would have to look like and the only solution I can think of right away is that you have are doing a callback back into safe code from the lock-holding trusted code? So, if you have to obtain locks from multiple sources, it either won't work out, or you have to use some serious kludges, like a deep call chain: while(tryagain) { trusted_code_obtains_lock_1--> safe_code_callback1 calls-> trusted_code_obtains_lock2 safe_code_callback2 processes and sets tryagain=false } Of course, that could lead to starvation. So to avoid all of that multi-threaded libraries have to have 2 APIs, one simple transactional wrapper-API for safe code and another API for trusted code that can be used to write a new wrapper that allows locking of multiple datastructures. I hope people try to sketch out what such libraries will look like before the semantics are decided on. Maybe an experimental branch would be appropriate.
Oct 12
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 5:26:42 AM MDT Ola Fosheim Grstad via 
Digitalmars-d wrote:
 I hope people try to sketch out what such libraries will look
 like before the semantics are decided on. Maybe an experimental
 branch would be appropriate.
The fact that we have thread-local in the language requires that the type system then have the concept of shared. So, there should be no question about that. As part of that, converting between thread-local and shared has to require casts (otherwise, the compiler can't guarantee that the thread-local stuff is actually thread-local). Those casts then have to be system, because they potentially violate the compiler's guarantees about thread-local and shared data. The question that then comes is what kinds of operations should be allowed on shared, and there are basically two approaches that can be taken: 1. Not allow read-write operations that aren't guaranteed to be atomic. 2. Ignore thread-safety and just let people do whatever they want to with shared other than implicitly convert between shared and thread-local (since that would violate the compiler guarantees about which data is shared and which is thread-local). #1 allows us to segregate all of the code that has to deal with threading stuff in a manner that requires the programmer to get involved, because operating on shared in a way that isn't guarantee to be atomic then becomes system. So, it becomes far easier to locate the pieces of code that need to be verified as thread-safe by the programmer, and you get actual compiler guarantees that any safe code operating on shared variables doesn't have any threading problems. Only the trusted code has to be examined. Unlike #1, #2 would mean that you couldn't know that code operating on shared was thread-safe. You wouldn't have to do as much casting, because basic operations would work on shared data, but you'd then have to examine all code involving shared to determine whether what it was doing was thread-safe. And because many types are designed to be thread-local rather than shared, you'd still have to cast to thread-local in many cases, and that code would still have to be system, because the programmer is potentially violating the compiler guarantees about thread-local when casting to or from shared. So, while #2 would reduce the amount of casting required, it would do a far worse job segregating code that needed to be examined when dealing with threading issues, and it doesn't really remove the casting and the like that folks like to complain about with regards to shared. We were locked into that once we had shared as part of the type system. Languages like C++ can only get away with not having it because they ignore the issue. That means that in such languages, you can just operate on shared stuff as if it were thread-local, but the programmer then also has to worry about threading issues throughout the entire program, because the type system isn't helping them at all. Good programming practices help make that problem more manageable, but by having shared as part of the type system in D, the language and the compiler make the problem much more manageable by segregating the threading code. Right now, we basically have #2. What this DIP is trying to do is move us to #1. How libraries should work is exactly the same in either case. It's just that with #1, the places where you operate on shared data in a manner which isn't guaranteed to be atomic, the compiler prevents you from doing it unless you use core.atomic or have system code with casts. Even if we have #2 and thus no such compiler errors, the code should still have been doing what #1 would have required, since if it doesn't, then it isn't thread-safe. It may be that if we can come up with ways for the compiler to know that it's thread-safe to implicitly remove shared in a piece of code, libraries won't have to have as much trusted code, but the low level semantics would still need to be the same, and any code that wasn't using whatever idiom or feature allowed the compiler to implicitly remove shared in a piece of code would still need to do what it would need to do right now, which is what it would need to do if this DIP were implemented. All this DIP really does is force code that the compiler can't guarantee is atomic to use either core.atomic or be system/ trusted. The idioms and design patterns involved with thread-safe code are the same either way. - Jonathan M Davis
Oct 12
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis 
wrote:
 The fact that we have thread-local in the language requires 
 that the type system then have the concept of shared. So, there 
 should be no question about that.
Well, my understanding from what you wrote is that the shared marker is 100% syntactical and has no semantic implications. In that case it can be done as a library meta programming construct (type wrapping). You just need a way to restrict application of the constructs based on code structure ( safe). So one could establish a syntactical solution that is perfectly suitable for meta programming? But you could probably do better by introducing an effectsbased typesystem. Anyway, based on what you said shared does not have to be in the language and could be replaced by more expressive metaprogramming mechanisms. I think.
Oct 12
parent reply Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 3:35 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis
 wrote:
 The fact that we have thread-local in the language requires
 that the type system then have the concept of shared. So, there
 should be no question about that.
Well, my understanding from what you wrote is that the shared marker is 100% syntactical and has no semantic implications.
The semantic implication is that there is no read or write access... just the same as how const says there is no write access.
 In that case it can be done as a library meta programming construct
 (type wrapping).
We can't define type constructors in a library sadly. You could apply the same argument to const. And aside from that, it belongs in the language due to the simple fact that it defines D's thread-local by default. If you think about it, it's not really the definition of thread-local by default that leads to shared... it's actually that shared existing in the language defines thread-local by default. If there was no shared, then there couldn't possibly be thread-local by default, as D is specified. shared is fundamental to D, it's just that it's semantics have been poorly understood because it's historically had relatively little use.
 You just need a way to restrict application of
 the constructs  based on code structure ( safe). So one could
 establish a syntactical solution that is perfectly suitable for
 meta programming?

 But you could probably do better by introducing an effectsbased
 typesystem.

 Anyway, based on what you said shared does not have to be in the
 language and could be replaced by more expressive metaprogramming
 mechanisms. I think.
It's like I said above; shared MUST be in the language, because it defines one of the fundamental language semantics; thread-local by default.
Oct 12
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 00:13:12 UTC, Manu wrote:
 On Sat, Oct 12, 2019 at 3:35 PM Ola Fosheim Grøstad via The 
 semantic implication is that there is no read or write 
 access... just the same as how const says there is no write 
 access.
Well, I understand what you mean, but if you can achieve the same effect by hiding, then it is syntactical in nature. That would be difficult with const in D ( might work in other languages ).
 It's like I said above; shared MUST be in the language, because 
 it defines one of the fundamental language semantics; 
 thread-local by default.
Does the shared marker have any effect on generated code? If not, then it is just a way to bypass (break) the type system on a syntactical level.
Oct 12
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 8:45 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 13 October 2019 at 00:13:12 UTC, Manu wrote:
 On Sat, Oct 12, 2019 at 3:35 PM Ola Fosheim Grøstad via The
 semantic implication is that there is no read or write
 access... just the same as how const says there is no write
 access.
Well, I understand what you mean, but if you can achieve the same effect by hiding, then it is syntactical in nature.
We can't though.
 That would be difficult with const in D ( might work in other
 languages ).

 It's like I said above; shared MUST be in the language, because
 it defines one of the fundamental language semantics;
 thread-local by default.
Does the shared marker have any effect on generated code?
Not at this time. And I'm not sure what your point is... does const have any effect in generated code? (it does not)
 If not,
 then it is just a way to bypass (break) the type system on a
 syntactical level.
I don't understand this.. how is it bypassing/breaking the type system?
Oct 12
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.no> writes:
On Sunday, 13 October 2019 at 03:49:17 UTC, Manu wrote:
 On Sat, Oct 12, 2019 at 8:45 PM Ola Fosheim Grøstad via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 Well, I understand what you mean, but if you can achieve the 
 same effect by hiding, then it is syntactical in nature.
We can't though.
Assume that symbols prefixed by trusted_ are only visible in trusted code. Then you can define trusted_unwrap that returns a reference to the unwrapped type. Then you have what you need for a simple solution to a library version of shared.
 Not at this time. And I'm not sure what your point is... does 
 const have any effect in generated code? (it does not)
Const could also be done as a library wrapper, but that would require full fledeged meta programming and strong deductive functionality. So not realistic for D.
 If not,
 then it is just a way to bypass (break) the type system on a
 syntactical level.
I don't understand this.. how is it bypassing/breaking the type system?
If you need a cast to make use of shared...
Oct 12
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 9:41:00 PM MDT Ola Fosheim Grstad via 
Digitalmars-d wrote:
 It's like I said above; shared MUST be in the language, because
 it defines one of the fundamental language semantics;
 thread-local by default.
Does the shared marker have any effect on generated code? If not, then it is just a way to bypass (break) the type system on a syntactical level.
The compiler is free to assume that anything that isn't shared is thread-local and optimize code accordingly. Also, it _does_ matter on some level with how memory is laid out, because thread-local storage gets used (IIRC, there were problems on older versions of OS X, because we had to fake the TLS, because OS X didn't provide proper TLS). Module-level and static variables which are shared only get initialized once, whereas module-level and static variables which are thread-local get initialized once per thread, because each thread gets their own copy. So, while the vast majority of D objects are thread-local, the distinction between what's thread-local and what's shared is very much built into how D functions. That's part of why it's so risky to cast away shared, and why it needs to be system. If the programmer screws up that code and allows any shared data to be treated as thread-local when any other thread could be operating on that data, then not only does that incur all of the typical issues that come with screwing up synchronizing data access, but the compiler could make things worse, because it generated code based on the assumption that the data being operated on was thread-local. Fortunately, as long as shared disallows non-atomic, read/write operations the code that needs to be examined for threading issues is restricted to system/ trusted code that involves shared, so the amount of code that the programmer has to examine and verify to ensure that shared data doesn't end up being treated as thread-local when multiple threads can operate on it is limited and should be fairly easy to find. - Jonathan M Davis
Oct 12
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 04:09:26 UTC, Jonathan M Davis 
wrote:
 The compiler is free to assume that anything that isn't shared 
 is thread-local and optimize code accordingly.
That has no real effect though, if you throw away shared before using it. That said, if you had a formalization of the threads and what states different threads are in then you could get some performance benefits by eliding shared overhead when it can be proven that no other threads are accessing a shared variable. But that is not on the table...
 Also, it _does_ matter on some level with how memory is laid 
 out, because thread-local storage gets used (IIRC, there were 
 problems on older versions of OS X, because we had to fake the 
 TLS, because OS X didn't provide proper TLS).
Actually, how the memory is laid out is not a big deal. Vendors can do this with a library solution too. What is a big deal with TLS is that you need different codegen for obtaining a reference, but that also mean that you cannot throw away that type information before you have a obtained an effective address (reference)? Same thing with shared. On an architecture that requires a different instruction sequences for shared memory and local memory, you would need to retain that information. Not for allocation, which can be dealt with at the library level, but for accessing it when executing generic functions. If anything that would be an argument for never being allowed to throw away the shared marker, and also an argument for being able to add more markers (for different types of memory or I/O).
 Module-level and static variables which are shared only get 
 initialized once, whereas module-level and static variables 
 which are thread-local get initialized once per thread, because 
 each thread gets their own copy.
I'd rather say that a TLS variable "value" is conceptually "value[thread_id]" or "value__12345".
 Fortunately, as long as shared disallows non-atomic, read/write 
 operations the code that needs to be examined for threading 
 issues is restricted to  system/ trusted code that involves 
 shared, so the amount of code that the programmer has to 
 examine and verify to ensure that shared data doesn't end up 
 being treated as thread-local when multiple threads can operate 
 on it is limited and should be fairly easy to find.
And you can achieve all that with a library type, because D doesn't actually deal with shared memory, it provides some instructions that results in barriers in the IR. If you wrap all those barrier-generating instructions in library calls then they can just avoid emitting them if the type they are applied to is not wrapped in a library provided Shared wrapper. I don't see why the compiler has to know anything about shared for this to work the same way as described. Anyway, I think D really needs to think twice about adding more and more narrow features and instead improve on the meta programming capabilities. Clearly many D features could be moved to the library (slicing, dicts etc). If there is a limited pool of people working on the compiler it would make a lot of sense to put more power into the hands of library authors. The meta-programming capabilities of C++ is evolving too...
Oct 13
parent reply Manu <turkeyman gmail.com> writes:
On Sun, Oct 13, 2019 at 12:55 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 13 October 2019 at 04:09:26 UTC, Jonathan M Davis
 wrote:
 The compiler is free to assume that anything that isn't shared
 is thread-local and optimize code accordingly.
That has no real effect though, if you throw away shared before using it. That said, if you had a formalization of the threads and what states different threads are in then you could get some performance benefits by eliding shared overhead when it can be proven that no other threads are accessing a shared variable.
What does that even mean? You're just making things up. Thread's don't have 'states', that's literally the point of threads! No code-gen can ever assume any stateful details of any kind about another thread. Maybe there's a 10 year research project there, but that idea is so ridiculous and unlikely to work that nobody would ever try. I think what you're talking about is something more like a framework; where different threads are running known code which is configured to cooperate in particular ways, and implements safe data transmission between threads. This is what will exist, there will be elaborate libraries of this kind. I don't expect many end-users would ever encounter a `shared` object, it would likely be behind the curtains in all cases, unless they're writing very low-level systems.
 But that is not on the table...
Definitely not. Not at the language level at least.
Oct 13
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 19:08:00 UTC, Manu wrote:
 On Sun, Oct 13, 2019 at 12:55 AM Ola Fosheim Grøstad via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 That said, if you had a formalization of the threads and what 
 states different threads are in then you could get some 
 performance benefits by eliding shared overhead when it can be 
 proven that no other threads are accessing a shared variable.
What does that even mean? You're just making things up.
(Drop the ad hominems... Especially when your in an area that is not your field.)
 Thread's don't have 'states', that's literally the point of 
 threads!
All program execution is in a state at any given point in time. So yes, you transition between states. That is the premise for building an advanced type system. That's what makes it possible to implement Rusts borrow checker. Anything that moves in a discrete fashion will move between states, that is a basic computer science conceptualization of computing. It is the foundation that theories about computing is built on.
 Maybe there's a 10 year research project there, but that idea 
 is so
 ridiculous and unlikely to work that nobody would ever try.
Oh. People do it. Sure, it is hard in the most general case, but you do have languages with high level concurrency constructs that provide type systems that makes it possible to write provably correct concurrent programs. It doesn't make much sense to claim that nobody would ever try to build something that actually exists... Does it?
 I think what you're talking about is something more like a 
 framework;
No. Languages/compilers that check concurrency states at compile time.
 But that is not on the table...
Definitely not. Not at the language level at least.
That's right. So those benefits are not avilable... which was my point. :-)
Oct 14
parent reply Manu <turkeyman gmail.com> writes:
On Mon, Oct 14, 2019 at 5:50 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 13 October 2019 at 19:08:00 UTC, Manu wrote:
 On Sun, Oct 13, 2019 at 12:55 AM Ola Fosheim Grøstad via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 That said, if you had a formalization of the threads and what
 states different threads are in then you could get some
 performance benefits by eliding shared overhead when it can be
 proven that no other threads are accessing a shared variable.
What does that even mean? You're just making things up.
(Drop the ad hominems... Especially when your in an area that is not your field.)
Threading and synchronisation is exactly my field. And it's not at hominem, I really do think you're just making stuff up (although I'm sure informed by ideas in competing ecosystems)... I'm sure you mean well and have some great idea, but you haven't presented anything that I can think on. The closest thing you've said to a real implementation proposal so far is strong-ownership based message passing like rust, but that's not really in contest with this DIP, and I can't imagine how `shared` would have any influence on that toolbox.
 Thread's don't have 'states', that's literally the point of
 threads!
All program execution is in a state at any given point in time. So yes, you transition between states. That is the premise for building an advanced type system. That's what makes it possible to implement Rusts borrow checker.
That's all thread-local concepts. There's nothing reasonable you can determine between threads without synchronising the threads. D is flirting with escape analysis, we have `scope` and `return` and things like that, and those are the things that will eventually allow us to do in-language transitions to/from shared without unsafe casts.
 Anything that moves in a discrete fashion will move between
 states, that is a basic computer science conceptualization of
 computing. It is the foundation that theories about computing is
 built on.
The only reasonable CS model of threading I've ever seen requires you to start talking about strong ownership and transfering ownership between threads. While I agree that strong ownership and message passing is robust and worth having in any async ecosystem, it's also tremendously inefficient for certain workloads, and if that were all we had baked into the language, it excludes other (useful) possibilities.
 Maybe there's a 10 year research project there, but that idea
 is so
 ridiculous and unlikely to work that nobody would ever try.
Oh. People do it. Sure, it is hard in the most general case, but you do have languages with high level concurrency constructs that provide type systems that makes it possible to write provably correct concurrent programs.
That's not so much language at that point, it becomes blessed library, or a 'framework' even. Point me to some examples which are more than in-language message passing? I don't think we're excluding those sorts of solutions from D.
 It doesn't make much sense to claim that nobody would ever try to
 build something that actually exists... Does it?
The way you initially described the premise was something along the lines of "the language knows the *state* of interacting threads and does magic thread-safe stuff", which I'm skeptical exists anywhere without a very elaborate framework to manage any such orchestration. I think now that you might be talking about ownership and message passing... but it's still not clear. What are you actually talking about? Do you have references? Incidentally, my code that I linked to back up this thread is effectively an implementation of a rust-like shared object where it maintains a 'borrow' ref count, and guards access safely... it's easy to write those tools in D.
 I think what you're talking about is something more like a
 framework;
No. Languages/compilers that check concurrency states at compile time.
Show me. That is exactly the opposite of what threads are for :/ I can't imagine it, and if a _language_ did that, it almost certainly comes at a high cost, and it's almost certainly not what I want it to do... but I'm willing to be surprised.
 But that is not on the table...
Definitely not. Not at the language level at least.
That's right. So those benefits are not avilable... which was my point. :-)
I can't predict what you're talking about, and you haven't shown anything substantial. I suspect if you show what you mean, I may find the term 'benefits' to be problematic ;)
Oct 14
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Monday, 14 October 2019 at 18:04:28 UTC, Manu wrote:
 And it's not at hominem, I really do think you're just making 
 stuff up
 (although I'm sure informed by ideas in competing ecosystems)...
If you go after the person and not the statement then that is "ad hominem", isn't it? Claiming that people are making stuff up does not compel me to expand on the topic, but I'll try to give you an idea of the landscape as I see it, anyway. I'm not sure what you mean by «competing eco systems», but I am talking about theoretical and applied research. And it is not a field I am highly interested in, but I have a passing interest in it. It makes a lot of sense to use such approaches if you are designing communication systems or critical control systems. It might be wasted on games. Anyway, there are various ways of reasoning formally about concurrency, and it has been done for a long time... And there are many approaches. Ranging from dedicated tools for programming in various niches where it is beneficial, to formal methods and tools for analyzing distributed protocols. It is a big field, and I think the latter theme is easier to get into, and you'll find plenty on this if you search on keywords like: formal methods, formally verified, distributed computing, distributed protocols...
 The closest thing you've said to a real implementation proposal 
 so far
 is strong-ownership based message passing like rust, but that's 
 not
 really in contest with this DIP, and I can't imagine how 
 `shared`
 would have any influence on that toolbox.
My argument was that D would not have it, so it would not be possible to make significant optimization based on modelling concurrency. Therefore "shared" as a library type ought to be sufficient. That was my line of argument. I do not think D should model concurrency formally. I belive such languages should be created from a clean slate. Though, some appears to do stuff with annotated C, but it does not sound very compelling to just slap it onto an existing language.
 That's all thread-local concepts. There's nothing reasonable 
 you can
 determine between threads without synchronising the threads.
You might need synchronization, but you can potentially have less of it if you have a proof that thread 1 is always in state B when thread 2 is in state A, or that thread 1 is never in state B if thread 2 is not in state D, etc. It is not uncommon to modell processes as statemachines and analyze them formally. When you have that model, you can implement it at a lower level if you want, and make a correspondence proof that tie each chunk of low level code to the high level model. Then there are languages where everything is handled for you in a "regular" programming language, but they will have a particular focus, with a particular set of trade-offs.
 The only reasonable CS model of threading I've ever seen 
 requires you
 to start talking about strong ownership and transfering 
 ownership
 between threads.
There are many ways to model and reason about concurrency. One approach was actually inspired by the modelling of chemical reactions using PetriNets. One "impractical", but interesting, low level instruction-by-instruction approach uses Hoare logic. While possible, you might be limited to very few instructions. Anyway, with this approach you might in theory be able to avoid explicit synchronization in some cases as you can then prove whether or not all possible interleaving of instructions during execution comply with the specification! Still, the combinatorial explosion makes this very limiting and hard to reason about. Anyway, Hoare logic is a very low level approach that has been researched in the 70s or 80s, so as you can see there is a history of various approaches. Maybe applicable to a tight low level scope like a simple high performance hardware driver. Many problems are only computationally prohibitively expensive if you want the optimal solution always, so researchers sometimes find ways to solve problems that appears to be impossible due to computational complexity, but with the caveat that you sometimes get the result "I don't know". In optimization, "I don't know" is good enough, because then you just fall back on the conservative implementation. A solver that is insufficient for unguided verification might be good enough for optimization. If "impossible" problems can be dealt with "sometimes", then you will get optimizations "sometimes". Anyway, applied solutions seem to focus more on high level representations and high level mechanisms, but I would not be surprised if you find applied solutions working on low level representations using low level mechanisms.
 While I agree that strong ownership and message passing is 
 robust and
 worth having in any async ecosystem, it's also tremendously
 inefficient for certain workloads, and if that were all we had 
 baked
 into the language, it excludes other (useful) possibilities.
Ok, but I didn't argue for it being built into the language... I am arguing that if it is built into the language then there should be a semantic reason for doing so. And static analysis of locking patterns might be one such reason. But in general I think D would be better off by providing generic mechanisms rather than adding more and more limited features. So maybe not add borrowing, but perhaps add some sort of limited linear/affine typing mechanism than can be used for implementing libraries that provides it to some extent. So people can write libraries that statically check that files that are opened also are closed. Or that you follow specific patterns when setting up hardware (register protocols or something like that). If meta-programming and library-authoring is D2's main selling point then that ought to be front and center.
 Oh. People do it. Sure, it is hard in the most general case, 
 but you do have languages with high level concurrency 
 constructs that provide type systems that makes it possible to 
 write provably correct concurrent programs.
That's not so much language at that point, it becomes blessed library, or a 'framework' even.
No, you cannot run a solver from a library. Well, I guess you could, but that is not how it is usually done.
 The way you initially described the premise was something along 
 the lines of "the language knows the *state* of interacting 
 threads and does magic thread-safe stuff", which I'm skeptical 
 exists anywhere without a very elaborate framework to manage 
 any such orchestration.
Not sure what you mean by elaborate framework. You need a language for expressing it. Then you need a solver for solving the constraints (analyzing/checking the model).
 I think now that you might be talking about ownership and 
 message passing...
It could be, but I am talking about formal verification of concurrency models. Or languages that are designed in such a way that they translates into such models that can be verified. What you are talking about might be the other direction where you have languages that are designed in such a way that some of the issues provably cannot arise (e.g. Pony, maybe Go). But that is not what I am referring to. I am referring to inference using a solver.
 Incidentally, my code that I linked to back up this thread is 
 effectively an implementation of a rust-like shared object 
 where it maintains a 'borrow' ref count, and guards access 
 safely... it's easy to write those tools in D.
Alright, but if you had access to more advanced typing machinery in D then you could go further down that road yourself. Maybe not to the extent that it could keep up with Rust, but closer.
 if a _language_ did that, it almost certainly
 comes at a high cost, and it's almost certainly not what I want 
 it to
 do... but I'm willing to be surprised.
I don't think you want to do it. You'd have to change how you approach it, and it is not really needed for more mundane tasks.
 That's right. So those benefits are not avilable... which was 
 my point. :-)
I can't predict what you're talking about, and you haven't shown anything substantial. I suspect if you show what you mean, I may find the term 'benefits' to be problematic ;)
What do you mean by substantial? The benefits is that you prove that the code follows the model, which in turn can be fed into the optimizer and assumed to always hold. It is not realistic that D will ever go there. It is more realistic that D would expand the type system or meta-programming mechanisms in ways that are useful to library authors and that they could create concurrent frameworks.
Oct 14
parent reply Manu <turkeyman gmail.com> writes:
On Mon, Oct 14, 2019 at 2:50 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 14 October 2019 at 18:04:28 UTC, Manu wrote:
 And it's not at hominem, I really do think you're just making
 stuff up
 (although I'm sure informed by ideas in competing ecosystems)...
If you go after the person and not the statement then that is "ad hominem", isn't it? Claiming that people are making stuff up does not compel me to expand on the topic, but I'll try to give you an idea of the landscape as I see it, anyway. I'm not sure what you mean by «competing eco systems», but I am talking about theoretical and applied research. And it is not a field I am highly interested in, but I have a passing interest in it. It makes a lot of sense to use such approaches if you are designing communication systems or critical control systems. It might be wasted on games. Anyway, there are various ways of reasoning formally about concurrency, and it has been done for a long time... And there are many approaches. Ranging from dedicated tools for programming in various niches where it is beneficial, to formal methods and tools for analyzing distributed protocols. It is a big field, and I think the latter theme is easier to get into, and you'll find plenty on this if you search on keywords like: formal methods, formally verified, distributed computing, distributed protocols...
I'm aware of a lot of work in this area, but I haven't seem how it influences a language at the lowest level (beyond ownership based solutions, where transfer of unique ownership is used to model thread-safe data transitions); it's usually in the realm of what I would call 'framework', although there are many examples of such being blessed by the language. I don't think D has a great history with blessing an implementation as sacred and installing it into the syntax, and D tends to be expressive enough to make library solutions for this sort of thing quite satisfying.
 The closest thing you've said to a real implementation proposal
 so far
 is strong-ownership based message passing like rust, but that's
 not
 really in contest with this DIP, and I can't imagine how
 `shared`
 would have any influence on that toolbox.
My argument was that D would not have it, so it would not be possible to make significant optimization based on modelling concurrency. Therefore "shared" as a library type ought to be sufficient. That was my line of argument. I do not think D should model concurrency formally.
Then how do you define thread-local by default, and how do you mark/identify data that violates that axiom? D doesn't "model concurrency", that's not what `shared` does, it simply establishes thread-locality.
 I belive such languages should be created from a clean slate.
 Though, some appears to do stuff with annotated C, but it does
 not sound very compelling to just slap it onto an existing
 language.
Are you talking about D here, or just a hypothetical application of your POV?
 That's all thread-local concepts. There's nothing reasonable
 you can
 determine between threads without synchronising the threads.
You might need synchronization, but you can potentially have less of it if you have a proof that thread 1 is always in state B when thread 2 is in state A, or that thread 1 is never in state B if thread 2 is not in state D, etc.
This is well into 'framework' territory; a thread is associated with static knowledge of a state machine, known threads are collected into a registry, and then state transitions can assert validity against the other threads in the pool. This sounds super complex, hard to express, and not worth of language syntax. It also has nothing to do with `shared`.
 It is not uncommon to modell processes as statemachines and
 analyze them formally.
Sure, but we don't do that, or anything that's even remotely in that ballpark. It's not a functional language, and we don't have *code* or data access introspection like that.
 When you have that model, you can
 implement it at a lower level if you want, and make a
 correspondence proof that tie each chunk of low level code to the
 high level model. Then there are languages where everything is
 handled for you in a "regular" programming language, but they
 will have a particular focus, with a particular set of trade-offs.
That sort of language goes all in on one very restrictive programming model. It's not anything like D, and I don't think it's a consideration when talking about `shared`. If you want to resist `shared`'s existence, then I don't think ideas like this will move us forward.
 The only reasonable CS model of threading I've ever seen
 requires you
 to start talking about strong ownership and transfering
 ownership
 between threads.
There are many ways to model and reason about concurrency. One approach was actually inspired by the modelling of chemical reactions using PetriNets. One "impractical", but interesting, low level instruction-by-instruction approach uses Hoare logic. While possible, you might be limited to very few instructions. Anyway, with this approach you might in theory be able to avoid explicit synchronization in some cases as you can then prove whether or not all possible interleaving of instructions during execution comply with the specification! Still, the combinatorial explosion makes this very limiting and hard to reason about. Anyway, Hoare logic is a very low level approach that has been researched in the 70s or 80s, so as you can see there is a history of various approaches. Maybe applicable to a tight low level scope like a simple high performance hardware driver.
I've read about ideas like this before, but like, it's strictly fantasy. I'm interested in what's real, where we are, what we have, and where we need to be in practical terms.
 While I agree that strong ownership and message passing is
 robust and
 worth having in any async ecosystem, it's also tremendously
 inefficient for certain workloads, and if that were all we had
 baked
 into the language, it excludes other (useful) possibilities.
Ok, but I didn't argue for it being built into the language...
There's gotta be something at the bottom of the stack, or you can't write anything above.
 I am arguing that if it is built into the language then there
 should be a semantic reason for doing so. And static analysis of
 locking patterns might be one such reason.
There is; D specifies thread-local by default, and that absolutely requires a marker (and associated semantics) to attribute the data that violates that principle. This has nothing to do with locking, that's a much higher-level problem and completely outside the language. Locking is this: * I have this shared data, I want to access it... * I must create a context where I have an exclusive window on the data. * One such implementation involves a mutex of some form, where I assert that I am the unique lease-holder, and then in effect the data becomes thread-local for the duration of that lease (ie, cast shared away, because the data is thread-local for that duration). In D, we can implement semantics like this by scope-guarded lock tools of various kinds. It is also possible to achieve thread-locality by deferring to a scheduling infrastructure with knowledge of access-control. Such frameworks will cast to thread-local when issuing access controlled data to the user. We should make use of D's recent escape analysis tricks to guarantee that thread-local leases remain properly scoped, and mutually exclusive with other leases.
 But in general I think D would be better off by providing generic
 mechanisms rather than adding more and more limited features.
What does this mean to you? We can write very rich libraries with `shared` as a safety mechanism. Nothing more is needed from the language, except perhaps some future implicit cast rules.
 So maybe not add borrowing, but perhaps add some sort of limited
 linear/affine typing mechanism than can be used for implementing
 libraries that provides it to some extent. So people can write
 libraries that statically check that files that are opened also
 are closed. Or that you follow specific patterns when setting up
 hardware (register protocols or something like that).
What does this have to do with `shared`, and why is the DIP at odds with this?
 If meta-programming and library-authoring is D2's main selling
 point then that ought to be front and center.
I agree. I wrote that borrow-inspired guarded access type which I lank above.
 Oh. People do it. Sure, it is hard in the most general case,
 but you do have languages with high level concurrency
 constructs that provide type systems that makes it possible to
 write provably correct concurrent programs.
That's not so much language at that point, it becomes blessed library, or a 'framework' even.
No, you cannot run a solver from a library. Well, I guess you could, but that is not how it is usually done.
If this relates to the ideas you mentioned at the top; it's dependent on threads with rich statically-associated data. D could do this sort of thing in library; we have user-defined attributes, and heavy-weight thread definition objects could carry that around and be consumed by solvers. In some ways it's a bit like Andrei's Big-O library, which carries metadata around with definitions and used for static analysis.
 The way you initially described the premise was something along
 the lines of "the language knows the *state* of interacting
 threads and does magic thread-safe stuff", which I'm skeptical
 exists anywhere without a very elaborate framework to manage
 any such orchestration.
Not sure what you mean by elaborate framework. You need a language for expressing it. Then you need a solver for solving the constraints (analyzing/checking the model).
This is exactly what I would call a framework library. An elaborated and opinionated implementation of a particular model.
 I think now that you might be talking about ownership and
 message passing...
It could be, but I am talking about formal verification of concurrency models. Or languages that are designed in such a way that they translates into such models that can be verified. What you are talking about might be the other direction where you have languages that are designed in such a way that some of the issues provably cannot arise (e.g. Pony, maybe Go). But that is not what I am referring to. I am referring to inference using a solver.
Your ideas rely on intensive attribution or extremely sophisticated introspection. We can easily do such attribution in D, but not introspection of code, and where to be practical, like you say, the low-level set of operations would need to be radically simplified into a small and reasonable set. You're not talking about D at that point.
 Incidentally, my code that I linked to back up this thread is
 effectively an implementation of a rust-like shared object
 where it maintains a 'borrow' ref count, and guards access
 safely... it's easy to write those tools in D.
Alright, but if you had access to more advanced typing machinery in D then you could go further down that road yourself. Maybe not to the extent that it could keep up with Rust, but closer.
I'm not sure what rust offers to my little lib up there that I can't do; I mean, rust basically does exactly that, in basically the same way. The only advantage of rust is that it has some language to track lifetime knowledge through function calls which can be used for escape analysis of the scoped lease object.
 if a _language_ did that, it almost certainly
 comes at a high cost, and it's almost certainly not what I want
 it to
 do... but I'm willing to be surprised.
I don't think you want to do it. You'd have to change how you approach it, and it is not really needed for more mundane tasks.
Is it needed for *any* task? I can't think of a more complex interactive environment in software than simulation/video games. Nowhere else are so many completely different distinct solutions all interacting in an asynchronous (and realtime) environment. Maybe you want a proof-able solution that guarantees progress, and willing to sacrifice efficiency for it... but in that world, why do you need threads at all? If such an environment exists, I think it's very niche.
 That's right. So those benefits are not avilable... which was
 my point. :-)
I can't predict what you're talking about, and you haven't shown anything substantial. I suspect if you show what you mean, I may find the term 'benefits' to be problematic ;)
What do you mean by substantial?
By 'substantial' I mean 'at all'. You didn't reference examples or specifics, or describe your vision until now.
 The benefits is that you prove that the code follows the model,
 which in turn can be fed into the optimizer and assumed to always
 hold.

 It is not realistic that D will ever go there.

 It is more realistic that D would expand the type system or
 meta-programming mechanisms in ways that are useful to library
 authors and that they could create concurrent frameworks.
Right. That's exactly what we're doing. I'm glad you agree :)
Oct 14
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 15 October 2019 at 04:09:52 UTC, Manu wrote:
 On Mon, Oct 14, 2019 at 2:50 PM Ola Fosheim Grøstad via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 I'm aware of a lot of work in this area, but I haven't seem how 
 it
 influences a language at the lowest level (beyond ownership 
 based
 solutions, where transfer of unique ownership is used to model
 thread-safe data transitions);
Well, that's because the research topic is «verification». So that is where the culture is, but most programmers have no interest in verification, because then they would have to think about what they are doing twice... They probably should!! But that takes time and we have a global programming culture of «fudging» things together :-). Doesn't mean the ideas cannot be leveraged for more mundane purposes with more limited scope than verification though. Rust is the outlier, and as such is quite impressive in the sense that they have managed to established a culture where programmers accept having to do extra (unnecessary) work to make their programs run in exchange for higher confidence of correctness. But as you can see in any discussion about Rust vs C++, a large number of programmers reject that and hold it against Rust. So establishing such a culture is not an everyday phenomenon. You seemed to suggest that I use Rust, but I personally have no need for Rust, as I currently don't have issues that Rust can solve. I might pick it up for mundane reasons like tight webasm code generation. What I find interesting about Rust is the culture that builds up around the language semantics. Which certainly is a shift from programming cultures that preceded it as far as I can tell.
 it's usually in the realm of  what I
 would call 'framework', although there are many examples of 
 such being blessed by the language.
Ok, so basically anything that is unfamiliar in the D context is a «framework», including type systems. Fair enough. («framework» is usually used for denoting a set of library types and possibly runtime with tight coupling that aims to provide a foundation for writing applications in a limited category.)
 Then how do you define thread-local by default, and how do you
 mark/identify data that violates that axiom?
The only reason you need to mark TLS is because it is referenced using a different instruction sequence. So to repeat myself: Other than that, D does not provide anything beyond atomics/barriers. Does it? Everything else is purely syntactical. Slam a -tag around your code and you get to reinterpret and thwart the type system. That's a syntactical approach in my mind. Not a sound type system approach. Therefore it offers nothing over a library type.
 static knowledge of a state machine, known threads are 
 collected into
 a registry, and then state transitions can assert validity 
 against the
 other threads in the pool.
You could do it dynamically, but also statically. Simple example, if you can establish that all Reader threads are spawned after the Writer thread has ended then you can ellide all the locking in the Reader threads. (without the programmer knowing about it).
 that ballpark. It's not a functional language, and we don't 
 have *code* or data access introspection like that.
Doesn't have to be FPL, people do interesting things on the IR level, but I am not saying D should, the manpower and knowhow isn't here.
 That sort of language goes all in on one very restrictive 
 programming model.
It might, but doesn't have to, programmers can restrict themselves to satisfy the verifier. It appears to work for Rust.
 If you want to resist `shared`'s existence, then I don't think 
 ideas like this will move us forward.
I am not resisting anything. I am trying to figure out why you cannot do «shared» equally well as a library type and what is missing in the language that prevents that from being possible. Because if that is not possible then I sense that there must be some glaring flaws in the D meta-programming capabilities that ought to be fixed. For me this is an exercise in figuring out what works and does not work in D's meta programming model.
 I've read about ideas like this before, but like, it's strictly 
 fantasy. I'm interested in what's real, where we are, what we 
 have, and where we need to be in practical terms.
Ok, so I presented two ends of the spectrum that has a long history, one from 1939/1960s (PetriNets) and one from 1980 (concurrent Hoare logic). In between those ends you have many different approaches. You wanted to discuss what is possible/not possible, and these two end points does say something about what people have been researching and therefore what might/might not be possible. There are probably future applications for concurrent Hoare logic in software synthesis (let the computer find instruction sequences that satisfy a specification.). At least that seems plausible, so I wouldn't call it fantasy or useless.
 There is; D specifies thread-local by default, and that 
 absolutely requires a marker (and associated semantics) to 
 attribute the data that violates that principle.
That's just a claim. No, you only need lower level constructs like barriers. You should be able to implement the marker in a library, if not then someone ought to take a hard look at the meta-programming capabilities.
 What does this mean to you? We can write very rich libraries 
 with `shared` as a safety mechanism. Nothing more is needed 
 from the language, except perhaps some future implicit cast 
 rules.
That is also not true. If you are allowed to turn "shared" into proper "local" then you've got a type system that is so weak that there is no need for it to be in the language. It is not sound. To get a proper and sound type system that requires language support you need something stronger than «D shared». Pony's take on this is to use more types to mark the legal transitions. Too late for D, so you might as well just figure out WHY the meta programming capabilities prevent you from implementing «shared» as a library type. I have not been able to figure that out, because nobody has showed me why it cannot be done as a library type. GOLDEN RULE OF THUMB FOR PROGRAMMING LANGUAGE DESIGN: Before you implement at feature: 1. Implement it as a library solution. 2. Use it and describe why it does not feel right/works right. 3. Figure out what weaknesses the language has that prevents it from being done properly as a library solution. 4. Figure out how that can be improved then go back to 1. 5. Give up, evaluate the situation and start planning a language feature.
 It is more realistic that D would expand the type system or 
 meta-programming mechanisms in ways that are useful to library 
 authors and that they could create concurrent frameworks.
Right. That's exactly what we're doing. I'm glad you agree :)
I agree with myself, but that is NOT exactly what you are doing... :) See the «golden rule of thumb» above. Before adding a feature, provide an in-depth analysis for why a library solution is insufficient. I don't see how any skilled language designer could disagree with that statement.
Oct 15
prev sibling parent reply IGotD- <nise nise.com> writes:
On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis 
wrote:
 Right now, we basically have #2. What this DIP is trying to do 
 is move us to #1. How libraries should work is exactly the same 
 in either case. It's just that with #1, the places where you 
 operate on shared data in a manner which isn't guaranteed to be 
 atomic, the compiler prevents you from doing it unless you  use 
 core.atomic or have  system code with casts. Even if we h
 and thus no such compiler errors, the code should still have 
 been doing what #1 would have required, since if it doesn't, 
 then it isn't thread-safe.
With this DIP, shared integers/small types will be automatically atomic. For complex/large types, will you still be able to use them as before between threads and you have protect the type yourself at least for a transitional period? "Atomic" here as I get it also mean atomically updating complex types. This usually means that you need to guard the operations with some kind of mutex. The compiler can of course detect this and issue a warning/error to the user which doesn't seem to be the scope of this DIP. Correct me if I'm wrong but we have the following scenarios. 1. shared integer/simple type (size dependent?) -> automatically HW atomic operations 2. shared complex type -> write to any member must be protected with a mutex. 3. shared complex type -> read to any member must be protected with a mutex or read/write mutex allowing multiple reads. The compiler is used the detect these scenarios so that the user doesn't forget protecting the shared types. As I get it this DIP is just a baby step towards this bigger scope for the shared keyword.
Oct 12
next sibling parent Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 4:40 PM IGotD- via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis
 wrote:
 Right now, we basically have #2. What this DIP is trying to do
 is move us to #1. How libraries should work is exactly the same
 in either case. It's just that with #1, the places where you
 operate on shared data in a manner which isn't guaranteed to be
 atomic, the compiler prevents you from doing it unless you  use
 core.atomic or have  system code with casts. Even if we h
 and thus no such compiler errors, the code should still have
 been doing what #1 would have required, since if it doesn't,
 then it isn't thread-safe.
With this DIP, shared integers/small types will be automatically atomic.
That's not what this DIP says, as had been clarified by Walter a number of times now.
 For complex/large types, will you still be able to use
 them as before between threads and you have protect the type
 yourself at least for a transitional period?

 "Atomic" here as I get it also mean atomically updating complex
 types. This usually means that you need to guard the operations
 with some kind of mutex. The compiler can of course detect this
 and issue a warning/error to the user which doesn't seem to be
 the scope of this DIP.

 Correct me if I'm wrong but we have the following scenarios.
 1. shared integer/simple type (size dependent?) -> automatically
 HW atomic operations
This is the misunderstanding through most of this thread.
 2. shared complex type -> write to any member must be protected
 with a mutex.
 3. shared complex type -> read to any member must be protected
 with a mutex or read/write mutex allowing multiple reads.

 The compiler is used the detect these scenarios so that the user
 doesn't forget protecting the shared types.

 As I get it this DIP is just a baby step towards this bigger
 scope for the shared keyword.
Useful libraries will follow.
Oct 12
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 12, 2019 5:37:50 PM MDT IGotD- via Digitalmars-d wrote:
 On Saturday, 12 October 2019 at 21:28:36 UTC, Jonathan M Davis

 wrote:
 Right now, we basically have #2. What this DIP is trying to do
 is move us to #1. How libraries should work is exactly the same
 in either case. It's just that with #1, the places where you
 operate on shared data in a manner which isn't guaranteed to be
 atomic, the compiler prevents you from doing it unless you  use
 core.atomic or have  system code with casts. Even if we h
 and thus no such compiler errors, the code should still have
 been doing what #1 would have required, since if it doesn't,
 then it isn't thread-safe.
With this DIP, shared integers/small types will be automatically atomic. For complex/large types, will you still be able to use them as before between threads and you have protect the type yourself at least for a transitional period? "Atomic" here as I get it also mean atomically updating complex types. This usually means that you need to guard the operations with some kind of mutex. The compiler can of course detect this and issue a warning/error to the user which doesn't seem to be the scope of this DIP. Correct me if I'm wrong but we have the following scenarios. 1. shared integer/simple type (size dependent?) -> automatically HW atomic operations 2. shared complex type -> write to any member must be protected with a mutex. 3. shared complex type -> read to any member must be protected with a mutex or read/write mutex allowing multiple reads. The compiler is used the detect these scenarios so that the user doesn't forget protecting the shared types. As I get it this DIP is just a baby step towards this bigger scope for the shared keyword.
When we're talking about atomic with regards to shared, we're talking about what core.atomic does. They're operations that are atomic with regards to CPU instructions. The most that that can work with is primitive types like integers or pointers. More complex types require stuff like mutexes to protect them in order to freely mutate them. Walter needs to make the DIP clearer, but in the discussions in this thread, he's made it clear that the intention is that read/write operations will become illegal for all shared data, forcing code to either use core.atomic to do atomic operations on the data or to use synchronization mechanisms such as mutexes to allow for thread-safe reading and writing of data, with the code needing to cast away shared in order to operate on the data (meaning that the code will then be system or trusted). In principle, the compiler could allow reading and writing shared variables by inserting the core.atomic stuff for you, but that's not the current plan. Either way, it's very difficult to have the compiler understand synchronization primitives well enough and have enough guarantees about what the code is doing to be able to do something like implicitly remove shared for you. So, it's highly unlikely that we'll ever get much in the language that would be able to implicitly remove shared for even simple pieces of code let alone complex types. AFAIK, only construct along those lines that's been proposed thus far that could work is TDPL's synchronized classes, and they can only implicitly remove a single layer of shared - and that can only do that much because of how restrictive they are. Having the compiler magically handle thread synchronization primitives for you is likely a pipe dream. Rather, what's likely going to tend to happen is that complex objects that are supposed to be used as shared will handle the appropriate atomics or mutexes internally, providing an safe API for the user. However, the internals will still have to do the dirty stuff and be appropriately vetted for thread-safety. That's already what you typically get in a language like C++. It's just that the type system doesn't have shared, so you have to manage everything yourself and don't have a way to segregate shared data and the functions operating on it other than by convention, whereas D's shared enforces it via the type system. Ultimately, shared is about segregating the code that operates on shared data - both so that most code can just be thread-local without worrying about it and so that you can easily determine which pieces of the program have to be examined for threading issues - just like system/ trusted/ safe isolates the portions of the program where you potentially have to worry about memory safety. shared isn't about magically handling threading stuff for you. If we can figure out how to add mechanisms on top of shared which make things easier, then great, but shared has to be properly locked down first, and given that D is a systems language (thus allowing all kinds of crazy stuff), and its type system really has no concept of ownership, I don't think that it's very likely that we're going to be able to add much to the language that's going to allow shared to be implicitly removed or allow you to otherwise operate on shared data without worrying about dealing with the synchronization primitives yourself. - Jonathan M Davis
Oct 12
prev sibling parent reply Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 4:30 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 12 October 2019 at 09:45:41 UTC, Jonathan M Davis
 wrote:
 It's the same as with any  trusted code. It's up to the
 programmer to ensure that what the code is doing is actually
  safe, and if the code isn't able to provide an API that is
  safe, then it shouldn't be  trusted.
So, I am trying to figure out what that API would have to look like and the only solution I can think of right away is that you have are doing a callback back into safe code from the lock-holding trusted code? So, if you have to obtain locks from multiple sources, it either won't work out, or you have to use some serious kludges, like a deep call chain: while(tryagain) { trusted_code_obtains_lock_1--> safe_code_callback1 calls-> trusted_code_obtains_lock2 safe_code_callback2 processes and sets tryagain=false } Of course, that could lead to starvation. So to avoid all of that multi-threaded libraries have to have 2 APIs, one simple transactional wrapper-API for safe code and another API for trusted code that can be used to write a new wrapper that allows locking of multiple datastructures. I hope people try to sketch out what such libraries will look like before the semantics are decided on. Maybe an experimental branch would be appropriate.
Here's a rough draft of one such sort of tool I use all the time in shared-intensive code: https://gist.github.com/TurkeyMan/c16db7a0be312e9a0a2083f5f4a6efec I just hacked that together the other night to do some experiments. I have a toolbox like that which we use at work. This code here feels fine to me in D, but it all depends on restrictive shared to work. There's nothing useful you can do with shared safely as it exists in the language today.
Oct 12
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 October 2019 at 23:36:55 UTC, Manu wrote:
 Here's a rough draft of one such sort of tool I use all the 
 time in shared-intensive code: 
 https://gist.github.com/TurkeyMan/c16db7a0be312e9a0a2083f5f4a6efec
Thanks! That looks quite low level, but now I understand more what you are looking for. What I had in mind was writing APIs that allows ordinary programmers to do parallell programming safely. Like taking the single threaded code they have written for processing a single array or merging two arrays with each other and then use a library for speeding it up. Anyway, my core argument is to put more meta-programming power in hands of library authors like you so that people who have the shoes on can define the semantics they are after. I really don't think this has to be done at the compiler level, given the right meta programming tools, based on the proposed semantics. And I don't think providing those meta programming tools are more work than hardwiring more stuff into the compiler. Clearly that is just my opinion. Others might feel differently. (Other semantics do require compiler support to work well, like heterogeneous memory architectures, but the proposal does not work with that.)
Oct 13
parent reply Manu <turkeyman gmail.com> writes:
On Sun, Oct 13, 2019 at 1:10 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 12 October 2019 at 23:36:55 UTC, Manu wrote:
 Here's a rough draft of one such sort of tool I use all the
 time in shared-intensive code:
 https://gist.github.com/TurkeyMan/c16db7a0be312e9a0a2083f5f4a6efec
Thanks! That looks quite low level, but now I understand more what you are looking for.
Well the implementation is low-level, obviously. But the tool is not. It's quite approachable at user level, the only risk being to avoid deadlock. If you want to systemically avoid dead-locks, then higher-level tools appear.
 What I had in mind was writing APIs that allows ordinary
 programmers to do parallell programming safely.
That doesn't exist. I would call that a 'framework', and that's a very high level library suite. What we're doing here is making it possible to write such a thing safely. 'Ordinary' programmers don't do concurrency on a low level. Period.
 Like taking the single threaded code they have written for
 processing a single array or merging two arrays with each other
 and then use a library for speeding it up.
I'm working towards a safe parallel-for... this needs to land first before I can attempt to argue for the next steps to make that work.
 Anyway, my core argument is to put more meta-programming power in
 hands of library authors like you so that people who have the
 shoes on can define the semantics they are after. I really don't
 think this has to be done at the compiler level, given the right
 meta programming tools, based on the proposed semantics. And I
 don't think providing those meta programming tools are more work
 than hardwiring more stuff into the compiler. Clearly that is
 just my opinion. Others might feel differently.
I've said this a few times now, and I'll say it again. Your insistence that shared should be a library thing applies equally to const. It behaves in an identical manner to const, and if you can't make a compelling case for moving const to a library, you can't possibly sell me on that idea with respect to shared. 1. `shared` defines D's thread-local by default semantics... without shared, there is no thread-local by default. That decision is so deep, there is absolutely no way to move away from that. 2. We must be able to overload on `shared`. We can't 'overload' on template wrappers; we can try and specialise and stuff and abuse clever inference and type deduction, but things break down quickly. I have tried to simulate type qualifiers with template wrappers before. It always goes *very* badly, and it's a mess to work with. 3. Most importantly, there is a future (not shown in this discussion) where shared should allow some implicit promotions safely; for instance: void fun(scope ref shared T arg); T x; fun(x); // OK; promote to shared, strictly bound to the lifetime of this call tree This will allow some of the simple deployment of algorithms you refer to above. Interaction with scope and other escape analysis tools is very low-level and quite complex, and we need shared to have a solid definition before we can experiment with that stuff. This is the key that will unlock parallel-for and other algorithms.
Oct 13
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 19:23:45 UTC, Manu wrote:
 Your insistence that shared should be a library thing applies 
 equally
 to const. It behaves in an identical manner to const, and if 
 you can't
 make a compelling case for moving const to a library, you can't
 possibly sell me on that idea with respect to shared.
I'm not trying to sell anything, but you cannot easily implement D's const using metaprogramming as metaprogramming is realized in D. You can easily extend the metaprogramming capabilities of D to implement shared. Apples and Oranges.
 1. `shared` defines D's thread-local by default semantics... 
 without
 shared, there is no thread-local by default. That decision is 
 so deep,
 there is absolutely no way to move away from that.
That argument makes no sense to me. Thread local or not thread local is essentially only by social agreement if you have syntactical means to get around it. It is predominantly a social contract. It is a matter of agreement, interpretation, conceptualization, mindset... That is what defines what is thread local in D. Not machinery. That is not how it is in a type system that is stronger like in Pony. Where it is strictly enforced and clearly defined fully semantically in the machinery with no means to break the type system. In that regard, D is not much different from C++, it is predominantly social and syntactical. Except in D you might have stronger agreement, stronger social contract, more visible syntactical symbols. So it appears to be stronger. But as long as D is going to interface tightly with C/C++ it will always rely on social contracts.
 2. We must be able to overload on `shared`. We can't 'overload' 
 on
 template wrappers; we can try and specialise and stuff and abuse
 clever inference and type deduction, but things break down 
 quickly. I
 have tried to simulate type qualifiers with template wrappers 
 before.
 It always goes *very* badly, and it's a mess to work with.
You have two alternatives: 1. you can do a simple one with template wrappers. 2. you can extend the meta programming capabilities and have a more advanced solution that is more in line with the current syntax But it certainly is possible, and not very hard to envision. After all making something completely unavailable, is much easier than only making some aspects of an object unavailable. Which is why "shared" is relatively easy to do on the syntactical level in D, but "const" is harder. If meta-programming in D was more deductive then const would be fair game too, but that is not the case, so... it becomes less obvious and probably not worth the effort.
 3. Most importantly, there is a future (not shown in this 
 discussion)
 where shared should allow some implicit promotions safely; for
 instance:
Ok, not sure if that really impacts whether it can be done in a library or not, but it certainly is VERY important to list such use-cases with sufficient detail when designing foundational mechanisms that is supposed to enable good interfaces to parallell programming libraries!
Oct 13
parent reply Manu <turkeyman gmail.com> writes:
On Sun, Oct 13, 2019 at 2:10 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 13 October 2019 at 19:23:45 UTC, Manu wrote:
 Your insistence that shared should be a library thing applies
 equally
 to const. It behaves in an identical manner to const, and if
 you can't
 make a compelling case for moving const to a library, you can't
 possibly sell me on that idea with respect to shared.
I'm not trying to sell anything, but you cannot easily implement D's const using metaprogramming as metaprogramming is realized in D. You can easily extend the metaprogramming capabilities of D to implement shared. Apples and Oranges.
How so? `shared` is exactly like const, but with an additional restriction that you can't read either.
 1. `shared` defines D's thread-local by default semantics...
 without
 shared, there is no thread-local by default. That decision is
 so deep,
 there is absolutely no way to move away from that.
That argument makes no sense to me. Thread local or not thread local is essentially only by social agreement if you have syntactical means to get around it. It is predominantly a social contract.
"if you have syntactical means to get around it" <- this is `shared`; if we lose shared, we lose thread-local.
 It is a matter of agreement, interpretation, conceptualization,
 mindset... That is what defines what is thread local in D. Not
 machinery.
Yeah, but if there's no way to describe not-thread-local, then thread-local doesn't really mean anything. Assuming there were no shared, and "data is thread-local" is specified, then you'd be doing UB anytime you spawn a second thread... then the language would just need to tolerate that, and accept that so-called thread-local data may or may not actually be thread-local (and conservatively assume not thread-local), because no way to distinguish. I mean, I understand your point of view, but in practical terms, it's not correct. thread-local by default exists because `shared` exists.
 2. We must be able to overload on `shared`. We can't 'overload'
 on
 template wrappers; we can try and specialise and stuff and abuse
 clever inference and type deduction, but things break down
 quickly. I
 have tried to simulate type qualifiers with template wrappers
 before.
 It always goes *very* badly, and it's a mess to work with.
You have two alternatives: 1. you can do a simple one with template wrappers.
It goes very badly every time I've tried this.
 2. you can extend the meta programming capabilities and have a
 more advanced  solution that is more in line with the current
 syntax
That's a huge development, and much uncertainty. And it's not like we're introducing shared here... he's been in D forever, we're just making it mean something.
 But it certainly is possible, and not very hard to envision.
 After all making something completely unavailable, is much easier
 than only making some aspects of an object unavailable. Which is
 why "shared" is relatively easy to do on the syntactical level in
 D, but "const" is harder.
I don't understand... I think they're almost identical hard-ness.
 3. Most importantly, there is a future (not shown in this
 discussion)
 where shared should allow some implicit promotions safely; for
 instance:
Ok, not sure if that really impacts whether it can be done in a library or not, but it certainly is VERY important to list such use-cases with sufficient detail when designing foundational mechanisms that is supposed to enable good interfaces to parallell programming libraries!
We're not designing foundational mechanisms, Walter designed shared decades ago and it's firmly embedded in the language. We're just making it not completely broken right now. Designing a better shared is future work, which needs to start from stable ground.
Oct 13
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 13 October 2019 at 21:44:45 UTC, Manu wrote:
 On Sun, Oct 13, 2019 at 2:10 PM Ola Fosheim Grøstad via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 How so?
 `shared` is exactly like const, but with an additional 
 restriction
 that you can't read either.
Shared provides only identity of the top level. Const provides recursively methods for reading.
 "if you have syntactical means to get around it" <- this is 
 `shared`; if we lose shared, we lose thread-local.
Nah, C++ memory was local... Threads came through a library/syscall. Didn't loose or win the concept of local memory... It is semiotic if not strictly enforced, a matter of interpretation. A matter of intersubjectivity, e.g people sharing the same interpretation.
 Assuming there were no shared, and "data is thread-local" is
 specified, then you'd be doing UB anytime you spawn a second 
 thread...
No, they could have disjunct stores. Like JavaScript where you transfer unique ownership. Conceptually close to using a mutex... Except in the mutex case there is no formal representation, the connection between mutex and object is in the head of the programmers.
 not correct. thread-local by default exists because `shared` 
 exists.
Nah, I'd rather say that you can transfer ownership. Shared just means that it is up for grabs, it is in transition. Just like in JavaScript. At some point the mediating system is in control. No conflict.
 It goes very badly every time I've tried this.
I'll have to take your word for it... Something serious?
 2. you can extend the meta programming capabilities and have a 
 more advanced  solution that is more in line with the current 
 syntax
That's a huge development, and much uncertainty. And it's not like we're introducing shared here... he's been in D forever, we're just making it mean something.
Yes, there might be a cultural barrier to changing the syntax/expected features.
 I don't understand... I think they're almost identical 
 hard-ness.
You can a simple shared wrapper in 10 lines, so I don't understand the "identical" part...
 We're not designing foundational mechanisms, Walter designed 
 shared
 decades ago and it's firmly embedded in the language.
Designed? If it was, there would be no DIP.
Oct 13
parent Manu <turkeyman gmail.com> writes:
On Sun, Oct 13, 2019 at 4:30 PM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 13 October 2019 at 21:44:45 UTC, Manu wrote:
 On Sun, Oct 13, 2019 at 2:10 PM Ola Fosheim Grøstad via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 How so?
 `shared` is exactly like const, but with an additional
 restriction
 that you can't read either.
Shared provides only identity of the top level. Const provides recursively methods for reading.
 "if you have syntactical means to get around it" <- this is
 `shared`; if we lose shared, we lose thread-local.
Nah, C++ memory was local... Threads came through a library/syscall. Didn't loose or win the concept of local memory... It is semiotic if not strictly enforced, a matter of interpretation. A matter of intersubjectivity, e.g people sharing the same interpretation.
 Assuming there were no shared, and "data is thread-local" is
 specified, then you'd be doing UB anytime you spawn a second
 thread...
No, they could have disjunct stores. Like JavaScript where you transfer unique ownership.
They could, but they don't.. You're talking about a lot of hypothetical, I'm talking about how D is, and how it's always been.
 It goes very badly every time I've tried this.
I'll have to take your word for it... Something serious?
It just doesn't scale, and relies extremely heavily on template constraints. You lose; function pointers, virtual functions, binary libs, DLL's, etc... you also lose auto-complete style features in the editor. Templates should be the solution to generic code problems, not to problems like this.
 2. you can extend the meta programming capabilities and have a
 more advanced  solution that is more in line with the current
 syntax
That's a huge development, and much uncertainty. And it's not like we're introducing shared here... he's been in D forever, we're just making it mean something.
Yes, there might be a cultural barrier to changing the syntax/expected features.
What's the point of that tangent though? We're not seriously going to consider changing the way const is defined in D, and make a proposal for user-defines type constructors in a conversation about making shared work properly.
 We're not designing foundational mechanisms, Walter designed
 shared
 decades ago and it's firmly embedded in the language.
Designed? If it was, there would be no DIP.
I agree it should have been like this from the start, but it exists, it's the tool we have, and we need to make it work properly.
Oct 14
prev sibling parent Manu <turkeyman gmail.com> writes:
On Sat, Oct 12, 2019 at 2:20 AM Ola Fosheim Grøstad via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 12 October 2019 at 06:31:18 UTC, Jonathan M Davis
 wrote:
 On Friday, October 11, 2019 10:58:29 PM MDT Walter Bright via
 Digitalmars-d wrote:
 On 10/2/2019 3:42 AM, Nicholas Wilson wrote:
 It should be that shared memory access is disabled in all
 contexts,
 which must be worked around with casts,
 which makes the function  system,
 which must then be encapsulated with a  trusted interface in
 order to
 use in  safe code.
Sounds right.
This is pretty much what several of us were arguing for during the discussions about shared at dconf this year.
How are you going to prove that safe code does not retain nonshared-references after the lock has been released? How does his work with array elements/slices.
I think `scope` and `return` have something to say about avoiding escaping references. When the semantics are all working, we can produce such test code.
Oct 12
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 2 October 2019 at 08:55:59 UTC, Kagamin wrote:
 Access to shared memory should be disallowed only for safe 
 code, but not for system code. For safe code it's justified, 
 because safety is unavoidable there, for system code there's a 
 concern of ease of use of shared data, which is affected by 
 littering and is not addressed in this DIP. Stripping shared 
 qualifier off for assignment isn't necessarily correct if, say, 
 the data type switches between atomic and nonatomic reference 
 counting based on its type.
system code can always cast away shared.
Oct 02
parent reply Kagamin <spam here.lot> writes:
On Wednesday, 2 October 2019 at 11:49:48 UTC, Atila Neves wrote:
  system code can always cast away shared.
It's justified for safe code, because safety is mandatory there, but not for system code. Access to shared data is well within semantics of system code. System code is even intended to access shared data because safe code can't do it in general case.
Oct 03
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 3 October 2019 at 09:29:28 UTC, Kagamin wrote:
 On Wednesday, 2 October 2019 at 11:49:48 UTC, Atila Neves wrote:
  system code can always cast away shared.
It's justified for safe code, because safety is mandatory there, but not for system code. Access to shared data is well within semantics of system code. System code is even intended to access shared data because safe code can't do it in general case.
A coworker of mine (Don Clugston) came up with this analogy. Multi-threading is a bit like radiating material, you put it in a safe-box, and put a warning sticker on it, such that only professionals may open it. And even the professional would not want to be around the material for very long and certainly would want to be warned before he accesses the material. Therefore the material should be in it's designated box sealed away fore the majority of the time.
Oct 03
parent reply Kagamin <spam here.lot> writes:
On Thursday, 3 October 2019 at 12:14:30 UTC, Stefan Koch wrote:
 A coworker of mine (Don Clugston) came up with this analogy.
 Multi-threading is a bit like radiating material, you put it in 
 a safe-box, and put a warning sticker on it, such that only 
 professionals may open it.
 And even the professional would not want to be around the 
 material for very long and certainly would want to be warned 
 before he accesses the material.
 Therefore the material should be in it's designated box sealed 
 away fore the majority of the time.
Sounds like that coworker wants safe code. The DIP is justified for safe code, but not for system code. Also the DIP provides memory safety, not thread safety.
Oct 04
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 4 October 2019 at 10:12:45 UTC, Kagamin wrote:
 On Thursday, 3 October 2019 at 12:14:30 UTC, Stefan Koch wrote:
 A coworker of mine (Don Clugston) came up with this analogy.
 Multi-threading is a bit like radiating material, you put it 
 in a safe-box, and put a warning sticker on it, such that only 
 professionals may open it.
 And even the professional would not want to be around the 
 material for very long and certainly would want to be warned 
 before he accesses the material.
 Therefore the material should be in it's designated box sealed 
 away fore the majority of the time.
Sounds like that coworker wants safe code. The DIP is justified for safe code, but not for system code. Also the DIP provides memory safety, not thread safety.
The reason why you want shared access to be forbidden by default is to see where synchronization is required and force you to think about it. In all your code not just safe code.
Oct 04
parent reply Kagamin <spam here.lot> writes:
On Friday, 4 October 2019 at 12:22:16 UTC, Stefan Koch wrote:
 The reason why you want shared access to be forbidden by 
 default is to see where synchronization is required and force 
 you to think about it.
 In all your code not just safe code.
That's achieved by writing safe code, so the failure requires transition to trusted code, where there's type system and lack of conversion between shared and unshared data. Adding churn indefinitely isn't very practical, it increases cognitive load on programmer and diverts resources that can be spent on thinking about synchronization. Also the idea ignores the problem that people deem it difficult to work with shared data, adding more churn only worsens that problem, which is traded for an alleged solution to a hypothetical problem.
Oct 07
parent reply Manu <turkeyman gmail.com> writes:
On Mon, Oct 7, 2019 at 1:05 AM Kagamin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Friday, 4 October 2019 at 12:22:16 UTC, Stefan Koch wrote:
 The reason why you want shared access to be forbidden by
 default is to see where synchronization is required and force
 you to think about it.
 In all your code not just safe code.
That's achieved by writing safe code, so the failure requires transition to trusted code, where there's type system and lack of conversion between shared and unshared data. Adding churn indefinitely isn't very practical, it increases cognitive load on programmer and diverts resources that can be spent on thinking about synchronization. Also the idea ignores the problem that people deem it difficult to work with shared data, adding more churn only worsens that problem, which is traded for an alleged solution to a hypothetical problem.
This made exactly no sense to me. I don't know what you're saying here. That said, based on your past posts, I think I need to assure you that it is **NEVER** acceptable to freely read/write to shared data, and that doesn't only apply to safe code. I wish Walter would update this DIP, but based on his comments, I think it's going the right way...?
Oct 09
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 9 October 2019 at 18:25:36 UTC, Manu wrote:
 This made exactly no sense to me. I don't know what you're 
 saying here. That said, based on your past posts, I think I 
 need to assure you that it is **NEVER** acceptable to freely 
 read/write to shared data, and that doesn't only apply to  safe 
 code.
Maybe set up a list of what problems and opportunities "shared" is supposed to deal with? What will it solve, and just as importantly, list explicitly which issues it won't deal with. As far as I can tell, one could get better optimization if one could assume that certain memory areas are not accessed by other threads and avoid C++-ish sequencing points even in code that use concurrency-primitives. That seems to be out of scope? If C++ is supposed to interface with D, then the spec ought to present an argument for how D's memory model is compatible with C++, which may be summed up as (quoting cppreference.com): «Any thread can potentially access any object in the program (objects with automatic and thread-local storage duration may still be accessed by another thread through a pointer or by reference). Different threads of execution are always allowed to access (read and modify) different memory locations concurrently, with no interference and no synchronization requirements.»
Oct 10
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Thursday, 10 October 2019 at 07:03:26 UTC, Ola Fosheim Grøstad 
wrote:
 As far as I can tell, one could get better optimization if one 
 could assume that certain memory areas are not accessed by 
 other threads and avoid C++-ish sequencing points even in code 
 that use concurrency-primitives. That seems to be out of scope?
Or to be blunt, if "shared" has no effect on the IR then it ought to be removed from the language and replaced with mechanisms that makes it possible to implement the "shared" marker functionality as a template wrapper.
Oct 10
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/9/2019 11:25 AM, Manu wrote:
 I wish Walter would update this DIP, but based on his comments, I
 think it's going the right way...?
I'll update it to clear up the ambiguity in it after this review cycle is over. It'll be clear that reads and writes to shared data will not be allowed directly, you'll have to go through a function.
Oct 11
prev sibling next sibling parent tn <no email.com> writes:
In my opinion, if a DIP is so confusingly written that most 
people don't understand what it is proposing, the review should 
be stopped, the DIP should be rewritten and then a new review 
should be started. Otherwise the review thread just gets full of 
irrelevant comments that are based on misunderstandings, and 
therefore becomes hard to follow.

This is the case for this DIP.

This comment by Walter made it more clear, where the 
inconsistencies originate from:

On Friday, 4 October 2019 at 03:35:45 UTC, Walter Bright wrote:
 Yes. The trouble stems from the DIP being originally written to 
 apply shared semantics to operators. This was changed to be 
 function only, and some vestiges of the original remain. Hence 
 the confusion.
It is not, however, an excuse not to fix the text before restarting the review. *** When it comes to the actual content of this DIP, it should describe clearly both 1) what the current behavior is before this DIP 2) what the behavior would be after this DIP (Currently 1 is either not addressed at all, or described very implicitly. And 1 is described inconsistently and not very clearly.)
Oct 04
prev sibling parent Mike Parker <aldacron gmail.com> writes:
On Tuesday, 1 October 2019 at 10:40:52 UTC, Mike Parker wrote:
 This is the feedback thread for the first round of Community 
 Review for DIP 1024, "Shared Atomics":

 https://github.com/dlang/DIPs/blob/0b892dd99aba74b9631572ad3a53000f5975b7c2/DIPs/DIP1024.md
Apologies to everyone who participated in the review. I didn't give the discussion my full attention, else I would have called off the review early so that the document could be corrected. Generally, that's what the policy should be when such confusing ambiguities slip through the revision process. So if it does happen again in the future, I ask that whoever catches it please ping me (email or slack is the best bet) in case I miss it. Walter is going to revise the DIP and we will do another round of community review soon.
Oct 17