digitalmars.D.announce - nogc v0.5.0 - DIP1008 works!
- Atila Neves (27/27) May 24 2019 I'd been holding off on announcing this until DIP1008 actually
- Seb (5/8) May 24 2019 Awesome!!
- ag0aep6g (50/54) May 24 2019 You've got safety violations:
- Atila Neves (6/59) May 24 2019 Thanks for this. I think the only violation is calling `stringz`
- ag0aep6g (6/11) May 24 2019 I think you're missing the point. When your function is marked as
- Atila Neves (3/14) May 24 2019 Could you please give an example of how @system allocator code
- ag0aep6g (39/45) May 24 2019 Sure. You just write beyond some buffer instead of calling `puts`:
- ag0aep6g (4/8) May 24 2019 [...]
- Meta (17/63) May 24 2019 That's right. If you are wrapping code that is provided by a
- Atila Neves (17/63) May 27 2019 I don't see the problem here. This example would throw RangeError
- Paolo Invernizzi (7/12) May 27 2019 ... and a @safe conscious programmer will not touch that library
- Atila Neves (3/13) May 27 2019 I'm not convinced either - I'm having a dialogue to figure out
- Paolo Invernizzi (7/23) May 27 2019 :-)
- ag0aep6g (9/23) May 27 2019 No, it doesn't. It's a complete, runnable example. You can try it at
- Atila Neves (7/22) May 27 2019 No, and I guess it can't. I'm trying to figure out what the
- ag0aep6g (5/8) May 27 2019 For @safe allocators, Vector can be @safe.
- Atila Neves (7/17) May 27 2019 It's ugly but would work. Right now I don't think I can do any
- ag0aep6g (9/12) May 27 2019 Oh, yeah. Getting @trusted right is hard. Getting it right when
- Valeriy Fedotov (12/16) May 29 2019 In my point of view @trusted means "I use pointer-related
- ag0aep6g (22/32) May 29 2019 @trusted means: "This function is as safe as an @safe function. But the
- Mike Franklin (10/19) May 24 2019 Ok, so exceptions don't rely on the GC anymore. That's super
I'd been holding off on announcing this until DIP1008 actually got implemented, and now it has: https://code.dlang.org/packages/nogc This dub package has a nogc version of `std.conv.text` (which probably isn't as good yet) that, instead of returning a `string` returns an `automem.vector.Vector` of char. This handles managing memory allocation for the exception message itself in `NoGcException`, which does what it says on the tin. Confused? Here's some code: safe nogc unittest { import nogc; import std.algorithm: equal; int a = 1, b = 2; try enforce(a == b, a, " was not equal to ", b); catch(NoGcException e) { assert(equal(e.msg, "1 was not equal to 2")); } try throw new NoGcException(42, " foobar ", 33.3); catch(NoGcException e) { assert(equal(e.msg, "42 foobar 33.300000")); assert(e.file == __FILE__); assert(e.line == __LINE__ - 4); } } It doesn't leak memory either, as proved by ldc's asan.
May 24 2019
On Friday, 24 May 2019 at 11:41:12 UTC, Atila Neves wrote:I'd been holding off on announcing this until DIP1008 actually got implemented, and now it has: [...]Awesome!! Now we just need to get to compile Druntime and Phobos with -preview=dip1008, s.t. we can enable it by default :) See e.g. https://github.com/dlang/dmd/pull/8508
May 24 2019
On 24.05.19 13:41, Atila Neves wrote:I'd been holding off on announcing this until DIP1008 actually got implemented, and now it has: https://code.dlang.org/packages/nogcYou've got safety violations: ---- /+ dub.sdl: name "test" dependency "nogc" version="~>0.5.0" +/ import core.stdc.stdio: puts; struct S1 { S2 s2; this(ref const S1 src) const nogc system { this.s2 = src.s2; } } struct S2 { this(ref const S2 src) const nogc system { puts(" system 1"); } } struct Z { char* stringz() const nogc system { puts(" system 2"); return null; } } struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; enum instance = UnsafeAllocator.init; void deallocate(void[] bytes) nogc system { puts(" system 3"); Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { puts(" system 4"); return Mallocator.instance.allocate(sz); } } void main() safe nogc { import nogc: BUFFER_SIZE, text; S1 a; Z* z; auto t = text!(BUFFER_SIZE, UnsafeAllocator)(a, z); } ---- All of the `puts` lines are executed. That should not be possible in safe code. You're applying trusted too liberally.
May 24 2019
On Friday, 24 May 2019 at 12:32:45 UTC, ag0aep6g wrote:On 24.05.19 13:41, Atila Neves wrote:Thanks for this. I think the only violation is calling `stringz` on `Z`, and that was due to a poorly designed DbI check on being able to call `stringz`. Allocating generally isn't system, and freeing is ok to trust since vector is taking care of it for us. I've pushed a fix.[...]You've got safety violations: ---- /+ dub.sdl: name "test" dependency "nogc" version="~>0.5.0" +/ import core.stdc.stdio: puts; struct S1 { S2 s2; this(ref const S1 src) const nogc system { this.s2 = src.s2; } } struct S2 { this(ref const S2 src) const nogc system { puts(" system 1"); } } struct Z { char* stringz() const nogc system { puts(" system 2"); return null; } } struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; enum instance = UnsafeAllocator.init; void deallocate(void[] bytes) nogc system { puts(" system 3"); Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { puts(" system 4"); return Mallocator.instance.allocate(sz); } } void main() safe nogc { import nogc: BUFFER_SIZE, text; S1 a; Z* z; auto t = text!(BUFFER_SIZE, UnsafeAllocator)(a, z); } ---- All of the `puts` lines are executed. That should not be possible in safe code. You're applying trusted too liberally.
May 24 2019
On Friday, 24 May 2019 at 13:13:12 UTC, Atila Neves wrote:Thanks for this. I think the only violation is calling `stringz` on `Z`, and that was due to a poorly designed DbI check on being able to call `stringz`. Allocating generally isn't system, and freeing is ok to trust since vector is taking care of it for us. I've pushed a fix.I think you're missing the point. When your function is marked as safe or trusted, then any execution of a user-provided system function is a safety violation. My `puts`s might not do any harm, but they could just as well be buffer overflows.
May 24 2019
On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:On Friday, 24 May 2019 at 13:13:12 UTC, Atila Neves wrote:Could you please give an example of how system allocator code could do that?Thanks for this. I think the only violation is calling `stringz` on `Z`, and that was due to a poorly designed DbI check on being able to call `stringz`. Allocating generally isn't system, and freeing is ok to trust since vector is taking care of it for us. I've pushed a fix.I think you're missing the point. When your function is marked as safe or trusted, then any execution of a user-provided system function is a safety violation. My `puts`s might not do any harm, but they could just as well be buffer overflows.
May 24 2019
On 24.05.19 18:19, Atila Neves wrote:On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:[...]Sure. You just write beyond some buffer instead of calling `puts`: ---- char[3] buf; char[3] foo = "foo"; char[3] bar = "bar"; struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; static instance = UnsafeAllocator.init; size_t i; void deallocate(void[] bytes) nogc system { buf.ptr[i .. i + 3] = '!'; Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { buf.ptr[i .. i + 3] = '!'; return Mallocator.instance.allocate(sz); } } void main() safe nogc { { import nogc: BUFFER_SIZE, text; UnsafeAllocator.instance.i = 8; /* greater than buf.length, whoops */ auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42); assert(foo == "foo"); /* fails */ UnsafeAllocator.instance.i = 16; /* also greater than buf.length, whoops again */ } assert(bar == "bar"); /* fails */ } ---- You just can't trust user-provided system code. It doesn't matter if it's allocator code or whatever.My `puts`s might not do any harm, but they could just as well be buffer overflows.Could you please give an example of how system allocator code could do that?
May 24 2019
On 24.05.19 18:51, ag0aep6g wrote:char[3] buf;[...]buf.ptr[i .. i + 3] = '!';[...]UnsafeAllocator.instance.i = 8; /* greater than buf.length, whoops */Actually, anything greater than zero is a whoops, of course.
May 24 2019
On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:On 24.05.19 18:19, Atila Neves wrote:That's right. If you are wrapping code that is provided by a third party, you should not mark any code as trusted that makes calls to the third party library. By doing this, you are saying "any third party code I call is memory safe (source: dude just trust me)". That may work in the case where this third party code is set in stone and has been hand-audited by either you or the maintainers (ideally both), but you're accepting any implementation through a template argument. Doing this is extremely dangerous, because you're making memory safety promises about every single Allocator implementation in existence, in the present AND for the future. What you have to do is leave the functions that make these calls unmarked (no system, trusted OR safe), and allow the compiler to infer it based on the whether the third party implementation is system/ trusted/ safe. That's the only sane way I can think of to do this.On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:[...]Sure. You just write beyond some buffer instead of calling `puts`: ---- char[3] buf; char[3] foo = "foo"; char[3] bar = "bar"; struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; static instance = UnsafeAllocator.init; size_t i; void deallocate(void[] bytes) nogc system { buf.ptr[i .. i + 3] = '!'; Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { buf.ptr[i .. i + 3] = '!'; return Mallocator.instance.allocate(sz); } } void main() safe nogc { { import nogc: BUFFER_SIZE, text; UnsafeAllocator.instance.i = 8; /* greater than buf.length, whoops */ auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42); assert(foo == "foo"); /* fails */ UnsafeAllocator.instance.i = 16; /* also greater than buf.length, whoops again */ } assert(bar == "bar"); /* fails */ } ---- You just can't trust user-provided system code. It doesn't matter if it's allocator code or whatever.My `puts`s might not do any harm, but they could just as well be buffer overflows.Could you please give an example of how system allocator code could do that?
May 24 2019
On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:On 24.05.19 18:19, Atila Neves wrote:I don't see the problem here. This example would throw RangeError at runtime instead of actually overwriting memory unless bounds checking is turned off. The other issue is that Mallocator has a safe allocate function and a system deallocate function since it's up to the user of the interface to supply a slice that was actually malloc'ed. It's clear that this interface is one that can be used safely (and is by automem.vector.Vector). Likewise, reallocating is system because there might be references to the old pointer, but it makes sense that a trusted block could exist where the reviewer makes sure that there's only ever one reference to the allocated memory. Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with trusted, and unless someone looks at their code nobody will be the wiser.On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:[...]Sure. You just write beyond some buffer instead of calling `puts`: ---- char[3] buf; char[3] foo = "foo"; char[3] bar = "bar"; struct UnsafeAllocator { import std.experimental.allocator.mallocator: Mallocator; static instance = UnsafeAllocator.init; size_t i; void deallocate(void[] bytes) nogc system { buf.ptr[i .. i + 3] = '!'; Mallocator.instance.deallocate(bytes); } void[] allocate(size_t sz) nogc system { buf.ptr[i .. i + 3] = '!'; return Mallocator.instance.allocate(sz); } } void main() safe nogc { { import nogc: BUFFER_SIZE, text; UnsafeAllocator.instance.i = 8; /* greater than buf.length, whoops */ auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42); assert(foo == "foo"); /* fails */ UnsafeAllocator.instance.i = 16; /* also greater than buf.length, whoops again */ } assert(bar == "bar"); /* fails */ } ---- You just can't trust user-provided system code. It doesn't matter if it's allocator code or whatever.My `puts`s might not do any harm, but they could just as well be buffer overflows.Could you please give an example of how system allocator code could do that?
May 27 2019
On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote:On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with trusted, and unless someone looks at their code nobody will be the wiser.... and a safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more? Thanks, Paolo
May 27 2019
On Monday, 27 May 2019 at 09:07:48 UTC, Paolo Invernizzi wrote:On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote:I'm not convinced either - I'm having a dialogue to figure out potential issues.On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with trusted, and unless someone looks at their code nobody will be the wiser.... and a safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more?
May 27 2019
On Monday, 27 May 2019 at 10:01:15 UTC, Atila Neves wrote:On Monday, 27 May 2019 at 09:07:48 UTC, Paolo Invernizzi wrote::-) My nice-try to reduce the problem is: trusted code block needs to by "manually verified" for safety by humans, so it should be " safe pure", aka, if you can't perform the analysis looking only at the statements in the trusted block, that can't be marked trusted.On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote:I'm not convinced either - I'm having a dialogue to figure out potential issues.On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with trusted, and unless someone looks at their code nobody will be the wiser.... and a safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more?
May 27 2019
On 27.05.19 10:54, Atila Neves wrote:I don't see the problem here. This example would throw RangeError at runtime instead of actually overwriting memory unless bounds checking is turned off.No, it doesn't. It's a complete, runnable example. You can try it at home. It does overwrite `foo` and `bar`. It does not throw a RangeError.The other issue is that Mallocator has a safe allocate function and a system deallocate function since it's up to the user of the interface to supply a slice that was actually malloc'ed. It's clear that this interface is one that can be used safely (and is by automem.vector.Vector). Likewise, reallocating is system because there might be references to the old pointer, but it makes sense that a trusted block could exist where the reviewer makes sure that there's only ever one reference to the allocated memory.Yes, you can use trusted to use Mallocator safely. And your code (probably) does that. But the allocator in my example isn't Mallocator, it's UnsafeAllocator. Your code doesn't use that one safely.Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with trusted, and unless someone looks at their code nobody will be the wiser.In this thread, you're the author of that 3rd party library. You've got the bad trusted functions that lead to memory corruption. I'm the guy who looked at it, noticed the problem, and tells you.
May 27 2019
On Monday, 27 May 2019 at 09:48:27 UTC, ag0aep6g wrote:On 27.05.19 10:54, Atila Neves wrote:You're right - I should have run it first.I don't see the problem here. This example would throw RangeError at runtime instead of actually overwriting memory unless bounds checking is turned off.No, it doesn't. It's a complete, runnable example. You can try it at home. It does overwrite `foo` and `bar`. It does not throw a RangeError.Yes, you can use trusted to use Mallocator safely. And your code (probably) does that. But the allocator in my example isn't Mallocator, it's UnsafeAllocator. Your code doesn't use that one safely.No, and I guess it can't. I'm trying to figure out what the implications are. Can Vector only be safe for Mallocator? Is it possible to write a safe Vector at all without having to force the allocator to be safe?In this thread, you're the author of that 3rd party library. You've got the bad trusted functions that lead to memory corruption. I'm the guy who looked at it, noticed the problem, and tells you.Thanks for bringing it up.
May 27 2019
On 27.05.19 12:06, Atila Neves wrote:No, and I guess it can't. I'm trying to figure out what the implications are. Can Vector only be safe for Mallocator? Is it possible to write a safe Vector at all without having to force the allocator to be safe?For safe allocators, Vector can be safe. For specific system allocators, like Mallocator, you can make special trusted cases in Vector. For generic system allocators, Vector cannot be safe (or trusted).
May 27 2019
On Monday, 27 May 2019 at 10:31:10 UTC, ag0aep6g wrote:On 27.05.19 12:06, Atila Neves wrote:It's ugly but would work. Right now I don't think I can do any better than to follow your suggestion, but I predict many beard-stroking walks for me along Lake Geneva in the near future. I'd be nice if I could detect at compile-time that it's not just Mallocator but an allocator that's built using it as well (e.g. FallBackAllocator).No, and I guess it can't. I'm trying to figure out what the implications are. Can Vector only be safe for Mallocator? Is it possible to write a safe Vector at all without having to force the allocator to be safe?For safe allocators, Vector can be safe. For specific system allocators, like Mallocator, you can make special trusted cases in Vector. For generic system allocators, Vector cannot be safe (or trusted).
May 27 2019
On 27.05.19 15:34, Atila Neves wrote:It's ugly but would work. Right now I don't think I can do any better than to follow your suggestion, but I predict many beard-stroking walks for me along Lake Geneva in the near future.Oh, yeah. Getting trusted right is hard. Getting it right when user-provided types are involved is extra hard, because you can't even trust fundamental operations like assignment or copying. Please note that the allocator stuff is just one of the three violations I had pointed out. You've already pushed a fix for the unsafe .stringz, but you haven't addressed unsafe copy constructors yet. And my list wasn't meant to be complete. There may be other safety holes I didn't notice.
May 27 2019
On Monday, 27 May 2019 at 14:26:16 UTC, ag0aep6g wrote:Oh, yeah. Getting trusted right is hard. Getting it right when user-provided types are involved is extra hard, because you can't even trust fundamental operations like assignment or copying.In my point of view trusted means "I use pointer-related operations correctly. Also I am using all system interfaces correctly". The code in question uses allocator interface correctly. User of this code has a contract to supply allocator that conforms to this interface. If a user supplies mallocator that is not correct, there are two possibilities: - Allocator is buggy. Nothing to do with trusted code. - Allocator do not conforms to allocator interface. User has broken the contract. Nothing to do with trusted code. I think we should keep in mind not only technical aspects of trusted and system, but this contract too.
May 29 2019
On 29.05.19 18:37, Valeriy Fedotov wrote:In my point of view trusted means "I use pointer-related operations correctly. Also I am using all system interfaces correctly".trusted means: "This function is as safe as an safe function. But the compiler doesn't verify the implementation." Whether you use anything correctly only matters when it affects safety. And "being safe" means that the function doesn't exhibit undefined behavior (when called with valid inputs), and that it doesn't lead to undefined behavior in safe code that is executed later on (e.g., an trusted function must not return an invalid pointer). [...]If a user supplies mallocator that is not correct, there are two possibilities: - Allocator is buggy. Nothing to do with trusted code. - Allocator do not conforms to allocator interface. User has broken the contract. Nothing to do with trusted code.In my example, UnsafeAllocator is correct. It's not buggy, and it conforms to the allocator interface (which doesn't require safe methods). Still the program exhibits memory corruption. So there must be something else that's wrong. The problem is that I'm setting UnsafeAllocator.instance.i to bad values which leads to out-of-bounds accesses. I do that in the safe `main`. You might say: "Don't do that then. It's your own fault if you do that. Your mistake is not the library's fault." But D's safe is supposed catch exactly that kind of mistake. The point of safe is that I can make all the mistakes in the world in safe code (except typing out " trusted") without ever seeing memory corruption.I think we should keep in mind not only technical aspects of trusted and system, but this contract too.No. A contract that can be broken in safe code cannot be relied upon in trusted code. This is fundamental to D's safety system.
May 29 2019
On Friday, 24 May 2019 at 11:41:12 UTC, Atila Neves wrote:I'd been holding off on announcing this until DIP1008 actually got implemented, and now it has: https://code.dlang.org/packages/nogc This dub package has a nogc version of `std.conv.text` (which probably isn't as good yet) that, instead of returning a `string` returns an `automem.vector.Vector` of char. This handles managing memory allocation for the exception message itself in `NoGcException`, which does what it says on the tin. Confused? Here's some code:Ok, so exceptions don't rely on the GC anymore. That's super cool. However, they are still classes. So does that mean they also need RTTI (i.e. TypeInfo)? BetterC builds and some of use trying to use D in a pay-as-you-go fashion intentionally eliminate RTTI from the runtime. Is there any way we can take this a bit further to no longer require RTTI? Do exceptions even necessarily need to be classes? Thanks, Mike
May 24 2019