digitalmars.D - A safer interface for core.stdc
- Andrei Alexandrescu (28/28) Feb 07 2015 I was looking into ways to make core.stdc safer. That should be
- bearophile (5/7) Feb 07 2015 I'd also like a safer templated wrapper for calloc() and malloc()
- H. S. Teoh via Digitalmars-d (19/25) Feb 07 2015 [...]
- Andrei Alexandrescu (27/49) Feb 07 2015 I think that would go as follows:
- H. S. Teoh via Digitalmars-d (8/14) Feb 07 2015 This is a truly strange way of writing it... why not:
- Vlad Levenfeld (3/6) Feb 07 2015 I think Andrei's version will remain in release builds, but yours
- Andrei Alexandrescu (2/11) Feb 07 2015 assert(0) is not removed in release mode. -- Andrei
- H. S. Teoh via Digitalmars-d (8/23) Feb 07 2015 Ah, right. But shouldn't it be enforce instead of assert, then? :-P
- FG (5/6) Feb 08 2015 How about this to prevent double free:
- John Colvin (4/16) Feb 08 2015 I don't have any data, but I'd image most double-frees come from
- Andrei Alexandrescu (4/23) Feb 08 2015 I think the same. In C++ circles zeroing the pointer after freeing is
- Walter Bright (4/6) Feb 10 2015 What worked for me was using a free() that overwrote the free'd memory w...
- Paulo Pinto (5/17) Feb 08 2015 A typical C debug library trick for when the money for a proper
- tcak (8/38) Feb 07 2015 One of the reasons why I use C functions is that I expect same
- Andrei Alexandrescu (2/7) Feb 07 2015 Just looking at making them safe. Not all can be made safe btw. -- Andre...
- H. S. Teoh via Digitalmars-d (14/23) Feb 07 2015 Come to think of it, is there any point in making malloc @safe/@trusted
- Andrei Alexandrescu (3/8) Feb 07 2015 Same goes about e.g. fopen vs. fclose. I'm thinking just of increasing
- "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= (5/17) Feb 08 2015 So are you going for this:
- bearophile (6/8) Feb 08 2015 I am not asking for a @trusted function. I'd like a @system
- Tobias Pankrath (2/7) Feb 08 2015 At least two programs, widely used by folks here, never release
- Johannes Pfau (16/52) Feb 08 2015 This might be a good idea, but it might also be more difficult than you
- Andrei Alexandrescu (10/44) Feb 08 2015 I thought of a few things, nothing is 100% foolproof. But I'm not too
- Dicebot (8/38) Feb 09 2015 I think this is crucial if we want to keep actual Phobos sources
- Andrei Alexandrescu (4/10) Feb 09 2015 Walter opposes this on grounds of increased maintenance burdens. He
- Dicebot (5/21) Feb 09 2015 I foresee certain stdc-based primitives being duplicated over and
- Andrei Alexandrescu (2/5) Feb 09 2015 Would be nice to encapsulate them in Phobos on a case basis. -- Andrei
- finalpatch (6/22) Feb 11 2015 Please don't waste your limited resource on this. When people use
I was looking into ways to make core.stdc safer. That should be relatively easy to do by defining a few wrappers. For example: int setvbuf(FILE* stream, char* buf, int mode, size_t size); is unsafe because there's no relationship between buf and size. But this is fine: trusted int setvbuf(T)(FILE* stream, T[] buf, int mode) if (is(T == char) || is(T == byte) || is(T == ubyte)) { return setvbuf(stream, cast(char*) buf.ptr, mode, buf.length); } Another example is: int stat(in char*, stat_t*); which may start reading through random memory if the string is not zero-terminated. Again, the solution is here to ensure the string does have a terminating zero: trusted int stat(in char[] name, stat_t* p) { if (isZeroTerminated(name)) return stat(name.ptr, p); auto t = cast(char*) malloc(name.length + 1); scope(exit) free(t); memcpy(t, name.ptr, name.length); t[name.length] = 0; return stat(t, p); } Such wrappers would allow safe code to use more C stdlib primitives. The question is whether these wrappers are worth adding to core.stdc.stdio. Thanks, Andrei
Feb 07 2015
Andrei Alexandrescu:Such wrappers would allow safe code to use more C stdlib primitives.I'd also like a safer templated wrapper for calloc() and malloc() and similar. Bye, bearophile
Feb 07 2015
On Sun, Feb 08, 2015 at 12:39:39AM +0000, bearophile via Digitalmars-d wrote:Andrei Alexandrescu:[...] You mean something like this? T* malloc(T)() trusted { return cast(T*)malloc(T.sizeof); } struct MyStruct { int x, y, z; } void main() { auto p = malloc!MyStruct(); // Not sure how to make free() usable from safe, unless // we wrap the pointer returned by malloc(). free(p); } T -- Leather is waterproof. Ever see a cow with an umbrella?Such wrappers would allow safe code to use more C stdlib primitives.I'd also like a safer templated wrapper for calloc() and malloc() and similar.
Feb 07 2015
On 2/7/15 5:26 PM, H. S. Teoh via Digitalmars-d wrote:On Sun, Feb 08, 2015 at 12:39:39AM +0000, bearophile via Digitalmars-d wrote:I think that would go as follows: private system T[] mallocUninitializedArrayImpl(T)(size_t n) { auto p = malloc(n * T.sizeof); p || assert(0, "Not enough memory"); return (cast(T*) p)[0 .. n]; } trusted T[] mallocUninitializedArray(size_t n) if (!hasIndirections!T) { return mallocUninitializedArrayImpl!T(n); } system T[] mallocUninitializedArray(size_t n) if (hasIndirections!T) { return mallocUninitializedArrayImpl!T(n); } Similarly there'd be a mallocMinimallyInitializedArray that zeroes only pointers and is trusted for all types. Then we'd probably have a trusted callocArray that blasts zeros throughout. It's trusted because we know pointers are zeroes (an assumption somewhat not robust in theory but fine in practice). Then we'd have a mallocArray that allocates an array and initializes each element with .init.Andrei Alexandrescu:[...] You mean something like this? T* malloc(T)() trusted { return cast(T*)malloc(T.sizeof); }Such wrappers would allow safe code to use more C stdlib primitives.I'd also like a safer templated wrapper for calloc() and malloc() and similar.struct MyStruct { int x, y, z; } void main() { auto p = malloc!MyStruct(); // Not sure how to make free() usable from safe, unless // we wrap the pointer returned by malloc(). free(p); }Indeed we have no safe way to wrap free. Andrei
Feb 07 2015
On Sat, Feb 07, 2015 at 06:19:19PM -0800, Andrei Alexandrescu via Digitalmars-d wrote: [...]private system T[] mallocUninitializedArrayImpl(T)(size_t n) { auto p = malloc(n * T.sizeof); p || assert(0, "Not enough memory");This is a truly strange way of writing it... why not: assert(p !is null, "Not enough memory"); ?return (cast(T*) p)[0 .. n]; }T -- Tell me and I forget. Teach me and I remember. Involve me and I understand. -- Benjamin Franklin
Feb 07 2015
On Sunday, 8 February 2015 at 04:02:52 UTC, H. S. Teoh wrote:I think Andrei's version will remain in release builds, but yours will be elided.p || assert(0, "Not enough memory");This is a truly strange way of writing it... why not: assert(p !is null, "Not enough memory");
Feb 07 2015
On 2/7/15 8:00 PM, H. S. Teoh via Digitalmars-d wrote:On Sat, Feb 07, 2015 at 06:19:19PM -0800, Andrei Alexandrescu via Digitalmars-d wrote: [...]assert(0) is not removed in release mode. -- Andreiprivate system T[] mallocUninitializedArrayImpl(T)(size_t n) { auto p = malloc(n * T.sizeof); p || assert(0, "Not enough memory");This is a truly strange way of writing it... why not: assert(p !is null, "Not enough memory"); ?
Feb 07 2015
On Sat, Feb 07, 2015 at 08:15:05PM -0800, Andrei Alexandrescu via Digitalmars-d wrote:On 2/7/15 8:00 PM, H. S. Teoh via Digitalmars-d wrote:Ah, right. But shouldn't it be enforce instead of assert, then? :-P T -- In theory, software is implemented according to the design that has been carefully worked out beforehand. In practice, design documents are written after the fact to describe the sorry mess that has gone on before.On Sat, Feb 07, 2015 at 06:19:19PM -0800, Andrei Alexandrescu via Digitalmars-d wrote: [...]assert(0) is not removed in release mode. -- Andreiprivate system T[] mallocUninitializedArrayImpl(T)(size_t n) { auto p = malloc(n * T.sizeof); p || assert(0, "Not enough memory");This is a truly strange way of writing it... why not: assert(p !is null, "Not enough memory"); ?
Feb 07 2015
On 2015-02-08 at 03:19, Andrei Alexandrescu wrote:Indeed we have no safe way to wrap free.How about this to prevent double free: Wrapped malloc keeps a static thread-local lookup structure for successful allocations (if having to release memory from the same thread is an acceptable requirement). Wrapped free looks up the pointer in that lookup structure and, if found, frees memory, removes the lookup entry and sets the argument of the call to zero (if it was a pointer) or sets its length and ptr to zero (if it was a dynamic array). It's not completely safe, but for that GC would have to be used instead.
Feb 08 2015
On Sunday, 8 February 2015 at 12:43:38 UTC, FG wrote:On 2015-02-08 at 03:19, Andrei Alexandrescu wrote:I don't have any data, but I'd image most double-frees come from multiple references to the same data, not repeated calls to free on the same reference.Indeed we have no safe way to wrap free.How about this to prevent double free: Wrapped malloc keeps a static thread-local lookup structure for successful allocations (if having to release memory from the same thread is an acceptable requirement). Wrapped free looks up the pointer in that lookup structure and, if found, frees memory, removes the lookup entry and sets the argument of the call to zero (if it was a pointer) or sets its length and ptr to zero (if it was a dynamic array). It's not completely safe, but for that GC would have to be used instead.
Feb 08 2015
On 2/8/15 5:16 AM, John Colvin wrote:On Sunday, 8 February 2015 at 12:43:38 UTC, FG wrote:I think the same. In C++ circles zeroing the pointer after freeing is considering an antipattern - what with false sense of security etc. -- AndreiOn 2015-02-08 at 03:19, Andrei Alexandrescu wrote:I don't have any data, but I'd image most double-frees come from multiple references to the same data, not repeated calls to free on the same reference.Indeed we have no safe way to wrap free.How about this to prevent double free: Wrapped malloc keeps a static thread-local lookup structure for successful allocations (if having to release memory from the same thread is an acceptable requirement). Wrapped free looks up the pointer in that lookup structure and, if found, frees memory, removes the lookup entry and sets the argument of the call to zero (if it was a pointer) or sets its length and ptr to zero (if it was a dynamic array). It's not completely safe, but for that GC would have to be used instead.
Feb 08 2015
On 2/8/2015 8:32 AM, Andrei Alexandrescu wrote:I think the same. In C++ circles zeroing the pointer after freeing is considering an antipattern - what with false sense of security etc. -- AndreiWhat worked for me was using a free() that overwrote the free'd memory with 0xFEFEFEFE or something like that. Worked great for finding bugs. These days, valgrind does an even better job.
Feb 10 2015
On Sunday, 8 February 2015 at 12:43:38 UTC, FG wrote:On 2015-02-08 at 03:19, Andrei Alexandrescu wrote:A typical C debug library trick for when the money for a proper tool isn't available. -- PauloIndeed we have no safe way to wrap free.How about this to prevent double free: Wrapped malloc keeps a static thread-local lookup structure for successful allocations (if having to release memory from the same thread is an acceptable requirement). Wrapped free looks up the pointer in that lookup structure and, if found, frees memory, removes the lookup entry and sets the argument of the call to zero (if it was a pointer) or sets its length and ptr to zero (if it was a dynamic array). It's not completely safe, but for that GC would have to be used instead.
Feb 08 2015
On Saturday, 7 February 2015 at 23:50:55 UTC, Andrei Alexandrescu wrote:I was looking into ways to make core.stdc safer. That should be relatively easy to do by defining a few wrappers. For example: int setvbuf(FILE* stream, char* buf, int mode, size_t size); is unsafe because there's no relationship between buf and size. But this is fine: trusted int setvbuf(T)(FILE* stream, T[] buf, int mode) if (is(T == char) || is(T == byte) || is(T == ubyte)) { return setvbuf(stream, cast(char*) buf.ptr, mode, buf.length); } Another example is: int stat(in char*, stat_t*); which may start reading through random memory if the string is not zero-terminated. Again, the solution is here to ensure the string does have a terminating zero: trusted int stat(in char[] name, stat_t* p) { if (isZeroTerminated(name)) return stat(name.ptr, p); auto t = cast(char*) malloc(name.length + 1); scope(exit) free(t); memcpy(t, name.ptr, name.length); t[name.length] = 0; return stat(t, p); } Such wrappers would allow safe code to use more C stdlib primitives. The question is whether these wrappers are worth adding to core.stdc.stdio. Thanks, AndreiOne of the reasons why I use C functions is that I expect same behaviour from D code what I would expect from C. I don't think it is a good idea to make wrapper on top of them. Maybe you could say, "Hey, look, it just makes safer, that's all", but, hmm there are so many functions, and this wrapping process can go in many directions.
Feb 07 2015
On 2/7/15 7:52 PM, tcak wrote:One of the reasons why I use C functions is that I expect same behaviour from D code what I would expect from C. I don't think it is a good idea to make wrapper on top of them. Maybe you could say, "Hey, look, it just makes safer, that's all", but, hmm there are so many functions, and this wrapping process can go in many directions.Just looking at making them safe. Not all can be made safe btw. -- Andrei
Feb 07 2015
On Sat, Feb 07, 2015 at 08:14:39PM -0800, Andrei Alexandrescu via Digitalmars-d wrote:On 2/7/15 7:52 PM, tcak wrote:Come to think of it, is there any point in making malloc safe/ trusted at all? I don't think it's possible to make free() safe, so what's the purpose of making malloc callable from safe code? Unless you make a ref-counted wrapper of some sort around it, in which case you might as well use RefCounted instead. I thought about making the equivalent of auto_ptr, but unless you make it non-copyable (or only destructively copyable, and no pointer extraction is permitted), there's no way it can be truly safe. The only possible advantage we could gain is *type* safety by wrapping malloc in a type-safe way (i.e., don't expose void*). T -- Long, long ago, the ancient Chinese invented a device that lets them see through walls. It was called the "window".One of the reasons why I use C functions is that I expect same behaviour from D code what I would expect from C. I don't think it is a good idea to make wrapper on top of them. Maybe you could say, "Hey, look, it just makes safer, that's all", but, hmm there are so many functions, and this wrapping process can go in many directions.Just looking at making them safe. Not all can be made safe btw. -- Andrei
Feb 07 2015
On 2/7/15 8:21 PM, H. S. Teoh via Digitalmars-d wrote:Come to think of it, is there any point in making malloc safe/ trusted at all? I don't think it's possible to make free() safe, so what's the purpose of making malloc callable from safe code? Unless you make a ref-counted wrapper of some sort around it, in which case you might as well use RefCounted instead.Same goes about e.g. fopen vs. fclose. I'm thinking just of increasing the quantity of safe code. -- Andrei
Feb 07 2015
On Sunday, 8 February 2015 at 04:45:55 UTC, Andrei Alexandrescu wrote:On 2/7/15 8:21 PM, H. S. Teoh via Digitalmars-d wrote:So are you going for this: http://forum.dlang.org/thread/ovoarcbexpvrrceysnrs forum.dlang.org ?Come to think of it, is there any point in making malloc safe/ trusted at all? I don't think it's possible to make free() safe, so what's the purpose of making malloc callable from safe code? Unless you make a ref-counted wrapper of some sort around it, in which case you might as well use RefCounted instead.Same goes about e.g. fopen vs. fclose. I'm thinking just of increasing the quantity of safe code. -- Andrei
Feb 08 2015
H. S. Teoh:Come to think of it, is there any point in making malloc safe/ trusted at all?I am not asking for a trusted function. I'd like a system template wrapper for malloc/calloc/free that is safer than the C functions (safer because it's type-aware). Bye, bearophile
Feb 08 2015
come to think of it, is there any point in making malloc safe/ trusted at all? I don't think it's possible to make free() safe, so what's the purpose of making malloc callable from safe code?At least two programs, widely used by folks here, never release their memory. Those could be made safe.
Feb 08 2015
Am Sat, 07 Feb 2015 15:50:53 -0800 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:I was looking into ways to make core.stdc safer. That should be relatively easy to do by defining a few wrappers. For example:This might be a good idea, but it might also be more difficult than you think:int setvbuf(FILE* stream, char* buf, int mode, size_t size); is unsafe because there's no relationship between buf and size. But this is fine: trusted int setvbuf(T)(FILE* stream, T[] buf, int mode) if (is(T == char) || is(T == byte) || is(T == ubyte)) { return setvbuf(stream, cast(char*) buf.ptr, mode, buf.length); }This can still cause memory corruption if `buf` is GC-allocated. You'd have to pin the buffer which might not be easy in such a low-level wrapper. OTOH in a higher level wrapper (std.stdio.File) you can simply keep a reference to the buffer.Another example is: int stat(in char*, stat_t*); which may start reading through random memory if the string is not zero-terminated. Again, the solution is here to ensure the string does have a terminating zero: trusted int stat(in char[] name, stat_t* p) { if (isZeroTerminated(name)) return stat(name.ptr, p);How would you implement `isZeroTerminated` in a memory safe way? We have exactly the same problem in toStringz and nobody ever came up with a really safe solution. The best you could do is using special types for zero-terminated strings but that might be cumbersome to use.auto t = cast(char*) malloc(name.length + 1); scope(exit) free(t); memcpy(t, name.ptr, name.length); t[name.length] = 0; return stat(t, p); } Such wrappers would allow safe code to use more C stdlib primitives. The question is whether these wrappers are worth adding to core.stdc.stdio.That's the main question. There's only a limited amount of stdc functions which can be wrapped in a safe way and std.stdio etc. are already kind of a safe wrapper. And it's also important to get these wrappers right and make sure they don't introduce memory safety bugs.
Feb 08 2015
On 2/8/15 2:54 AM, Johannes Pfau wrote:Am Sat, 07 Feb 2015 15:50:53 -0800 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:Good point, thanks. Moving GCs didn't occur to me.trusted int setvbuf(T)(FILE* stream, T[] buf, int mode) if (is(T == char) || is(T == byte) || is(T == ubyte)) { return setvbuf(stream, cast(char*) buf.ptr, mode, buf.length); }This can still cause memory corruption if `buf` is GC-allocated. You'd have to pin the buffer which might not be easy in such a low-level wrapper. OTOH in a higher level wrapper (std.stdio.File) you can simply keep a reference to the buffer.I thought of a few things, nothing is 100% foolproof. But I'm not too worried - many of these functions issue system calls, and the cost of a malloc/free pulse is unlikely to be measurable. With opportunistic use of alloca it gets even better.trusted int stat(in char[] name, stat_t* p) { if (isZeroTerminated(name)) return stat(name.ptr, p);How would you implement `isZeroTerminated` in a memory safe way? We have exactly the same problem in toStringz and nobody ever came up with a really safe solution. The best you could do is using special types for zero-terminated strings but that might be cumbersome to use.I see it as increased opportunity to rely on simple manually checkable low-level functions, both in Phobos and outside it. It seems there's merit in that. Andreiauto t = cast(char*) malloc(name.length + 1); scope(exit) free(t); memcpy(t, name.ptr, name.length); t[name.length] = 0; return stat(t, p); } Such wrappers would allow safe code to use more C stdlib primitives. The question is whether these wrappers are worth adding to core.stdc.stdio.That's the main question. There's only a limited amount of stdc functions which can be wrapped in a safe way and std.stdio etc. are already kind of a safe wrapper. And it's also important to get these wrappers right and make sure they don't introduce memory safety bugs.
Feb 08 2015
On Saturday, 7 February 2015 at 23:50:55 UTC, Andrei Alexandrescu wrote:I was looking into ways to make core.stdc safer. That should be relatively easy to do by defining a few wrappers. For example: int setvbuf(FILE* stream, char* buf, int mode, size_t size); is unsafe because there's no relationship between buf and size. But this is fine: trusted int setvbuf(T)(FILE* stream, T[] buf, int mode) if (is(T == char) || is(T == byte) || is(T == ubyte)) { return setvbuf(stream, cast(char*) buf.ptr, mode, buf.length); } Another example is: int stat(in char*, stat_t*); which may start reading through random memory if the string is not zero-terminated. Again, the solution is here to ensure the string does have a terminating zero: trusted int stat(in char[] name, stat_t* p) { if (isZeroTerminated(name)) return stat(name.ptr, p); auto t = cast(char*) malloc(name.length + 1); scope(exit) free(t); memcpy(t, name.ptr, name.length); t[name.length] = 0; return stat(t, p); } Such wrappers would allow safe code to use more C stdlib primitives. The question is whether these wrappers are worth adding to core.stdc.stdio. Thanks, AndreiI think this is crucial if we want to keep actual Phobos sources easily review-able within your requirements. There is a good value in having `core.stdc` to map C headers 1-to-1 though. Would you consider separate `core.safestdc` package tree where such wrappers could be put on per need basis (duplicating tree structure of core.stdc modules internally)
Feb 09 2015
On 2/9/15 1:39 AM, Dicebot wrote:I think this is crucial if we want to keep actual Phobos sources easily review-able within your requirements. There is a good value in having `core.stdc` to map C headers 1-to-1 though. Would you consider separate `core.safestdc` package tree where such wrappers could be put on per need basis (duplicating tree structure of core.stdc modules internally)Walter opposes this on grounds of increased maintenance burdens. He points out (rightly imho) that we better focus on the higher-level primitives present in Phobos. -- Andrei
Feb 09 2015
On Monday, 9 February 2015 at 17:45:09 UTC, Andrei Alexandrescu wrote:On 2/9/15 1:39 AM, Dicebot wrote:I foresee certain stdc-based primitives being duplicated over and over again as nested functions - in scope of complying with "minimal trusted function that exposes safe API" rule.I think this is crucial if we want to keep actual Phobos sources easily review-able within your requirements. There is a good value in having `core.stdc` to map C headers 1-to-1 though. Would you consider separate `core.safestdc` package tree where such wrappers could be put on per need basis (duplicating tree structure of core.stdc modules internally)Walter opposes this on grounds of increased maintenance burdens. He points out (rightly imho) that we better focus on the higher-level primitives present in Phobos. -- Andrei
Feb 09 2015
On 2/9/15 9:47 AM, Dicebot wrote:I foresee certain stdc-based primitives being duplicated over and over again as nested functions - in scope of complying with "minimal trusted function that exposes safe API" rule.Would be nice to encapsulate them in Phobos on a case basis. -- Andrei
Feb 09 2015
On Monday, 9 February 2015 at 17:45:09 UTC, Andrei Alexandrescu wrote:On 2/9/15 1:39 AM, Dicebot wrote:Please don't waste your limited resource on this. When people use core.stdc they know what they are doing and take full responsibility of the consequence. If they want safety or nice APIs they wouldn't be using these functions in the first place.I think this is crucial if we want to keep actual Phobos sources easily review-able within your requirements. There is a good value in having `core.stdc` to map C headers 1-to-1 though. Would you consider separate `core.safestdc` package tree where such wrappers could be put on per need basis (duplicating tree structure of core.stdc modules internally)Walter opposes this on grounds of increased maintenance burdens. He points out (rightly imho) that we better focus on the higher-level primitives present in Phobos. -- Andrei
Feb 11 2015