digitalmars.D - memcpy, memset, memcmp and friends in the D Runtime
- Mike (24/24) Apr 07 2014 I'm trying to port the D Runtime to a bare metal platform. In
- Steven Schveighoffer (11/17) Apr 07 2014 I don't think it's convenience so much as being very experienced and tun...
- Mike (3/10) Apr 07 2014 Oh! that is SO much better. I should have known. Thank you.
- Orvid King (4/21) Apr 07 2014 I know that with DMD at least, memcpy, memset, and memcmp are all
- John Colvin (3/27) Apr 07 2014 What's this for then?
- Mike (12/20) Apr 09 2014 I received a very informative answer on D.learn
- Daniel Murphy (3/6) Apr 09 2014 What platform doesn't have a C library? And you can always cast arrays ...
- Mike (23/30) Apr 09 2014 My custom, bare-metal, D-only platform has no C library, nor will
- monarch_dodra (8/40) Apr 09 2014 I think arguably, there should be D equivalents of the C runtime
- Daniel Murphy (6/10) Apr 09 2014 The utility of a 'slightly safer' layer between the C functions and the
- Dmitry Olshansky (12/22) Apr 10 2014 Pros:
- Daniel Murphy (6/17) Apr 10 2014 I don't know why you'd ever use memcpy/memset when you have:
- monarch_dodra (2/20) Apr 10 2014 Because those are assignments.
- Daniel Murphy (4/5) Apr 10 2014 You mean they might call postblit etc? I'd still rather cast to ubyte[]...
- monarch_dodra (4/9) Apr 10 2014 Yeah, but you can't cast to ubyte with CTFE. So you still end up
- Dmitry Olshansky (10/26) Apr 10 2014 Calls postblit/dtor etc.? What I if I need bitwise assignment? (which is...
- Steven Schveighoffer (8/35) Apr 10 2014 is =
- Brad Anderson (5/52) Apr 11 2014 This list of secure alternatives to C functions Microsoft has
- Brad Anderson (3/7) Apr 11 2014 Oops, broken link.
- monarch_dodra (5/9) Apr 11 2014 I just wrote this as a rough draft. I'll make "formal"
- Brad Anderson (3/13) Apr 11 2014 I know :). We #d IRC users see all! (Vladimir Panteleev's DFeed
- Dmitry Olshansky (10/19) Apr 12 2014 Some thoughts.
- Daniel Murphy (21/39) Apr 09 2014 druntime and phobos both assume a c library is present, and this is a ve...
- Mike (9/20) Apr 09 2014 Thank you for your thoughtful answer (although I'm not ready to
- Daniel Murphy (12/17) Apr 09 2014 From a principled D design point of view, only targets with a C library ...
I'm trying to port the D Runtime to a bare metal platform. In the short time I've been doing bare metal work, I've come to rely on two very fundamental methods: memcpy and memset. In my D runtime, however, I'm not using the C standard library; I'm doing it all in D. It seems the D Runtime and Phobos also often make use of memcpy and memset, but they do so through core.stdc which doesn't appear to be part of the D Runtime. Was this just a matter of convenience or thoughtful design? I'm guessing the former based on the fact that core.stdc is not in the official language reference. So, I would like to know what the design philosophy is for fundamental memory utilities like memcpy, memset, memcmp etc..., because I need them and would like to implement harmony with D's design. * Were these methods intentionally omitted from the D Runtime, or just taken for granted since they were so conveniently available in the C standard library? * Should they be added to core.memory or some other module and exposed as official, supported, methods of the D Runtime (hopefully changing their names)? Please advise. Thanks, Mike
Apr 07 2014
On Mon, 07 Apr 2014 07:51:38 -0400, Mike <none none.com> wrote:* Were these methods intentionally omitted from the D Runtime, or just taken for granted since they were so conveniently available in the C standard library?I don't think it's convenience so much as being very experienced and tuned functions.* Should they be added to core.memory or some other module and exposed as official, supported, methods of the D Runtime (hopefully changing their names)?I think using using D's safer syntax of a[] = b[] and a[] = 0; Is much preferable to using memcpy/memset directly. They should do the "Right Thing", including calling memcpy/memset when correct. -Steve
Apr 07 2014
On Monday, 7 April 2014 at 12:38:55 UTC, Steven Schveighoffer wrote:I think using using D's safer syntax of a[] = b[] and a[] = 0; Is much preferable to using memcpy/memset directly. They should do the "Right Thing", including calling memcpy/memset when correct.Oh! that is SO much better. I should have known. Thank you.
Apr 07 2014
I know that with DMD at least, memcpy, memset, and memcmp are all intrinsics, meaning the compiler emits their implementations directly, thus why there isn't a body for them in druntime. On 4/7/14, Mike <none none.com> wrote:On Monday, 7 April 2014 at 12:38:55 UTC, Steven Schveighoffer wrote:I think using using D's safer syntax of a[] = b[] and a[] = 0; Is much preferable to using memcpy/memset directly. They should do the "Right Thing", including calling memcpy/memset when correct.Oh! that is SO much better. I should have known. Thank you.
Apr 07 2014
On Monday, 7 April 2014 at 13:17:37 UTC, Orvid King wrote:I know that with DMD at least, memcpy, memset, and memcmp are all intrinsics, meaning the compiler emits their implementations directly, thus why there isn't a body for them in druntime. On 4/7/14, Mike <none none.com> wrote:What's this for then? https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memset.dOn Monday, 7 April 2014 at 12:38:55 UTC, Steven Schveighoffer wrote:I think using using D's safer syntax of a[] = b[] and a[] = 0; Is much preferable to using memcpy/memset directly. They should do the "Right Thing", including calling memcpy/memset when correct.Oh! that is SO much better. I should have known. Thank you.
Apr 07 2014
On Monday, 7 April 2014 at 14:05:38 UTC, John Colvin wrote:What's this for then? https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memset.dThose appear to be the implementations of the a[] = x syntax as compiler builtins and provide even more of a reason to it. They are internal to the runtime. I was wondering why such methods aren't part of the runtime's interface and I believe I received the answer. I'm still wondering if the usage of memcpy in phobos could be replaced with something more 'D'. a[] = b[] could always be translated to a call to memcpy in the platform's runtime port. Maybe I'll do some tests and submit a pull request.
Apr 07 2014
On Monday, 7 April 2014 at 22:42:54 UTC, Mike wrote:On Monday, 7 April 2014 at 14:05:38 UTC, John Colvin wrote:Apologies, John. I see now you were responding to Orvid, not me.What's this for then? https://github.com/D-Programming-Language/druntime/blob/master/src/rt/memset.dThose appear to be the implementations of the a[] = x syntax as compiler builtins and provide even more of a reason to it. They are internal to the runtime. I was wondering why such methods aren't part of the runtime's interface and I believe I received the answer. I'm still wondering if the usage of memcpy in phobos could be replaced with something more 'D'. a[] = b[] could always be translated to a call to memcpy in the platform's runtime port. Maybe I'll do some tests and submit a pull request.
Apr 07 2014
On Monday, 7 April 2014 at 12:38:55 UTC, Steven Schveighoffer wrote:I think using using D's safer syntax of a[] = b[] and a[] = 0; Is much preferable to using memcpy/memset directly. They should do the "Right Thing", including calling memcpy/memset when correct. -SteveI received a very informative answer on D.learn (http://forum.dlang.org/post/xdnbfbdvruevjgqvbhfb forum.dlang.org) that describes why a function that copies memory still may be needed: "Depending on the type being copied, "a[] = b[]" is not straight up "memory copy" it's the actual full assignment, with postblit and all. That's why the call we want is *really* memcpy." So, my question remains: If I'm porting D to a platform that has no C library, shouldn't a public, supported function that copies memory be added in the D Runtime?
Apr 09 2014
"Mike" wrote in message news:iflykgpsrmdbfhuwzqtp forum.dlang.org...So, my question remains: If I'm porting D to a platform that has no C library, shouldn't a public, supported function that copies memory be added in the D Runtime?What platform doesn't have a C library? And you can always cast arrays to ubyte[] before assigning to avoid postblit.
Apr 09 2014
On Wednesday, 9 April 2014 at 11:50:13 UTC, Daniel Murphy wrote:"Mike" wrote in message news:iflykgpsrmdbfhuwzqtp forum.dlang.org...My custom, bare-metal, D-only platform has no C library, nor will it ever. And, Yes, I know I can cast, but should I? memcpy is used quite often in Phobos. Here's a small sample from std.array: void trustedMemcopy(T[] dest, T[] src) trusted { assert(src.length == dest.length); if (!__ctfe) memcpy(dest.ptr, src.ptr, src.length * T.sizeof); else { dest[] = src[]; } } You could ask the same question there, "Why isn't a cast used instead?" and remove the dependency on the C library. So far, it seems D has only been used on one subset of computer systems that happen to have a C library, but some of us our trying to break some new ground with D. I'm challenging the "D is only useful on PCs with a C library" bias and looking to see if D has the will to stand on its own two feet instead of leaning on C.So, my question remains: If I'm porting D to a platform that has no C library, shouldn't a public, supported function that copies memory be added in the D Runtime?What platform doesn't have a C library? And you can always cast arrays to ubyte[] before assigning to avoid postblit.
Apr 09 2014
On Wednesday, 9 April 2014 at 12:12:35 UTC, Mike wrote:On Wednesday, 9 April 2014 at 11:50:13 UTC, Daniel Murphy wrote:I think arguably, there should be D equivalents of the C runtime functions, if only for the added safety of slices (for example, memchr could be 100% certifiably safe), and to avoid the rampant deduplication of things as in the above, because of CTFE. These may or may not use the C-runtime library. But at that point, it would become a "druntime implementation detail", rather than rampant usage throughout phobos."Mike" wrote in message news:iflykgpsrmdbfhuwzqtp forum.dlang.org...My custom, bare-metal, D-only platform has no C library, nor will it ever. And, Yes, I know I can cast, but should I? memcpy is used quite often in Phobos. Here's a small sample from std.array: void trustedMemcopy(T[] dest, T[] src) trusted { assert(src.length == dest.length); if (!__ctfe) memcpy(dest.ptr, src.ptr, src.length * T.sizeof); else { dest[] = src[]; } } You could ask the same question there, "Why isn't a cast used instead?" and remove the dependency on the C library. So far, it seems D has only been used on one subset of computer systems that happen to have a C library, but some of us our trying to break some new ground with D. I'm challenging the "D is only useful on PCs with a C library" bias and looking to see if D has the will to stand on its own two feet instead of leaning on C.So, my question remains: If I'm porting D to a platform that has no C library, shouldn't a public, supported function that copies memory be added in the D Runtime?What platform doesn't have a C library? And you can always cast arrays to ubyte[] before assigning to avoid postblit.
Apr 09 2014
"monarch_dodra" wrote in message news:pdzhmmnjxclrjtkgupoz forum.dlang.org...I think arguably, there should be D equivalents of the C runtime functions, if only for the added safety of slices (for example, memchr could be 100% certifiably safe), and to avoid the rampant deduplication of things as in the above, because of CTFE.The utility of a 'slightly safer' layer between the C functions and the phobos range-based functions is questionable IMO. eg in cases where memcpy on arrays is safe, phobos' copy would likely optimize to the exact same code.
Apr 09 2014
09-Apr-2014 16:56, Daniel Murphy пишет:"monarch_dodra" wrote in message news:pdzhmmnjxclrjtkgupoz forum.dlang.org...Pros: 1. No extra scuffolding for CTFE. 2. Type-safe 3. Optional (non-release, etc.) bounds-checking. 4. Less error prone interface, e.g. memset is a known source of bugs in C, even for tried and true code bases. 5. More flexible, see e.g. memchr as an example. Cons: A set of primitives to learn. -- Dmitry OlshanskyI think arguably, there should be D equivalents of the C runtime functions, if only for the added safety of slices (for example, memchr could be 100% certifiably safe), and to avoid the rampant deduplication of things as in the above, because of CTFE.The utility of a 'slightly safer' layer between the C functions and the phobos range-based functions is questionable IMO. eg in cases where memcpy on arrays is safe, phobos' copy would likely optimize to the exact same code.
Apr 10 2014
"Dmitry Olshansky" wrote in message news:li5lfd$203s$1 digitalmars.com...Pros: 1. No extra scuffolding for CTFE. 2. Type-safe 3. Optional (non-release, etc.) bounds-checking. 4. Less error prone interface, e.g. memset is a known source of bugs in C, even for tried and true code bases. 5. More flexible, see e.g. memchr as an example. Cons: A set of primitives to learn.I don't know why you'd ever use memcpy/memset when you have: arr[] = 0; foreach(ref v; arr) v = 0; etc
Apr 10 2014
On Thursday, 10 April 2014 at 12:22:37 UTC, Daniel Murphy wrote:"Dmitry Olshansky" wrote in message news:li5lfd$203s$1 digitalmars.com...Because those are assignments.Pros: 1. No extra scuffolding for CTFE. 2. Type-safe 3. Optional (non-release, etc.) bounds-checking. 4. Less error prone interface, e.g. memset is a known source of bugs in C, even for tried and true code bases. 5. More flexible, see e.g. memchr as an example. Cons: A set of primitives to learn.I don't know why you'd ever use memcpy/memset when you have: arr[] = 0; foreach(ref v; arr) v = 0; etc
Apr 10 2014
"monarch_dodra" wrote in message news:obksandhskaldxplghme forum.dlang.org...Because those are assignments.You mean they might call postblit etc? I'd still rather cast to ubyte[] before assigning than call memset.
Apr 10 2014
On Thursday, 10 April 2014 at 16:59:27 UTC, Daniel Murphy wrote:"monarch_dodra" wrote in message news:obksandhskaldxplghme forum.dlang.org...Yeah, but you can't cast to ubyte with CTFE. So you still end up pulling out boilerplate code "at call site", when you could just "deal with it" it a generic way in a named function.Because those are assignments.You mean they might call postblit etc? I'd still rather cast to ubyte[] before assigning than call memset.
Apr 10 2014
10-Apr-2014 16:22, Daniel Murphy пишет:"Dmitry Olshansky" wrote in message news:li5lfd$203s$1 digitalmars.com...Calls postblit/dtor etc.? What I if I need bitwise assignment? (which is what memcpy is all about) Yes, and with CTFE please. See even std.algo.move and std.algo.swap...Pros: 1. No extra scuffolding for CTFE. 2. Type-safe 3. Optional (non-release, etc.) bounds-checking. 4. Less error prone interface, e.g. memset is a known source of bugs in C, even for tried and true code bases. 5. More flexible, see e.g. memchr as an example. Cons: A set of primitives to learn.I don't know why you'd ever use memcpy/memset when you have: arr[] = 0;foreach(ref v; arr) v = 0;Same but also sucks performance-wise. There is a reason Phobos is littered with carefully laid out special cases around C-runtime stuff. -- Dmitry Olshansky
Apr 10 2014
On Thu, 10 Apr 2014 16:13:31 -0400, Dmitry Olshansky = <dmitry.olsh gmail.com> wrote:10-Apr-2014 16:22, Daniel Murphy =D0=BF=D0=B8=D1=88=D0=B5=D1=82:"Dmitry Olshansky" wrote in message =news:li5lfd$203s$1 digitalmars.com...Pros: 1. No extra scuffolding for CTFE. 2. Type-safe 3. Optional (non-release, etc.) bounds-checking. 4. Less error prone interface, e.g. memset is a known source of bugs=is =Calls postblit/dtor etc.? What I if I need bitwise assignment? (which =in C, even for tried and true code bases. 5. More flexible, see e.g. memchr as an example. Cons: A set of primitives to learn.I don't know why you'd ever use memcpy/memset when you have: arr[] =3D 0;what memcpy is all about) Yes, and with CTFE please. See even std.algo.move and std.algo.swap...foreach(ref v; arr) v =3D 0;Same but also sucks performance-wise. There is a reason Phobos is littered with carefully laid out special =cases around C-runtime stuff.I think a simple set of functions would be sufficient. It wouldn't even = be = that complex. core.memops -Steve
Apr 10 2014
On Wednesday, 9 April 2014 at 12:38:20 UTC, monarch_dodra wrote:On Wednesday, 9 April 2014 at 12:12:35 UTC, Mike wrote:This list of secure alternatives to C functions Microsoft has would probably be a good source of inspiration when deciding what functions should be considered for reimplementation: http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7ClistOn Wednesday, 9 April 2014 at 11:50:13 UTC, Daniel Murphy wrote:I think arguably, there should be D equivalents of the C runtime functions, if only for the added safety of slices (for example, memchr could be 100% certifiably safe), and to avoid the rampant deduplication of things as in the above, because of CTFE. These may or may not use the C-runtime library. But at that point, it would become a "druntime implementation detail", rather than rampant usage throughout phobos."Mike" wrote in message news:iflykgpsrmdbfhuwzqtp forum.dlang.org...My custom, bare-metal, D-only platform has no C library, nor will it ever. And, Yes, I know I can cast, but should I? memcpy is used quite often in Phobos. Here's a small sample from std.array: void trustedMemcopy(T[] dest, T[] src) trusted { assert(src.length == dest.length); if (!__ctfe) memcpy(dest.ptr, src.ptr, src.length * T.sizeof); else { dest[] = src[]; } } You could ask the same question there, "Why isn't a cast used instead?" and remove the dependency on the C library. So far, it seems D has only been used on one subset of computer systems that happen to have a C library, but some of us our trying to break some new ground with D. I'm challenging the "D is only useful on PCs with a C library" bias and looking to see if D has the will to stand on its own two feet instead of leaning on C.So, my question remains: If I'm porting D to a platform that has no C library, shouldn't a public, supported function that copies memory be added in the D Runtime?What platform doesn't have a C library? And you can always cast arrays to ubyte[] before assigning to avoid postblit.
Apr 11 2014
On Friday, 11 April 2014 at 20:17:35 UTC, Brad Anderson wrote:This list of secure alternatives to C functions Microsoft has would probably be a good source of inspiration when deciding what functions should be considered for reimplementation: http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7ClistOops, broken link. http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx
Apr 11 2014
On Friday, 11 April 2014 at 20:17:35 UTC, Brad Anderson wrote:This list of secure alternatives to C functions Microsoft has would probably be a good source of inspiration when deciding what functions should be considered for reimplementation: http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7ClistI just wrote this as a rough draft. I'll make "formal" announcement for it sometime soon, if initial feedback is positive. http://wiki.dlang.org/DIP59
Apr 11 2014
On Friday, 11 April 2014 at 20:25:21 UTC, monarch_dodra wrote:On Friday, 11 April 2014 at 20:17:35 UTC, Brad Anderson wrote:I know :). We #d IRC users see all! (Vladimir Panteleev's DFeed bot announces basically everything going on everywhere in D).This list of secure alternatives to C functions Microsoft has would probably be a good source of inspiration when deciding what functions should be considered for reimplementation: http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7ClistI just wrote this as a rough draft. I'll make "formal" announcement for it sometime soon, if initial feedback is positive. http://wiki.dlang.org/DIP59
Apr 11 2014
12-Apr-2014 00:25, monarch_dodra пишет:On Friday, 11 April 2014 at 20:17:35 UTC, Brad Anderson wrote:Some thoughts. Provide only bitwise comparison, search etc. It's the only case that can reliably benefit from SIMD or hand-written ASM, has problem with CTFE support, etc. Then requiring that T be a POD or built-in type is fine IMHO. Maybe go for copy ---> rawCopy, find --> rawFind (i.e. model after std.algorithm) rather then mimicking C functions. This way we are open to a superset of C-runtime ;) -- Dmitry OlshanskyThis list of secure alternatives to C functions Microsoft has would probably be a good source of inspiration when deciding what functions should be considered for reimplementation: http://msdn.microsoft.com/en-us/library/wd3wzwts.aspx%7C%7ClistI just wrote this as a rough draft. I'll make "formal" announcement for it sometime soon, if initial feedback is positive. http://wiki.dlang.org/DIP59
Apr 12 2014
"Mike" wrote in message news:nxretodrkqizyiukwpze forum.dlang.org...My custom, bare-metal, D-only platform has no C library, nor will it ever. And, Yes, I know I can cast, but should I?Ah.So far, it seems D has only been used on one subset of computer systems that happen to have a C library, but some of us our trying to break some new ground with D. I'm challenging the "D is only useful on PCs with a C library" bias and looking to see if D has the will to stand on its own two feet instead of leaning on C.druntime and phobos both assume a c library is present, and this is a very safe assumption to make. It's like assuming that bytes are 8 bits - you can certainly create platforms where this is true but those uses are not worth worrying about. For a bare metal runtime that doesn't support C, it makes sense to only have a single runtime and add all the needed D and C runtime functions there. So:It seems the D Runtime and Phobos also often make use of memcpy and memset, but they do so through core.stdc which doesn't appear to be part of the D Runtime. Was this just a matter of convenience or thoughtful design? I'm guessing the former based on the fact that core.stdc is not in the official language reference.I don't think you can read that much into the lack of documentation. I think it's a combination of "these are already documented on other sites" + "these are discourages" + "copying documentation has licensing issues". The interface to the c library is a part of druntime and here to stay.* Were these methods intentionally omitted from the D Runtime, or just taken for granted since they were so conveniently available in the C standard library?They were _not_ omitted from druntime, they were exposed though core.stdc. (and other places before that) An implementation in druntime was not necessary because all platforms druntime targets provide a c runtime.* Should they be added to core.memory or some other module and exposed as official, supported, methods of the D Runtime (hopefully changing their names)?No, they are already exposed via core.stdc. So, if you want to support the full interface that druntime does, you will need to implement these functions somewhere. There is no point in adding a second interface to them in upstream druntime.
Apr 09 2014
On Wednesday, 9 April 2014 at 12:50:23 UTC, Daniel Murphy wrote:Thank you for your thoughtful answer (although I'm not ready to accept it just yet :-) Let me ask you this: From a principled D design point of view, is core.stdc an interface only for ports with a C library, an implementation detail of the current druntime that phobos is circumventing, or an official, supported interface that all complying ports must implement? If the latter, why is it non-existing in the library reference?* Were these methods intentionally omitted from the D Runtime, or just taken for granted since they were so conveniently available in the C standard library?They were _not_ omitted from druntime, they were exposed though core.stdc. (and other places before that) An implementation in druntime was not necessary because all platforms druntime targets provide a c runtime.* Should they be added to core.memory or some other module and exposed as official, supported, methods of the D Runtime (hopefully changing their names)?No, they are already exposed via core.stdc.
Apr 09 2014
"Mike" wrote in message news:ycvyulqzunbsuweipais forum.dlang.org...Let me ask you this: From a principled D design point of view, is core.stdc an interface only for ports with a C library, an implementation detail of the current druntime that phobos is circumventing, or an official, supported interface that all complying ports must implement?From a principled D design point of view, only targets with a C library are supported. core.stdc is part of the public interface of druntime (as is everything in core) and not an implementation. There is no real definition in D for what a 'complying port' is, but many D applications and libraries will not run if you do not implement the commonly-used parts of core.stdc. Full coverage is likely unrealistic for a bare-metal implementation.If the latter, why is it non-existing in the library reference?Because there is no ddoc for these files, and nobody has gone to the effort to make them show up on the website. Whatever the original intention, enough code using them exists that removing them will never be an option. They probably should just link to the official c-lib documentation.
Apr 09 2014