digitalmars.D - safe and null dereferencing
- Steven Schveighoffer (48/48) Jul 27 2017 Inside the thread for adding @safe/@trusted attributes to OS functions,
- Adrian Matoga (12/65) Jul 27 2017 Why can't we just make the compiler insert null checks in @safe
- H. S. Teoh via Digitalmars-d (16/25) Jul 27 2017 Because not inserting null checks is a sacred cow we inherited from the
- Adrian Matoga (2/12) Jul 27 2017 That's exactly what I thought.
- Marco Leise (8/22) Jul 27 2017 A typical non-synthetic worst case candidate should be in the
- Steven Schveighoffer (10/20) Jul 27 2017 The rationale from Walter has always been that the hardware is already
- Steven Schveighoffer (3/5) Jul 27 2017 that we *should* handle this situation.
- H. S. Teoh via Digitalmars-d (21/28) Jul 27 2017 [...]
- Steven Schveighoffer (6/24) Jul 27 2017 Very true. You wouldn't want to store anything there as any @safe code
- Moritz Maxeiner (7/26) Jul 27 2017 The point of the example was to show that exploiting the "null
- Jonathan M Davis via Digitalmars-d (21/29) Jul 27 2017 I see no problem whatsoever requiring that the platform segfaults when y...
- Steven Schveighoffer (5/17) Jul 27 2017 Yes: https://issues.dlang.org/show_bug.cgi?id=5176
- Moritz Maxeiner (6/22) Jul 27 2017 Except that when someone gets (root) access to any modern Linux
- Steven Schveighoffer (9/33) Jul 27 2017 Well, let's not forget that the services should not be dereferencing
- Moritz Maxeiner (10/18) Jul 27 2017 Of course, but statistically speaking, all software is buggy so
- H. S. Teoh via Digitalmars-d (7/18) Jul 27 2017 If someone malicious has root access to your server, you already have
- Moritz Maxeiner (9/29) Jul 27 2017 That depends on how valuable you are as a target, how hard it was
- H. S. Teoh via Digitalmars-d (10/21) Jul 27 2017 Yes, and therefore "you already have much bigger things to worry about
- Moritz Maxeiner (4/13) Jul 28 2017 Sure, but this kind of subversive attack *keeps* you *the
- Jonathan M Davis via Digitalmars-d (14/29) Jul 27 2017 Agreed. And Safe D has never made any promises about denial of service
Inside the thread for adding safe/ trusted attributes to OS functions, it has come to light that safe has conflicting rules. For the definition of safe, it says: "Safe functions are functions that are statically checked to exhibit no possibility of undefined behavior." In the definition of trusted, it says: "Trusted functions are guaranteed by the programmer to not exhibit any undefined behavior if called by a safe function." Yet, safe functions allow dereferencing of null pointers. Example: void foo() safe { int *x; *x = 5; } There are various places on the forum where Walter argues that null pointer dereferencing should cause a segmentation fault (or crash) and is checked by the hardware/OS. Therefore, checking for null pointers before any dereferencing would be a waste of cycles. However, there do exist places where dereferencing null may NOT cause a segmentation fault. For example, see this post by Moritz Maxeiner: https://forum.dlang.org/post/udkdqogtrvanhbotdoik forum.dlang.org In such cases, the compiled program can have no knowledge that the zero page is mapped somehow. There is no way to prevent it, or guarantee it during compilation. It's also worth noting that C/C++ identifies null dereferencing as undefined behavior. So if we are being completely pedantic, we could say that no C/C++ code could be marked safe if there is a possibility that a null pointer would be dereferenced. The way I see it, we have 2 options. First, we can disallow null pointer dereferencing in safe code. This would be hugely disruptive. We may not have to instrument all safe code with null checks, we could do it with flow analysis, and assuming that all pointers passed into a safe function are not null. But it would likely disallow a lot of existing safe code. The other option is to explicitly state what happens in such cases. I would opt for this second option, as the likelihood of these situations is very low. If we were to update the spec to take this into account, how would it look? A possibility: " safe D does not support platforms or processes where dereferencing a null pointer does not crash the program. In such situations, dereferencing null is not defined, and safe code will not prevent this from happening." In terms of not marking C/C++ code safe, I am not convinced we need to go that far, but it's not as horrible a prospect as having to unmark D safe code that might dereference null. Thoughts? -Steve
Jul 27 2017
On Thursday, 27 July 2017 at 15:03:02 UTC, Steven Schveighoffer wrote:Inside the thread for adding safe/ trusted attributes to OS functions, it has come to light that safe has conflicting rules. For the definition of safe, it says: "Safe functions are functions that are statically checked to exhibit no possibility of undefined behavior." In the definition of trusted, it says: "Trusted functions are guaranteed by the programmer to not exhibit any undefined behavior if called by a safe function." Yet, safe functions allow dereferencing of null pointers. Example: void foo() safe { int *x; *x = 5; } There are various places on the forum where Walter argues that null pointer dereferencing should cause a segmentation fault (or crash) and is checked by the hardware/OS. Therefore, checking for null pointers before any dereferencing would be a waste of cycles. However, there do exist places where dereferencing null may NOT cause a segmentation fault. For example, see this post by Moritz Maxeiner: https://forum.dlang.org/post/udkdqogtrvanhbotdoik forum.dlang.org In such cases, the compiled program can have no knowledge that the zero page is mapped somehow. There is no way to prevent it, or guarantee it during compilation. It's also worth noting that C/C++ identifies null dereferencing as undefined behavior. So if we are being completely pedantic, we could say that no C/C++ code could be marked safe if there is a possibility that a null pointer would be dereferenced. The way I see it, we have 2 options. First, we can disallow null pointer dereferencing in safe code. This would be hugely disruptive. We may not have to instrument all safe code with null checks, we could do it with flow analysis, and assuming that all pointers passed into a safe function are not null. But it would likely disallow a lot of existing safe code. The other option is to explicitly state what happens in such cases. I would opt for this second option, as the likelihood of these situations is very low. If we were to update the spec to take this into account, how would it look? A possibility: " safe D does not support platforms or processes where dereferencing a null pointer does not crash the program. In such situations, dereferencing null is not defined, and safe code will not prevent this from happening." In terms of not marking C/C++ code safe, I am not convinced we need to go that far, but it's not as horrible a prospect as having to unmark D safe code that might dereference null. Thoughts? -SteveWhy can't we just make the compiler insert null checks in safe code? We can afford bounds checking even in system -O -release. C++ can afford null check upon executing an std::function. The pointer would most likely be in a register anyway, and the conditional branch would almost always not be taken, so the cost of that check would be barely measurable. Moreover, the compiler can elide the check e.g. if the access via pointer is made in a loop in which the pointer doesn't change. And if you prove that this tiny little check ruins performance of your code, there's trusted to help you.
Jul 27 2017
On Thu, Jul 27, 2017 at 05:33:22PM +0000, Adrian Matoga via Digitalmars-d wrote: [...]Why can't we just make the compiler insert null checks in safe code?Because not inserting null checks is a sacred cow we inherited from the C/C++ days of POOP (premature optimization oriented programming), and we are loathe to slaughter it. :-P We should seriously take some measurements of this in a large D project to determine whether or not inserting null checks actually makes a significant difference in performance.We can afford bounds checking even in system -O -release. C++ can afford null check upon executing an std::function. The pointer would most likely be in a register anyway, and the conditional branch would almost always not be taken, so the cost of that check would be barely measurable. Moreover, the compiler can elide the check e.g. if the access via pointer is made in a loop in which the pointer doesn't change. And if you prove that this tiny little check ruins performance of your code, there's trusted to help you.The compiler can (and should, if it doesn't already) also propagate non-nullness (ala VRP) as part of its dataflow analysis, so that once a pointer has been established to be non-null, all subsequent checks of that pointer can be elided (until the next assignment to the pointer, of course). T -- Public parking: euphemism for paid parking. -- Flora
Jul 27 2017
On Thursday, 27 July 2017 at 17:43:17 UTC, H. S. Teoh wrote:On Thu, Jul 27, 2017 at 05:33:22PM +0000, Adrian Matoga via Digitalmars-d wrote: [...]That's exactly what I thought.Why can't we just make the compiler insert null checks in safe code?Because not inserting null checks is a sacred cow we inherited from the C/C++ days of POOP (premature optimization oriented programming), and we are loathe to slaughter it. :-P We should seriously take some measurements of this in a large D project to determine whether or not inserting null checks actually makes a significant difference in performance.
Jul 27 2017
Am Thu, 27 Jul 2017 17:59:41 +0000 schrieb Adrian Matoga <dlang.spam matoga.info>:On Thursday, 27 July 2017 at 17:43:17 UTC, H. S. Teoh wrote:A typical non-synthetic worst case candidate should be in the tests that would invoke a lot of null checks. (Could be a function call at first to count checks per run of executable and pick a good project.) -- MarcoOn Thu, Jul 27, 2017 at 05:33:22PM +0000, Adrian Matoga via Digitalmars-d wrote: [...]That's exactly what I thought.Why can't we just make the compiler insert null checks in safe code?Because not inserting null checks is a sacred cow we inherited from the C/C++ days of POOP (premature optimization oriented programming), and we are loathe to slaughter it. :-P We should seriously take some measurements of this in a large D project to determine whether or not inserting null checks actually makes a significant difference in performance.
Jul 27 2017
On 7/27/17 1:33 PM, Adrian Matoga wrote:Why can't we just make the compiler insert null checks in safe code? We can afford bounds checking even in system -O -release. C++ can afford null check upon executing an std::function. The pointer would most likely be in a register anyway, and the conditional branch would almost always not be taken, so the cost of that check would be barely measurable. Moreover, the compiler can elide the check e.g. if the access via pointer is made in a loop in which the pointer doesn't change. And if you prove that this tiny little check ruins performance of your code, there's trusted to help you.The rationale from Walter has always been that the hardware is already doing this for us. I was always under the assumption that D only supported environments/systems where this happens. But technically there's nothing in the spec to require it. And it does seem apparent that we handle this situation. This question/query is asking whether we should amend the spec with (what I think is) Walter's view, or if we should change the compiler to insert the checks. -Steve
Jul 27 2017
On 7/27/17 2:09 PM, Steven Schveighoffer wrote:there's nothing in the spec to require it. And it does seem apparent that we handle this situation.that we *should* handle this situation. -Steve
Jul 27 2017
On Thu, Jul 27, 2017 at 11:03:02AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...]However, there do exist places where dereferencing null may NOT cause a segmentation fault. For example, see this post by Moritz Maxeiner: https://forum.dlang.org/post/udkdqogtrvanhbotdoik forum.dlang.org In such cases, the compiled program can have no knowledge that the zero page is mapped somehow. There is no way to prevent it, or guarantee it during compilation.[...] There is one flaw with Moritz's example: if the zero page is mapped somehow, that means 0 is potentially a valid address of a variable, and therefore checking for null is basically not only useless but wrong: a null check of the address of this variable will fail, yet the pointer is actually pointing at a valid address that just happens to be 0. IOW, if the zero page is mapped, we're *already* screwed anyway, might as well just give up now. One workaround for this is to redefine a null pointer as size_t.max (i.e., all bits set) instead of 0. It's far less likely for a valid address to be size_t.max than for 0 to be a valid address in a system where the zero page is mappable (due to alignment issues, the only possibility is if you have a ubyte* pointing to data stored at the address size_t.max, whereas address 0 can be a valid address for any data type). However, this will break basically *all* code out there in C/C++/D land, so I don't see it ever happening in this lifetime. T -- Those who've learned LaTeX swear by it. Those who are learning LaTeX swear at it. -- Pete Bleackley
Jul 27 2017
On 7/27/17 1:52 PM, H. S. Teoh via Digitalmars-d wrote:On Thu, Jul 27, 2017 at 11:03:02AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...]Very true. You wouldn't want to store anything there as any safe code could easily get a pointer to that data at any time! Either way, the guarantees of safe go out the window if dereferencing null is not a crashing error. -SteveHowever, there do exist places where dereferencing null may NOT cause a segmentation fault. For example, see this post by Moritz Maxeiner: https://forum.dlang.org/post/udkdqogtrvanhbotdoik forum.dlang.org In such cases, the compiled program can have no knowledge that the zero page is mapped somehow. There is no way to prevent it, or guarantee it during compilation.[...] There is one flaw with Moritz's example: if the zero page is mapped somehow, that means 0 is potentially a valid address of a variable, and therefore checking for null is basically not only useless but wrong: a null check of the address of this variable will fail, yet the pointer is actually pointing at a valid address that just happens to be 0. IOW, if the zero page is mapped, we're *already* screwed anyway, might as well just give up now.
Jul 27 2017
On Thursday, 27 July 2017 at 17:52:09 UTC, H. S. Teoh wrote:On Thu, Jul 27, 2017 at 11:03:02AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...]The point of the example was to show that exploiting the "null dereferences segfault" assumption on a modern Linux system to create completely unexpected behaviour (in the case I showed fgetc is going to make the process hang -> denial of service with hard to detect cause) and break any D program's safe correctness is almost trivial.However, there do exist places where dereferencing null may NOT cause a segmentation fault. For example, see this post by Moritz Maxeiner: https://forum.dlang.org/post/udkdqogtrvanhbotdoik forum.dlang.org In such cases, the compiled program can have no knowledge that the zero page is mapped somehow. There is no way to prevent it, or guarantee it during compilation.[...] There is one flaw with Moritz's example: if the zero page is mapped somehow, that means 0 is potentially a valid address of a variable, and therefore checking for null is basically not only useless but wrong: a null check of the address of this variable will fail, yet the pointer is actually pointing at a valid address that just happens to be 0. IOW, if the zero page is mapped, we're *already* screwed anyway, might as well just give up now.
Jul 27 2017
On Thursday, July 27, 2017 11:03:02 Steven Schveighoffer via Digitalmars-d wrote:A possibility: " safe D does not support platforms or processes where dereferencing a null pointer does not crash the program. In such situations, dereferencing null is not defined, and safe code will not prevent this from happening." In terms of not marking C/C++ code safe, I am not convinced we need to go that far, but it's not as horrible a prospect as having to unmark D safe code that might dereference null.I see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever. However, one issue that has been brought up from time to time and AFAIK has never really been addressed is that apparently if an object is large enough, when you access one of its members when the pointer is null, you won't get a segfault (I think that it was something like if the object was greater than a page in size). So, as I understand it, ludicrously large objects _could_ result in safety problems with null pointers. This would not happen in normal code, but it can happen. And if we want safe to make the guarantees that it claims, we really should either disallow such objects or insert null checks for them. For smaller objects though, what's the point? It buys us nothing if the hardware is already doing it, and the only hardware that wouldn't do it should be too old to matter at this point. So, I say that we need to deal with the problem with ludicrously large objects, but beyond that, we should just change the spec, because inserting the checks buys us nothing. - Jonathan M Davis
Jul 27 2017
On 7/27/17 2:46 PM, Jonathan M Davis via Digitalmars-d wrote:However, one issue that has been brought up from time to time and AFAIK has never really been addressed is that apparently if an object is large enough, when you access one of its members when the pointer is null, you won't get a segfault (I think that it was something like if the object was greater than a page in size). So, as I understand it, ludicrously large objects _could_ result in safety problems with null pointers. This would not happen in normal code, but it can happen. And if we want safe to make the guarantees that it claims, we really should either disallow such objects or insert null checks for them. For smaller objects though, what's the point? It buys us nothing if the hardware is already doing it, and the only hardware that wouldn't do it should be too old to matter at this point.Yes: https://issues.dlang.org/show_bug.cgi?id=5176 There is a way to "fix" this: any time you access an object field that goes outside the page size, do a null check on the base pointer. -Steve
Jul 27 2017
On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote:On Thursday, July 27, 2017 11:03:02 Steven Schveighoffer via Digitalmars-d wrote:Except that when someone gets (root) access to any modern Linux servers running D services he now has an easy way to create a denial of service attack the owner of the server won't easily be able to find the cause of, because pretty much everything *looks* right, except that somehow the D services hang.A possibility: " safe D does not support platforms or processes where dereferencing a null pointer does not crash the program. In such situations, dereferencing null is not defined, and safe code will not prevent this from happening." In terms of not marking C/C++ code safe, I am not convinced we need to go that far, but it's not as horrible a prospect as having to unmark D safe code that might dereference null.I see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever.
Jul 27 2017
On 7/27/17 3:50 PM, Moritz Maxeiner wrote:On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote:Well, let's not forget that the services should not be dereferencing null. It's still a bug in the code. It just may result in something other than a process exit. I bet if you lowered that limit, you would cause all sorts of trouble, not just in D safe code. Imagine, any function that returns null specifically to mean an error, now may return it casually as the address of a valid item! You are going to screw up all checks for null! -SteveOn Thursday, July 27, 2017 11:03:02 Steven Schveighoffer via Digitalmars-d wrote:Except that when someone gets (root) access to any modern Linux servers running D services he now has an easy way to create a denial of service attack the owner of the server won't easily be able to find the cause of, because pretty much everything *looks* right, except that somehow the D services hang.A possibility: " safe D does not support platforms or processes where dereferencing a null pointer does not crash the program. In such situations, dereferencing null is not defined, and safe code will not prevent this from happening." In terms of not marking C/C++ code safe, I am not convinced we need to go that far, but it's not as horrible a prospect as having to unmark D safe code that might dereference null.I see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever.
Jul 27 2017
On Thursday, 27 July 2017 at 20:09:46 UTC, Steven Schveighoffer wrote:Well, let's not forget that the services should not be dereferencing null. It's still a bug in the code.Of course, but statistically speaking, all software is buggy so it's not an unreasonable assumption on the attackers part that there is at least one null dereference in complex server code that will eventually trigger.It just may result in something other than a process exit.Which is really bad for process supervision, because it'll likely not detect a problem and not kill+respawn the service.I bet if you lowered that limit, you would cause all sorts of trouble, not just in D safe code. Imagine, any function that returns null specifically to mean an error, now may return it casually as the address of a valid item! You are going to screw up all checks for null!Indeed, but atm I was only concerned about the implications for D safe code.
Jul 27 2017
On Thu, Jul 27, 2017 at 07:50:52PM +0000, Moritz Maxeiner via Digitalmars-d wrote:On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote:[...]If someone malicious has root access to your server, you already have much bigger things to worry about than D services hanging. :-D T -- Don't get stuck in a closet---wear yourself out.I see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever.Except that when someone gets (root) access to any modern Linux servers running D services he now has an easy way to create a denial of service attack the owner of the server won't easily be able to find the cause of, because pretty much everything *looks* right, except that somehow the D services hang.
Jul 27 2017
On Thursday, 27 July 2017 at 20:48:51 UTC, H. S. Teoh wrote:On Thu, Jul 27, 2017 at 07:50:52PM +0000, Moritz Maxeiner via Digitalmars-d wrote:That depends on how valuable you are as a target, how hard it was to gain root access, and what the attacker's intentions are. If you are a high value target for which root access was hard to get, the attacker is unlikely to risk detection by doing things that someone (or an IDS) will categorize as an attack; the attacker is much more likely to try and subvert the system without being detected; see for example how Stuxnet was used to slowly damage centrifuge machines.On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote:[...]If someone malicious has root access to your server, you already have much bigger things to worry about than D services hanging. :-DI see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever.Except that when someone gets (root) access to any modern Linux servers running D services he now has an easy way to create a denial of service attack the owner of the server won't easily be able to find the cause of, because pretty much everything *looks* right, except that somehow the D services hang.
Jul 27 2017
On Thu, Jul 27, 2017 at 09:32:12PM +0000, Moritz Maxeiner via Digitalmars-d wrote:On Thursday, 27 July 2017 at 20:48:51 UTC, H. S. Teoh wrote:[...]Yes, and therefore "you already have much bigger things to worry about than D services hanging". That you're ignorant of the compromise does not negate the fact that you do have bigger things to worry about, you're just blissfully unaware of them. :-P Until the good stuff hits the proverbial fan, of course. T -- Tell me and I forget. Teach me and I remember. Involve me and I understand. -- Benjamin FranklinIf someone malicious has root access to your server, you already have much bigger things to worry about than D services hanging. :-DThat depends on how valuable you are as a target, how hard it was to gain root access, and what the attacker's intentions are. If you are a high value target for which root access was hard to get, the attacker is unlikely to risk detection by doing things that someone (or an IDS) will categorize as an attack; the attacker is much more likely to try and subvert the system without being detected; see for example how Stuxnet was used to slowly damage centrifuge machines.
Jul 27 2017
On Thursday, 27 July 2017 at 23:52:27 UTC, H. S. Teoh wrote:On Thu, Jul 27, 2017 at 09:32:12PM +0000, Moritz Maxeiner via Digitalmars-d wrote:Sure, but this kind of subversive attack *keeps* you *the defender) ignorant, i.e. you are unlikely to notice that you have bigger issues to worry about.[...]Yes, and therefore "you already have much bigger things to worry about than D services hanging". That you're ignorant of the compromise does not negate the fact that you do have bigger things to worry about, you're just blissfully unaware of them. :-P Until the good stuff hits the proverbial fan, of course.
Jul 28 2017
On Thursday, July 27, 2017 13:48:51 H. S. Teoh via Digitalmars-d wrote:On Thu, Jul 27, 2017 at 07:50:52PM +0000, Moritz Maxeiner via Digitalmars-d wrote:Agreed. And Safe D has never made any promises about denial of service attacks and whatnot, let alone preventing things going wrong if someone has root access. If you don't want segfaulting to open a window for someone to hit you with a DoS attack, then don't dereference null pointers, and if you don't want someone to do nasty things to your server that would require them to be root, then do the appropriate things to protect your machine so that they don't have root. We can _always_ find ways that a badly written program can have issues with DoS attacks or have trouble if someone malicious has access to the machine that it's running on. safe is about guaranteeing memory safety, not about stopping people from screwing you over when you write bad code or fail to protect your computer from attacks. - Jonathan M DavisOn Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote:[...]If someone malicious has root access to your server, you already have much bigger things to worry about than D services hanging. :-DI see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever.Except that when someone gets (root) access to any modern Linux servers running D services he now has an easy way to create a denial of service attack the owner of the server won't easily be able to find the cause of, because pretty much everything *looks* right, except that somehow the D services hang.
Jul 27 2017