www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - safety of Array

reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
I've been spot checking parts of phobos today to see what all isn't 
 safe ready.  I'm not shocked that Array isn't, but doesn't it need to be?

For instance, not even the most basic of uses works:

$ git diff -U5
diff --git a/std/container/array.d b/std/container/array.d
index 4a1bfb4..2672bc6 100644
--- a/std/container/array.d
+++ b/std/container/array.d
   -826,11 +826,11    $(D r)
          length = offset1 + tailLength;
          return this[length - tailLength .. length];
      }
  }

-unittest
+ safe unittest
  {
      Array!int a;
      assert(a.empty);
  }

==> std/container/array.d(833): Error: safe function 
'std.container.array.__unittestL831_1' cannot call system function 
'std.container.array.Array!int.Array.~this'

I know it's a tricky implementation, but let's focus on the goal.. 
should Array be usable in  safe code?
Oct 12 2014
next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 13 October 2014 at 00:42:09 UTC, Brad Roberts via 
Digitalmars-d wrote:
 I know it's a tricky implementation, but let's focus on the 
 goal.. should Array be usable in  safe code?
Looks like the `RefCounted` dependency has to be made safe first.
Oct 12 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:
 I know it's a tricky implementation, but let's focus on the goal..
 should Array be usable in  safe code?
Yes. In order for that to be 100% automatically checkable, we need the rules restricting escape of addresses of returns by reference. -- Andrei
Oct 13 2014
parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d wrote:
 On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:
 I know it's a tricky implementation, but let's focus on the goal..
 should Array be usable in  safe code?
Yes. In order for that to be 100% automatically checkable, we need the rules restricting escape of addresses of returns by reference. -- Andrei
100% checkable isn't required right now. For it to be used in an safe context all that's needed is liberal use of trusted. That can be refined over time to a more checked version. We shouldn't wait for checkability. Will one of you experts in the impl of Array volunteer to make the appropriate changes?
Oct 13 2014
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via 
Digitalmars-d wrote:
 On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d 
 wrote:
 On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:
 I know it's a tricky implementation, but let's focus on the 
 goal..
 should Array be usable in  safe code?
Yes. In order for that to be 100% automatically checkable, we need the rules restricting escape of addresses of returns by reference. -- Andrei
100% checkable isn't required right now. For it to be used in an safe context all that's needed is liberal use of trusted. That can be refined over time to a more checked version. We shouldn't wait for checkability. Will one of you experts in the impl of Array volunteer to make the appropriate changes?
The issue is that it's *not* safe though. You can escape the reference, destroy it, and end up with a dangling pointer. Arbitrarily marking things as trusted seriously undermines what safe means. trusted should be used with extreme caution.
Oct 13 2014
next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
Example:

ref int getAsRef(int a)  safe //Magic!
{
RefCounted!int rc = a;
return rc.getPayload();
}

I wouldn't want to be on the calling end of this "safe" 
function...
Oct 13 2014
prev sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10/13/2014 1:28 PM, monarch_dodra via Digitalmars-d wrote:
 On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via
 Digitalmars-d wrote:
 On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d wrote:
 On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:
 I know it's a tricky implementation, but let's focus on the goal..
 should Array be usable in  safe code?
Yes. In order for that to be 100% automatically checkable, we need the rules restricting escape of addresses of returns by reference. -- Andrei
100% checkable isn't required right now. For it to be used in an safe context all that's needed is liberal use of trusted. That can be refined over time to a more checked version. We shouldn't wait for checkability. Will one of you experts in the impl of Array volunteer to make the appropriate changes?
The issue is that it's *not* safe though. You can escape the reference, destroy it, and end up with a dangling pointer. Arbitrarily marking things as trusted seriously undermines what safe means. trusted should be used with extreme caution.
That's why I asked the question I did. The core question isn't about what the current implementation is or does but about where it should end up. Should Array be usable in safe code. So far: Jakob: focused on impl Andrei: yes Monarch: focused on impl I totally agree that trusted must be used with lots of caution. But my point in that post was that impl isn't the issue and requiring that everything be fixed and perfect also isn't the issue. If we don't know and understand where we want to be, the chances of accidentally landing there are rather low. More and more code is being created in Phobos all the time, and it's use in safe code is largely an afterthought. Please don't derail this thread and talk about process.. keep this thread focused on Array. Thanks, Brad
Oct 13 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via 
Digitalmars-d wrote:
 That's why I asked the question I did.  The core question isn't 
 about what the current implementation is or does but about 
 where it should end up.  Should Array be usable in  safe code.
It should but it can't unless something like `scope` is implemented. We can't have safe in standard library that doesn't hold safe promises.
Oct 13 2014
prev sibling next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via 
Digitalmars-d wrote:
 On 10/13/2014 1:28 PM, monarch_dodra via Digitalmars-d wrote:
 On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via
 Digitalmars-d wrote:
 On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d 
 wrote:
 On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:
 I know it's a tricky implementation, but let's focus on the 
 goal..
 should Array be usable in  safe code?
Yes. In order for that to be 100% automatically checkable, we need the rules restricting escape of addresses of returns by reference. -- Andrei
100% checkable isn't required right now. For it to be used in an safe context all that's needed is liberal use of trusted. That can be refined over time to a more checked version. We shouldn't wait for checkability. Will one of you experts in the impl of Array volunteer to make the appropriate changes?
The issue is that it's *not* safe though. You can escape the reference, destroy it, and end up with a dangling pointer. Arbitrarily marking things as trusted seriously undermines what safe means. trusted should be used with extreme caution.
That's why I asked the question I did. The core question isn't about what the current implementation is or does but about where it should end up. Should Array be usable in safe code. So far: Jakob: focused on impl Andrei: yes Monarch: focused on impl I totally agree that trusted must be used with lots of caution. But my point in that post was that impl isn't the issue and requiring that everything be fixed and perfect also isn't the issue. If we don't know and understand where we want to be, the chances of accidentally landing there are rather low. More and more code is being created in Phobos all the time, and it's use in safe code is largely an afterthought. Please don't derail this thread and talk about process.. keep this thread focused on Array. Thanks, Brad
You say I'm focused on impl, but safe *is* an implementation certification. I'm not derailing the thread or talking about process. If Array can't be certified memory safe, then it can't be marked as safe. That's really all there is to it.
Oct 14 2014
parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10/14/2014 3:49 AM, monarch_dodra via Digitalmars-d wrote:

 You say I'm focused on impl, but  safe *is* an implementation
 certification.

 I'm not derailing the thread or talking about process. If Array can't be
 certified memory safe, then it can't be marked as  safe. That's really
 all there is to it.
Sorry, the request to not derail was for future posts to the thread, not a reaction to your comment. I was/am worried that the "more and more code being created" comment would spiral the discussion sideways. Your response wasn't making any statements of what should happen but rather why it can't based on the current state. Useful, but still irrelevant to the Should question. Unless you were saying that it shouldn't become usable due to that part of the api. To that I'd respond that your thinking is too narrow in scope or too black and white. As to the rest, once we decide if Array should be usable in the safe subset of the language, then we can start to make choices about how to achieve that. Some obvious choices: 1) remove the parts that aren't (unlikely to be a good choice) 2) partition the api into parts that are and parts that aren't (only some parts get the safe attribute, maybe some gets trusted) 3) improve the implementation of safe to cover all the parts that can't right now (likely to result in significant delay before any useful progress is made) 4) force the parts that aren't anyway (probably violates the basic precepts of safety but including for the sake of completeness) 5) ? 6) some combination of the above long as a reasonably large subset of the functionality can be made usable (right now we can't even construct one). With a likely very
Oct 14 2014
next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 14 October 2014 at 17:59:43 UTC, Brad Roberts via 
Digitalmars-d wrote:
 On 10/14/2014 3:49 AM, monarch_dodra via Digitalmars-d wrote:

 You say I'm focused on impl, but  safe *is* an implementation
 certification.

 I'm not derailing the thread or talking about process. If 
 Array can't be
 certified memory safe, then it can't be marked as  safe. 
 That's really
 all there is to it.
Sorry, the request to not derail was for future posts to the thread, not a reaction to your comment. I was/am worried that the "more and more code being created" comment would spiral the discussion sideways. Your response wasn't making any statements of what should happen but rather why it can't based on the current state. Useful, but still irrelevant to the Should question. Unless you were saying that it shouldn't become usable due to that part of the api. To that I'd respond that your thinking is too narrow in scope or too black and white. As to the rest, once we decide if Array should be usable in the safe subset of the language, then we can start to make choices about how to achieve that. Some obvious choices: 1) remove the parts that aren't (unlikely to be a good choice) 2) partition the api into parts that are and parts that aren't (only some parts get the safe attribute, maybe some gets trusted) 3) improve the implementation of safe to cover all the parts that can't right now (likely to result in significant delay before any useful progress is made) 4) force the parts that aren't anyway (probably violates the basic precepts of safety but including for the sake of completeness) 5) ? 6) some combination of the above term as long as a reasonably large subset of the functionality can be made usable (right now we can't even construct one). With a likely very strategic sprinkling of trusted where
I'm on vacation on a phone, so I'll be brief for now. I replied what I said because I felt some arguments were leading to: let's make it trusted and worry about implementation layer. I'la ignore the comment that my vision is narrow and discuss improvement possibilities. The issue witH safe/unsafe split is that the functions themselves aren't actually unsafe, but rather their vombination: deletion is only unsafe *if* an escape has occurred. The funny thing is that Array used to be sealed (specifically to avoid escapes) abd could have been safe. I unsealed it with Andrei because sealed containers come with their own problems. To the topic at hand though, I don't think safety should dictate our implementations. In particular, dmd improvements can and will mean that something unsafe can become safe later. The real question here is when will we implement the long promised scope?
Oct 14 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 14 October 2014 at 17:59:43 UTC, Brad Roberts via 
Digitalmars-d wrote:
 On 10/14/2014 3:49 AM, monarch_dodra via Digitalmars-d wrote:

 You say I'm focused on impl, but  safe *is* an implementation
 certification.

 I'm not derailing the thread or talking about process. If 
 Array can't be
 certified memory safe, then it can't be marked as  safe. 
 That's really
 all there is to it.
Sorry, the request to not derail was for future posts to the thread, not a reaction to your comment. I was/am worried that the "more and more code being created" comment would spiral the discussion sideways. Your response wasn't making any statements of what should happen but rather why it can't based on the current state. Useful, but still irrelevant to the Should question. Unless you were saying that it shouldn't become usable due to that part of the api. To that I'd respond that your thinking is too narrow in scope or too black and white. As to the rest, once we decide if Array should be usable in the safe subset of the language, then we can start to make choices about how to achieve that. Some obvious choices: 1) remove the parts that aren't (unlikely to be a good choice) 2) partition the api into parts that are and parts that aren't (only some parts get the safe attribute, maybe some gets trusted)
There are no distinct system parts in Array API, it is a general reference escaping issue with manually managed memory.
 3) improve the implementation of  safe to cover all the parts 
 that can't right now (likely to result in significant delay 
 before any useful progress is made)
There no issues in safe itself, language is simply missing the tools to ensure no reference escaping (== scope) right now.

 term as long as a reasonably large subset of the functionality 
 can be made usable (right now we can't even construct one).  
 With a likely very strategic sprinkling of  trusted where 


There is nothing that can be done here right now as far as I see it. If you think otherwise PR is welcome :)
Oct 14 2014
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via 
Digitalmars-d wrote:
 That's why I asked the question I did.  The core question isn't 
 about what the current implementation is or does but about 
 where it should end up.  Should Array be usable in  safe code.
Well, I think it goes without saying that every piece of interface that is memory safe should be safe. Granted, for abstract interfaces, like the range and container concepts, or virtual functions in classes and `interface`'s, some care is needed to determine whether or not memory safety should be a requirement, but for concrete implementations like `Array` we should provide the strongest guarantees possible. To elaborate on my thinking for my previous reply: I don't know if Array's interface in its entirety is memory safe (element access looks murky, as was pointed out), but construction is definitely a memory safe part of it; so yes, I think there's work to be done here, and to that end, the safety of `RefCounted` has to be considered first. My initial thoughts are that at least construction of `RefCounted` should also be memory safe. safe vs trusted is a red herring, it's just a matter of whether memory safety is automatically or manually checked. (That said, the trap here is to disregard safe because of current implementation deficits and just *assume* memory safety without proper checking and applying trusted everywhere as a stop-gap measure. Just because it's clear that the interface is SafeD doesn't mean the implementation is (yet), and this is what minimized trusted helps figure out. For example - although I'm not sure if it technically violates SafeD - if malloc signals OOM in Array.this, the code will steam ahead and try to emplace elements at an offset from null. Just applying trusted and moving on would not catch cases like this.) I don't think it's true that safe is an afterthought for new code. New generic code uses safe unit tests to test safe-readiness, and new non-generic code without regard for attributes is also pointed out in review. The problem with attributes is overwhelmingly with old code, not new code.
Oct 14 2014