www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - safety of Array

reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
I've been spot checking parts of phobos today to see what all isn't 
 safe ready.  I'm not shocked that Array isn't, but doesn't it need to be?

For instance, not even the most basic of uses works:

$ git diff -U5
diff --git a/std/container/array.d b/std/container/array.d
index 4a1bfb4..2672bc6 100644
--- a/std/container/array.d
+++ b/std/container/array.d
   -826,11 +826,11    $(D r)
          length = offset1 + tailLength;
          return this[length - tailLength .. length];
      }
  }

-unittest
+ safe unittest
  {
      Array!int a;
      assert(a.empty);
  }

==> std/container/array.d(833): Error: safe function 
'std.container.array.__unittestL831_1' cannot call system function 
'std.container.array.Array!int.Array.~this'

I know it's a tricky implementation, but let's focus on the goal.. 
should Array be usable in  safe code?
Oct 12 2014
next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 13 October 2014 at 00:42:09 UTC, Brad Roberts via 
Digitalmars-d wrote:

 I know it's a tricky implementation, but let's focus on the 
 goal.. should Array be usable in  safe code?


Looks like the `RefCounted` dependency has to be made  safe first.
Oct 12 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:

 I know it's a tricky implementation, but let's focus on the goal..
 should Array be usable in  safe code?


Yes. In order for that to be 100% automatically checkable, we need the 
rules restricting escape of addresses of returns by reference. -- Andrei
Oct 13 2014
parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d wrote:

 On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:

 I know it's a tricky implementation, but let's focus on the goal..
 should Array be usable in  safe code?


 Yes. In order for that to be 100% automatically checkable, we need the
 rules restricting escape of addresses of returns by reference. -- Andrei


100% checkable isn't required right now.  For it to be used in an  safe 
context all that's needed is liberal use of  trusted.  That can be 
refined over time to a more checked version.  We shouldn't wait for 
checkability.

Will one of you experts in the impl of Array volunteer to make the 
appropriate changes?
Oct 13 2014
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via 
Digitalmars-d wrote:

 On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d 
 wrote:

 On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:

 I know it's a tricky implementation, but let's focus on the 
 goal..
 should Array be usable in  safe code?


 Yes. In order for that to be 100% automatically checkable, we 
 need the
 rules restricting escape of addresses of returns by reference. 
 -- Andrei


 100% checkable isn't required right now.  For it to be used in 
 an  safe context all that's needed is liberal use of  trusted.  
 That can be refined over time to a more checked version.  We 
 shouldn't wait for checkability.

 Will one of you experts in the impl of Array volunteer to make 
 the appropriate changes?


The issue is that it's *not* safe though. You can escape the 
reference, destroy it, and end up with a dangling pointer. 
Arbitrarily marking things as trusted seriously undermines what 
safe means.  trusted should be used with extreme caution.
Oct 13 2014
next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
Example:

ref int getAsRef(int a)  safe //Magic!
{
RefCounted!int rc = a;
return rc.getPayload();
}

I wouldn't want to be on the calling end of this "safe" 
function...
Oct 13 2014
prev sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10/13/2014 1:28 PM, monarch_dodra via Digitalmars-d wrote:

 On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via
 Digitalmars-d wrote:

 On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d wrote:

 On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:

 I know it's a tricky implementation, but let's focus on the goal..
 should Array be usable in  safe code?


 Yes. In order for that to be 100% automatically checkable, we need the
 rules restricting escape of addresses of returns by reference. -- Andrei


 100% checkable isn't required right now.  For it to be used in an
  safe context all that's needed is liberal use of  trusted. That can
 be refined over time to a more checked version.  We shouldn't wait for
 checkability.

 Will one of you experts in the impl of Array volunteer to make the
 appropriate changes?


 The issue is that it's *not* safe though. You can escape the reference,
 destroy it, and end up with a dangling pointer. Arbitrarily marking
 things as trusted seriously undermines what safe means.  trusted should
 be used with extreme caution.


That's why I asked the question I did.  The core question isn't about 
what the current implementation is or does but about where it should end 
up.  Should Array be usable in  safe code.  So far:

Jakob: focused on impl
Andrei: yes
Monarch: focused on impl

I totally agree that  trusted must be used with lots of caution.  But my 
point in that post was that impl isn't the issue and requiring that 
everything be fixed and perfect also isn't the issue.  If we don't know 
and understand where we want to be, the chances of accidentally landing 
there are rather low.

More and more code is being created in Phobos all the time, and it's use 
in  safe code is largely an afterthought.  Please don't derail this 
thread and talk about process.. keep this thread focused on Array.

Thanks,
Brad
Oct 13 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via 
Digitalmars-d wrote:

 That's why I asked the question I did.  The core question isn't 
 about what the current implementation is or does but about 
 where it should end up.  Should Array be usable in  safe code.


It should but it can't unless something like `scope` is 
implemented. We can't have  safe in standard library that doesn't 
hold  safe promises.
Oct 13 2014
prev sibling next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via 
Digitalmars-d wrote:

 On 10/13/2014 1:28 PM, monarch_dodra via Digitalmars-d wrote:

 On Monday, 13 October 2014 at 17:16:40 UTC, Brad Roberts via
 Digitalmars-d wrote:

 On 10/13/2014 7:47 AM, Andrei Alexandrescu via Digitalmars-d 
 wrote:

 On 10/12/14, 5:41 PM, Brad Roberts via Digitalmars-d wrote:

 I know it's a tricky implementation, but let's focus on the 
 goal..
 should Array be usable in  safe code?


 Yes. In order for that to be 100% automatically checkable, 
 we need the
 rules restricting escape of addresses of returns by 
 reference. -- Andrei


 100% checkable isn't required right now.  For it to be used 
 in an
  safe context all that's needed is liberal use of  trusted. 
 That can
 be refined over time to a more checked version.  We shouldn't 
 wait for
 checkability.

 Will one of you experts in the impl of Array volunteer to 
 make the
 appropriate changes?


 The issue is that it's *not* safe though. You can escape the 
 reference,
 destroy it, and end up with a dangling pointer. Arbitrarily 
 marking
 things as trusted seriously undermines what safe means. 
  trusted should
 be used with extreme caution.


 That's why I asked the question I did.  The core question isn't 
 about what the current implementation is or does but about 
 where it should end up.  Should Array be usable in  safe code.  
 So far:

 Jakob: focused on impl
 Andrei: yes
 Monarch: focused on impl

 I totally agree that  trusted must be used with lots of 
 caution.  But my point in that post was that impl isn't the 
 issue and requiring that everything be fixed and perfect also 
 isn't the issue.  If we don't know and understand where we want 
 to be, the chances of accidentally landing there are rather low.

 More and more code is being created in Phobos all the time, and 
 it's use in  safe code is largely an afterthought.  Please 
 don't derail this thread and talk about process.. keep this 
 thread focused on Array.

 Thanks,
 Brad


You say I'm focused on impl, but  safe *is* an implementation 
certification.

I'm not derailing the thread or talking about process. If Array 
can't be certified memory safe, then it can't be marked as  safe. 
That's really all there is to it.
Oct 14 2014
parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10/14/2014 3:49 AM, monarch_dodra via Digitalmars-d wrote:


 You say I'm focused on impl, but  safe *is* an implementation
 certification.

 I'm not derailing the thread or talking about process. If Array can't be
 certified memory safe, then it can't be marked as  safe. That's really
 all there is to it.


Sorry, the request to not derail was for future posts to the thread, not 
a reaction to your comment.  I was/am worried that the "more and more 
code being created" comment would spiral the discussion sideways.

Your response wasn't making any statements of what should happen but 
rather why it can't based on the current state.  Useful, but still 
irrelevant to the Should question.  Unless you were saying that it 
shouldn't become usable due to that part of the api.  To that I'd 
respond that your thinking is too narrow in scope or too black and white.

As to the rest, once we decide if Array should be usable in the  safe 
subset of the language, then we can start to make choices about how to 
achieve that.  Some obvious choices:

1) remove the parts that aren't (unlikely to be a good choice)

2) partition the api into parts that are and parts that aren't (only 
some parts get the  safe attribute, maybe some gets  trusted)

3) improve the implementation of  safe to cover all the parts that can't 
right now (likely to result in significant delay before any useful 
progress is made)

4) force the parts that aren't anyway (probably violates the basic 
precepts of  safety but including for the sake of completeness)

5) ?

6) some combination of the above


long as a reasonably large subset of the functionality can be made 
usable (right now we can't even construct one).  With a likely very
Oct 14 2014
next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Tuesday, 14 October 2014 at 17:59:43 UTC, Brad Roberts via 
Digitalmars-d wrote:

 On 10/14/2014 3:49 AM, monarch_dodra via Digitalmars-d wrote:


 You say I'm focused on impl, but  safe *is* an implementation
 certification.

 I'm not derailing the thread or talking about process. If 
 Array can't be
 certified memory safe, then it can't be marked as  safe. 
 That's really
 all there is to it.


 Sorry, the request to not derail was for future posts to the 
 thread, not a reaction to your comment.  I was/am worried that 
 the "more and more code being created" comment would spiral the 
 discussion sideways.

 Your response wasn't making any statements of what should 
 happen but rather why it can't based on the current state.  
 Useful, but still irrelevant to the Should question.  Unless 
 you were saying that it shouldn't become usable due to that 
 part of the api.  To that I'd respond that your thinking is too 
 narrow in scope or too black and white.

 As to the rest, once we decide if Array should be usable in the 
  safe subset of the language, then we can start to make choices 
 about how to achieve that.  Some obvious choices:

 1) remove the parts that aren't (unlikely to be a good choice)

 2) partition the api into parts that are and parts that aren't 
 (only some parts get the  safe attribute, maybe some gets 
  trusted)

 3) improve the implementation of  safe to cover all the parts 
 that can't right now (likely to result in significant delay 
 before any useful progress is made)

 4) force the parts that aren't anyway (probably violates the 
 basic precepts of  safety but including for the sake of 
 completeness)

 5) ?

 6) some combination of the above


 term as long as a reasonably large subset of the functionality 
 can be made usable (right now we can't even construct one).  
 With a likely very strategic sprinkling of  trusted where 




I'm on vacation on a phone, so I'll be brief for now.

I replied what I said because I felt some arguments were leading 
to: let's make it trusted and worry about implementation layer.

I'la ignore the comment that my vision is narrow and discuss 
improvement possibilities.

The issue witH safe/unsafe split is that the functions themselves 
aren't actually unsafe, but rather their vombination: deletion is 
only unsafe *if* an escape has occurred.

The funny thing is that Array used to be sealed (specifically to 
avoid escapes) abd could have been safe. I unsealed it with 
Andrei because sealed containers come with their own problems.

To the topic at hand though, I don't think safety should dictate 
our implementations. In particular, dmd improvements can and will 
mean that something unsafe can become safe later.

The real question here is when will we implement the long 
promised scope?
Oct 14 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 14 October 2014 at 17:59:43 UTC, Brad Roberts via 
Digitalmars-d wrote:

 On 10/14/2014 3:49 AM, monarch_dodra via Digitalmars-d wrote:


 You say I'm focused on impl, but  safe *is* an implementation
 certification.

 I'm not derailing the thread or talking about process. If 
 Array can't be
 certified memory safe, then it can't be marked as  safe. 
 That's really
 all there is to it.


 Sorry, the request to not derail was for future posts to the 
 thread, not a reaction to your comment.  I was/am worried that 
 the "more and more code being created" comment would spiral the 
 discussion sideways.

 Your response wasn't making any statements of what should 
 happen but rather why it can't based on the current state.  
 Useful, but still irrelevant to the Should question.  Unless 
 you were saying that it shouldn't become usable due to that 
 part of the api.  To that I'd respond that your thinking is too 
 narrow in scope or too black and white.

 As to the rest, once we decide if Array should be usable in the 
  safe subset of the language, then we can start to make choices 
 about how to achieve that.  Some obvious choices:

 1) remove the parts that aren't (unlikely to be a good choice)

 2) partition the api into parts that are and parts that aren't 
 (only some parts get the  safe attribute, maybe some gets 
  trusted)


There are no distinct  system parts in Array API, it is a general 
reference escaping issue with manually managed memory.


 3) improve the implementation of  safe to cover all the parts 
 that can't right now (likely to result in significant delay 
 before any useful progress is made)


There no issues in  safe itself, language is simply missing the 
tools to ensure no reference escaping (== scope) right now.



 term as long as a reasonably large subset of the functionality 
 can be made usable (right now we can't even construct one).  
 With a likely very strategic sprinkling of  trusted where 




There is nothing that can be done here right now as far as I see 
it. If you think otherwise PR is welcome :)
Oct 14 2014
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Tuesday, 14 October 2014 at 01:47:10 UTC, Brad Roberts via 
Digitalmars-d wrote:

 That's why I asked the question I did.  The core question isn't 
 about what the current implementation is or does but about 
 where it should end up.  Should Array be usable in  safe code.


Well, I think it goes without saying that every piece of 
interface that is memory safe should be  safe. Granted, for 
abstract interfaces, like the range and container concepts, or 
virtual functions in classes and `interface`'s, some care is 
needed to determine whether or not memory safety should be a 
requirement, but for concrete implementations like `Array` we 
should provide the strongest guarantees possible.

To elaborate on my thinking for my previous reply: I don't know 
if Array's interface in its entirety is memory safe (element 
access looks murky, as was pointed out), but construction is 
definitely a memory safe part of it; so yes, I think there's work 
to be done here, and to that end, the safety of `RefCounted` has 
to be considered first. My initial thoughts are that at least 
construction of `RefCounted` should also be memory safe.

 safe vs  trusted is a red herring, it's just a matter of whether 
memory safety is automatically or manually checked.

(That said, the trap here is to disregard  safe because of 
current implementation deficits and just *assume* memory safety 
without proper checking and applying  trusted everywhere as a 
stop-gap measure. Just because it's clear that the interface is 
SafeD doesn't mean the implementation is (yet), and this is what 
minimized  trusted helps figure out. For example - although I'm 
not sure if it technically violates SafeD - if malloc signals OOM 
in Array.this, the code will steam ahead and try to emplace 
elements at an offset from null. Just applying  trusted and 
moving on would not catch cases like this.)

I don't think it's true that  safe is an afterthought for new 
code. New generic code uses  safe unit tests to test 
safe-readiness, and new non-generic code without regard for 
attributes is also pointed out in review. The problem with 
attributes is overwhelmingly with old code, not new code.
Oct 14 2014