www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Proposing std.typecons : Optional (with PR)

reply FeepingCreature <feepingcreature gmail.com> writes:
Since my rfc regarding deprecating `alias get this` in Nullable 
(see https://github.com/dlang/phobos/pull/7060 ) encountered no 
meaningful criticism (as I'm tactfully choosing to interpret the 
total absence of reaction), but somebody mentioned that a new 
type may have more success, since it doesn't have to justify 
itself in the same way as a change to existing behavior, I 
present `Optional`: https://github.com/dlang/phobos/pull/7065 .

To reiterate the difference:

`Nullable` is the standard type for "adding an undefined value 
state to an existing type". It's being abused as an `Optional` 
type. Why "abused"? Well:

1. `get` is **undefined** if Nullable is null

2. `get` is called implicitly.

This is not good behavior for an `Optional` type! Since my 
attempt to turn `Nullable` into an `Optional` type encountered 
zero enthusiasm and mild resistance, I've started a second 
parallel attempt to introduce an `Optional` type from scratch.

(Translation: I've copypasted `Nullable` and changed a few names.)

Opinion? Anyone? Anyone at all? I'm so alone...
Jun 11
next sibling parent reply aliak <something something.com> writes:
On Tuesday, 11 June 2019 at 07:46:38 UTC, FeepingCreature wrote:
 Since my rfc regarding deprecating `alias get this` in Nullable 
 (see https://github.com/dlang/phobos/pull/7060 ) encountered no 
 meaningful criticism (as I'm tactfully choosing to interpret 
 the total absence of reaction), but somebody mentioned that a 
 new type may have more success, since it doesn't have to 
 justify itself in the same way as a change to existing 
 behavior, I present `Optional`: 
 https://github.com/dlang/phobos/pull/7065 .

 [...]
If we're going to put an optional type there, I would consider not copy pasting nullable as it has some other issues as well, and maybe nail the interface down properly? E.g. having an optional that's a range would be rather awesome. Have you looked at: https://code.dlang.org/packages/optional. It has a lot of other added features as well but those can be removed if they're too much.
Jun 11
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 11 June 2019 at 09:58:48 UTC, aliak wrote:
 If we're going to put an optional type there, I would consider 
 not copy pasting nullable as it has some other issues as well, 
 and maybe nail the interface down properly? E.g. having an 
 optional that's a range would be rather awesome. Have you 
 looked at: https://code.dlang.org/packages/optional. It has a 
 lot of other added features as well but those can be removed if 
 they're too much.
I've looked at that and I specifically disagree with the decision to make it a range; that's why I didn't just internally switch to dub optional. Ranges are not monads. We may wish we had a concept of monads but we don't; ranges are not a general replacement for any conceivable container. `Optional.front` is just *weird* and unintuitive at first glance. It's not a bad decision in isolation, but it doesn't fit what I consider "the D style" of type design. Sorry if that explanation is too fuzzy.
Jun 11
next sibling parent reply aliak <something something.com> writes:
On Tuesday, 11 June 2019 at 10:01:18 UTC, FeepingCreature wrote:
 On Tuesday, 11 June 2019 at 09:58:48 UTC, aliak wrote:
 If we're going to put an optional type there, I would consider 
 not copy pasting nullable as it has some other issues as well, 
 and maybe nail the interface down properly? E.g. having an 
 optional that's a range would be rather awesome. Have you 
 looked at: https://code.dlang.org/packages/optional. It has a 
 lot of other added features as well but those can be removed 
 if they're too much.
I've looked at that and I specifically disagree with the decision to make it a range; that's why I didn't just internally switch to dub optional. Ranges are not monads. We may wish we had a concept of monads but we don't; ranges are not a general replacement for any conceivable container. `Optional.front` is just *weird* and unintuitive at first glance. It's not a bad decision in isolation, but it doesn't fit what I consider "the D style" of type design. Sorry if that explanation is too fuzzy.
The explanation is a start :) Ranges are not monads in the strictly mathematical sense you mean or? Why do you say this? You can add a .get or .value to a range as well to not have the weirdness. Though ranges and their .front are a central concept in D so it should be understood *if* option is defined as a range. Plus, whether or not you agree ranges are/can be monads or not is a tangential issue to seeing an optional as a monad or a collection (it's not mutually exclusive). In scala for e.g. it's a collection (also a monad), rust implements FromIterator and IntoIterator, haskell adheres to foldable, applicative, traversable and i guess others... I'm not saying it has to be a range either, but you lose out on functional composition if it's not. Or you re-implement all the stuff from std.range/algorithm you want as part of option's interface to get them.
Jun 11
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 11 June 2019 at 11:56:01 UTC, aliak wrote:
 On Tuesday, 11 June 2019 at 10:01:18 UTC, FeepingCreature wrote:
 I've looked at that and I specifically disagree with the 
 decision to make it a range; that's why I didn't just 
 internally switch to dub optional. Ranges are not monads. We 
 may wish we had a concept of monads but we don't; ranges are 
 not a general replacement for any conceivable container. 
 `Optional.front` is just *weird* and unintuitive at first 
 glance. It's not a bad decision in isolation, but it doesn't 
 fit what I consider "the D style" of type design.

 Sorry if that explanation is too fuzzy.
The explanation is a start :) Ranges are not monads in the strictly mathematical sense you mean or? Why do you say this? You can add a .get or .value to a range as well to not have the weirdness. Though ranges and their .front are a central concept in D so it should be understood *if* option is defined as a range. Plus, whether or not you agree ranges are/can be monads or not is a tangential issue to seeing an optional as a monad or a collection (it's not mutually exclusive). In scala for e.g. it's a collection (also a monad), rust implements FromIterator and IntoIterator, haskell adheres to foldable, applicative, traversable and i guess others... I'm not saying it has to be a range either, but you lose out on functional composition if it's not. Or you re-implement all the stuff from std.range/algorithm you want as part of option's interface to get them.
Right, my point is that in functional languages things like monads or generic interfaces in general are used to compose reusable containers into larger processing chains. However, I see `Optional` less as a container and more as a metaphor. As a metaphor, it's not "a range of elements with a length of either 0 or 1", it's "a type that may be either of those type's values plus 'unset'". That's not something that easily lends itself to the set of verbs that are used to manipulate ranges. D is not a functional language, it's a multiparadigm language. It shouldn't be afraid to have a type that deviates from the functional paradigm if the functional paradigm is a bad metaphoric fit for the semantic message that the type expresses.
Jun 11
parent reply aliak <something something.com> writes:
On Tuesday, 11 June 2019 at 12:01:34 UTC, FeepingCreature wrote:
 On Tuesday, 11 June 2019 at 11:56:01 UTC, aliak wrote:
 On Tuesday, 11 June 2019 at 10:01:18 UTC, FeepingCreature 
 wrote:
 I've looked at that and I specifically disagree with the 
 decision to make it a range; that's why I didn't just 
 internally switch to dub optional. Ranges are not monads. We 
 may wish we had a concept of monads but we don't; ranges are 
 not a general replacement for any conceivable container. 
 `Optional.front` is just *weird* and unintuitive at first 
 glance. It's not a bad decision in isolation, but it doesn't 
 fit what I consider "the D style" of type design.

 Sorry if that explanation is too fuzzy.
The explanation is a start :) Ranges are not monads in the strictly mathematical sense you mean or? Why do you say this? You can add a .get or .value to a range as well to not have the weirdness. Though ranges and their .front are a central concept in D so it should be understood *if* option is defined as a range. Plus, whether or not you agree ranges are/can be monads or not is a tangential issue to seeing an optional as a monad or a collection (it's not mutually exclusive). In scala for e.g. it's a collection (also a monad), rust implements FromIterator and IntoIterator, haskell adheres to foldable, applicative, traversable and i guess others... I'm not saying it has to be a range either, but you lose out on functional composition if it's not. Or you re-implement all the stuff from std.range/algorithm you want as part of option's interface to get them.
Right, my point is that in functional languages things like monads or generic interfaces in general are used to compose reusable containers into larger processing chains. However, I see `Optional` less as a container and more as a metaphor. As a metaphor, it's not "a range of elements with a length of either 0 or 1", it's "a type that may be either of those type's values plus 'unset'". That's not something that easily lends itself to the set of verbs that are used to manipulate ranges.
It does actually. I use it like a range all the time: arrayOfStuff .map!maybeParseThing .each!processThing; Your argument against it being a range sounds a bit too philosophical. I think we should consider the actual technical advantages/disadvantages of having it as a range or not. And if not, at least getting functions like map/flatmap/each/whatever as usable with optional. I basically think a bit of thought should be put in to putting an optional type in the standard library instead of copying nullable (as mentioned, the range thing is not the only problem - another big one is treating null as a valid value for classes).
Jun 11
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 11 June 2019 at 12:30:10 UTC, aliak wrote:
 Your argument against it being a range sounds a bit too 
 philosophical. I think we should  consider the actual technical 
 advantages/disadvantages of having it as a range or not. And if 
 not, at least getting functions like map/flatmap/each/whatever 
 as usable with optional.
We have "map for Optional/Nullable", that's `apply`.
 I basically think a bit of thought should be put in to putting 
 an optional type in the standard library instead of copying 
 nullable (as mentioned, the range thing is not the only problem 
 - another big one is treating null as a valid value for 
 classes).
That's a fundamental problem with D though. Null *is* a valid value for classes. Making a class type optional just means that you're adding *another* type that also means "absent" at a different level of abstraction. Was it the right decision to allow null in class types? Who knows. (No.) But I don't think that's a decision that should be patched in library types.
Jun 11
next sibling parent reply Nick Treleaven <nick geany.org> writes:
On Tuesday, 11 June 2019 at 13:55:52 UTC, FeepingCreature wrote:
 On Tuesday, 11 June 2019 at 12:30:10 UTC, aliak wrote:
 (as mentioned, the range thing is not the only problem - 
 another big one is treating null as a valid value for classes).
That's a fundamental problem with D though. Null *is* a valid value for classes.
It is accepted but it is not valid. There is no object there, you can't dereference it (unless null is a valid memory address, which it is in some cases). Null is a gaping hole in any static type system. Rant: IMO the worst thing Walter ever said was "Non-nullable pointers are just plugging one hole in a cheese grater": https://twitter.com/walterbright/status/338821357762654208 Optional types with language enforcement can allow a static guarantee that any data is constructed with particular values, returning an optional that is none when the initializer was outside those values. I'd actually rather have language-assisted optional types than features like function overloading (a convenience feature, sometimes abused), because it means *the type system can model runtime checks as a static type guarantee*. They are a game changer.
 Making a class type optional just means that you're adding 
 *another* type that also means "absent" at a different level of 
 abstraction.
Re-use null to mean absent.
 Was it the right decision to allow null in class types? Who 
 knows. (No.) But I don't think that's a decision that should be 
 patched in library types.
Then I vote against including your Optional.
Jun 11
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 11 June 2019 at 18:01:32 UTC, Nick Treleaven wrote:
 I'd actually rather have language-assisted optional types than 
 features like function overloading (a convenience feature, 
 sometimes abused), because it means *the type system can model 
 runtime checks as a static type guarantee*. They are a game 
 changer.
I don't disagree, but one often end up creating a Nobody subclass singleton that is used as a replacement for null. Although that does have benefits (e.g. if you call a nobody.name() you get "nobody" rather than a crash).
Jun 11
parent Aliak <something something.com> writes:
On Tuesday, 11 June 2019 at 19:02:12 UTC, Ola Fosheim Grøstad 
wrote:
 On Tuesday, 11 June 2019 at 18:01:32 UTC, Nick Treleaven wrote:
 I'd actually rather have language-assisted optional types than 
 features like function overloading (a convenience feature, 
 sometimes abused), because it means *the type system can model 
 runtime checks as a static type guarantee*. They are a game 
 changer.
I don't disagree, but one often end up creating a Nobody subclass singleton that is used as a replacement for null. Although that does have benefits (e.g. if you call a nobody.name() you get "nobody" rather than a crash).
If there’s a possibility your class can be null then it should probably be an optional. Or lazy. Have not seen this pattern you describe in swift scala or Kotlin.
Jun 11
prev sibling parent Johannes Loher <johannes.loher fg4f.de> writes:
Am 11.06.19 um 20:01 schrieb Nick Treleaven:
 Making a class type optional just means that you're adding *another*
 type that also means "absent" at a different level of abstraction.
Re-use null to mean absent.
Java has the exact same problem with all non primitive types being nullable. They also solved it on the library level with Optional. In Java, you can never extract null from an Optional: If you try to constrcut an Optional from null using Optional.of, you will get an exception and if you use Optional.ofNullable, the result will simply be an empty Optional (which technically probably holds the null reference inside, but you can never retrieve it because trying to access it will throw an exception). This approach seems to have worked out quite well for Java, so I suggest we try a similar approach. Of course the situation is a bit easier for Java than for D because they only need to care about class types and we have to make everything work for pointers, arrays, associstive arrays, ..., as well, and the desired semantics are not that clear for those... But at least for classes, I think the route that Java took is the correct one.
Jun 11
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-06-11 15:55, FeepingCreature wrote:

 We have "map for Optional/Nullable", that's `apply`.
And the rest of the algorithms? -- /Jacob Carlborg
Jun 12
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 12 June 2019 at 09:53:03 UTC, Jacob Carlborg wrote:
 On 2019-06-11 15:55, FeepingCreature wrote:

 We have "map for Optional/Nullable", that's `apply`.
And the rest of the algorithms?
As the saying goes, "PRs welcome." Personally I get along fine without them, so I didn't see the need to include them. For instance, we have a local orElse implementation hanging around. The point of having `Optional` in the stdlib is that the rest of the ecosystem can standardize around the data structure. There is none such need for algorithms, especially oneliners.
Jun 12
parent reply aliak <something something.com> writes:
On Wednesday, 12 June 2019 at 10:52:22 UTC, FeepingCreature wrote:
 On Wednesday, 12 June 2019 at 09:53:03 UTC, Jacob Carlborg 
 wrote:
 On 2019-06-11 15:55, FeepingCreature wrote:

 We have "map for Optional/Nullable", that's `apply`.
And the rest of the algorithms?
As the saying goes, "PRs welcome." Personally I get along fine without them, so I didn't see the need to include them. For instance, we have a local orElse implementation hanging around. The point of having `Optional` in the stdlib is that the rest of the ecosystem can standardize around the data structure. There is none such need for algorithms, especially oneliners.
These algorithms are already there if it's implemented as a range. There's no need to reimplement them.
Jun 12
next sibling parent aliak <something something.com> writes:
On Wednesday, 12 June 2019 at 11:56:57 UTC, aliak wrote:
 There's no need to reimplement them.
... I would hope.
Jun 12
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Wednesday, 12 June 2019 at 11:56:57 UTC, aliak wrote:
 These algorithms are already there if it's implemented as a 
 range. There's no need to reimplement them.
For instance, `orElse` is the very clear and understandable `5.optional.chain(7.only).front`. Which, if you are at all interested in readability, you'll `alias orElse = (a, b) => a.chain(b.only).front;` anyways. At which point you might as well `alias orElse = (a, b) a.present ? a.value : b;`
Jun 12
parent Jacob Carlborg <doob me.com> writes:
On 2019-06-12 17:58, FeepingCreature wrote:
 On Wednesday, 12 June 2019 at 11:56:57 UTC, aliak wrote:
 These algorithms are already there if it's implemented as a range. 
 There's no need to reimplement them.
For instance, `orElse` is the very clear and understandable `5.optional.chain(7.only).front`. Which, if you are at all interested in readability, you'll `alias orElse = (a, b) => a.chain(b.only).front;` anyways. At which point you might as well `alias orElse = (a, b) a.present ? a.value : b;`
As I mentioned in the PR, I expect a "orElse" function to be available. -- /Jacob Carlborg
Jun 13
prev sibling parent Johannes Loher <johannes.loher fg4f.de> writes:
Am 11.06.19 um 12:01 schrieb FeepingCreature:
 Ranges are not monads. We may wish we had a concept of monads
 but we don't; ranges are not a general replacement for any conceivable
 container.
Yes, ranges are not a general replacement for the concept of a monad, but the concept "range" is basically a monad: Type constructor: For any type T, a range of T is something for that isInputRange!T is true (yes, technically that's not a single "type", but rather something like a typeclass, but it is close enough). Type converter (unit / return): unit: T -> Range T, x -> only(x) Combinator (bind / >>=): (Range T, T → Range U) → Range U, (r,f) -> r.map!f.joiner Obviously, using these definitions, unit is both a left- and right-identity for bind and bind is essentially associative. So aside from the fact that there is no "real" type constructor because there is no actual general range type, I believe this qualifies the concept of a range as a monad. All of this aside, I agree that ranges are not a good general replacement for every container. I do however disagree with you in this specific case. I personally have to use Java a lot at work and I am regularly annoyed by the fact, that Optionals in Java are not Iterables (or Streams...). I like the aproach of Scala's and Vavr's "Option" much more, which implement the corresponding Iterable interfaces. It makes a lot of things just that much easier, e.g. List.of(Optional.of(42), Optional.of(1234)).stream() .filter(Optional::isPresent) .map(Optional::get) becomes List.of(Optional.of(42), Optional.of(1234)).stream() .flatMap(o -> o) or in D: only(optional(42), optional(1234)) .filter!(o -> o.present) .map!(o -> o.value) becomes only(optional(42), optional(1234)) .joiner Of course it is still debatable if we want this, but in my opinion this is a great improvement. If we decide against Optional being a "real" optional type as used in other more functional languages, I'm actually against adding it to the standard library. I'd much prefer to fix the `alias get this` issue of Nullable in that case. Don't get me wrong however, I'd love a real Optional type. But so far, the `optional` dub package works reasonably well for me. I like the pattern matching functionality in particular, but it also provides other goodies you'd expect from a real optional type, too, e.g. `orElse`. Your suggested implementation is lacking these unfortunately.
Jun 11
prev sibling parent reply Marco de Wild <mdwild sogyo.nl> writes:
On Tuesday, 11 June 2019 at 07:46:38 UTC, FeepingCreature wrote:
 Since my rfc regarding deprecating `alias get this` in Nullable 
 (see https://github.com/dlang/phobos/pull/7060 ) encountered no 
 meaningful criticism (as I'm tactfully choosing to interpret 
 the total absence of reaction), but somebody mentioned that a 
 new type may have more success, since it doesn't have to 
 justify itself in the same way as a change to existing 
 behavior, I present `Optional`: 
 https://github.com/dlang/phobos/pull/7065 .

 To reiterate the difference:

 `Nullable` is the standard type for "adding an undefined value 
 state to an existing type". It's being abused as an `Optional` 
 type. Why "abused"? Well:

 1. `get` is **undefined** if Nullable is null

 2. `get` is called implicitly.

 This is not good behavior for an `Optional` type! Since my 
 attempt to turn `Nullable` into an `Optional` type encountered 
 zero enthusiasm and mild resistance, I've started a second 
 parallel attempt to introduce an `Optional` type from scratch.

 (Translation: I've copypasted `Nullable` and changed a few 
 names.)

 Opinion? Anyone? Anyone at all? I'm so alone...
I agree that Nullable has behaviour that is unwanted. However, does this really warrant a new type in the standard library that contains the same functionality but with different implementation details? From the perspective of someone who is not too familiar with the standard library, which one do they need to choose? Why would they pick Optional over Nullable? Why would they pick Nullable over Optional? If Optional coexists with Nullable, IMO it needs to have a distinct use case to prevent confusion. I would prefer to fix Nullable and have one consistent way of dealing with this problem in the standard library, rather than a (so it seems) "battle" between Nullable and Optional fighting for attention and usage. I don't know if any Phobos' functions use Nullable, but those would need to be updated to work with Optional as well (for said consistency). I didn't comment earlier, but I am in favour of your earlier pull request.
Jun 11
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 11 June 2019 at 11:02:34 UTC, Marco de Wild wrote:
 I agree that Nullable has behaviour that is unwanted. However, 
 does this really warrant a new type in the standard library 
 that contains the same functionality but with different 
 implementation details? From the perspective of someone who is 
 not too familiar with the standard library, which one do they 
 need to choose? Why would they pick Optional over Nullable? Why 
 would they pick Nullable over Optional? If Optional coexists 
 with Nullable, IMO it needs to have a distinct use case to 
 prevent confusion.

 I would prefer to fix Nullable and have one consistent way of 
 dealing with this problem in the standard library, rather than 
 a (so it seems) "battle" between Nullable and Optional fighting 
 for attention and usage. I don't know if any Phobos' functions 
 use Nullable, but those would need to be updated to work with 
 Optional as well (for said consistency).

 I didn't comment earlier, but I am in favour of your earlier 
 pull request.
My impression is that the distinction turns on whether the null case can be expected to occur in "normal use" of the type. As `Nullable` says, accessing `get` while the `Nullable` `isNull` is *undefined*; that is, at the level of accessing missing array keys. (Except you can access them implicitly.) From this I infer that `Nullable` is mostly supposed to be non-null, maybe to enable delayed initialization or the like, which would justify the `alias get this` and match the unittests. This is a *very* different thing from an `Optional` type! An `Optional` type is expected to be unset in normal operation. (That's why my implementation throws an Exception on access, not an Error.) That is, the user is expected to regularly encounter `Optional`s whose value is absent, and handle this state in his ordinary control flow. Hence no implicit alias, and no AssertError on access. I too, would prefer to get `Nullable` into a state where it can serve as an Optional type. However, given the amount of forum interest in this has been an approximate **zero**, I've chosen to take any and every avenue that may at some point lead to a sane Optional type in the standard library. I am **sick** of spending hours debugging `alias get this` issues, or being told that a coworker spent hours debugging an `alias get this` issue. Whatever gets me rid of them, I will do. I don't care which of these PRs gets attention. The `Optional` PR does not obsolete the `deprecate alias get this` PR. If either of them gets traction, I will be glad of it.
Jun 11
parent Johannes Loher <johannes.loher fg4f.de> writes:
Am 11.06.19 um 13:47 schrieb FeepingCreature:
 My impression is that the distinction turns on whether the null case can
 be expected to occur in "normal use" of the type. As `Nullable` says,
 accessing `get` while the `Nullable` `isNull` is *undefined*; that is,
 at the level of accessing missing array keys. (Except you can access
 them implicitly.) From this I infer that `Nullable` is mostly supposed
 to be non-null, maybe to enable delayed initialization or the like,
 which would justify the `alias get this` and match the unittests.
 
 This is a *very* different thing from an `Optional` type! An `Optional`
 type is expected to be unset in normal operation. (That's why my
 implementation throws an Exception on access, not an Error.) That is,
 the user is expected to regularly encounter `Optional`s whose value is
 absent, and handle this state in his ordinary control flow. Hence no
 implicit alias, and no AssertError on access.
Exactly. This is why I'd expect a "real" Optional type to provide a lot more utilities to actually work with it, such as pattern matching, orElse, iterating, mapping, flatMapping, filtering, etc.
Jun 11