www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DMD supports ranges, but druntime can't see them?

reply Manu <turkeyman gmail.com> writes:
I noticed a bizarre situation just now; while DMD has magic in the
compiler to understand ranges and lower foreach loops to call range
primitives and such, the material to determine any range API is not
available to druntime, it's all the way out in std.range.primitives.

Why is there no core.range with said primitives in it? I think it's
fine that chain and zip and iota all that jazz is in phobos, but
surely isInputRange, isRandomAccessRange, hasLength, etc should all be
in the lowest level runtime possible?

It's kinda the same deal as those select traits which are practically
part of the language, they don't generate any code, and it's virtually
impossible to do anything useful without them.

Should they be moved?
May 22
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 23 May 2019 at 06:16:08 UTC, Manu wrote:
 I noticed a bizarre situation just now; while DMD has magic in 
 the compiler to understand ranges and lower foreach loops to 
 call range primitives and such, the material to determine any 
 range API is not available to druntime, it's all the way out in 
 std.range.primitives.

 Why is there no core.range with said primitives in it? I think 
 it's fine that chain and zip and iota all that jazz is in 
 phobos, but surely isInputRange, isRandomAccessRange, 
 hasLength, etc should all be in the lowest level runtime 
 possible?
I think Andrei said it best here: https://forum.dlang.org/post/q7j4sl$17pe$1 digitalmars.com ---
 Oh, and druntime must go.
 
 The whole distinction between the runtime library and the 
 standard library is clowny and has more to do with poorly 
 ported tradition from other languages, than with anything else.
 
 We need one standard library that is entirely pay-as-you-go 
 (e.g. empty main() means no library file is ever needed for 
 linking) and that offers an opt-in continuum.
--- So, the divide between druntime and phobos shouldn't even exist. But what does a united druntime/Phobos look like? How do we organize it logically, and modularly with clear boundaries, so it doesn't turn into a tangled mess and users only pay for what they use? I don't know, but I'm trying to figure that out. Andrei and I have a difference of opinion on how to achieve that which is apparent in this conversation: https://github.com/dlang/dmd/pull/9814 I'm trying to engage with him via e-mail and other PRs (e.g. https://github.com/JinShil/utiliD/pull/7) to incrementally work something out and get a clearer vision of what the future should look like. There are also some prerequisites that need to be implemented first before some of this stuff will work right (e.g. converting some runtime hooks to templates). That's also being worked on.
 It's kinda the same deal as those select traits which are 
 practically part of the language, they don't generate any code, 
 and it's virtually impossible to do anything useful without 
 them.

 Should they be moved?
For now, yeah, perhaps they should just be moved to `core.range` or something like that until a better vision of what a united, pay-as-you-go druntime/phobos is worked out. I'm mentoring a GSoC project this year and we could actually benefit from having some of that range stuff in druntime. And this isn't unique to the range stuff. There's quite a bit in Phobos (std.traits, std.conv, std.meta, etc...) that really needs to be available to all D programs including druntime and the compiler itself. Mike
May 22
next sibling parent reply Manu <turkeyman gmail.com> writes:
On Wed, May 22, 2019 at 11:45 PM Mike Franklin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Thursday, 23 May 2019 at 06:16:08 UTC, Manu wrote:
 I noticed a bizarre situation just now; while DMD has magic in
 the compiler to understand ranges and lower foreach loops to
 call range primitives and such, the material to determine any
 range API is not available to druntime, it's all the way out in
 std.range.primitives.

 Why is there no core.range with said primitives in it? I think
 it's fine that chain and zip and iota all that jazz is in
 phobos, but surely isInputRange, isRandomAccessRange,
 hasLength, etc should all be in the lowest level runtime
 possible?
I think Andrei said it best here: https://forum.dlang.org/post/q7j4sl$17pe$1 digitalmars.com
Right... but we're a decade away from that vision, we should do something for the time being.
 It's kinda the same deal as those select traits which are
 practically part of the language, they don't generate any code,
 and it's virtually impossible to do anything useful without
 them.

 Should they be moved?
For now, yeah, perhaps they should just be moved to `core.range` or something like that until a better vision of what a united, pay-as-you-go druntime/phobos is worked out. I'm mentoring a GSoC project this year and we could actually benefit from having some of that range stuff in druntime. And this isn't unique to the range stuff. There's quite a bit in Phobos (std.traits, std.conv, std.meta, etc...) that really needs to be available to all D programs including druntime and the compiler itself.
Indeed. Although only subsets of each of those files for my money, and they feel improperly organised to me. So, std.range.primitives -> core.range ?
May 23
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 23 May 2019 at 07:17:19 UTC, Manu wrote:

 So, std.range.primitives -> core.range ?
Take your best guess, submit a PR and see what the reviewers say. If I had to make a decision, I'd just mimic what's in Phobos (i.e. replace `std` with `core`), unless there's a good reason not to. Mike
May 23
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 23 May 2019 at 07:26:34 UTC, Mike Franklin wrote:
 On Thursday, 23 May 2019 at 07:17:19 UTC, Manu wrote:

 So, std.range.primitives -> core.range ?
Take your best guess, submit a PR and see what the reviewers say. If I had to make a decision, I'd just mimic what's in Phobos (i.e. replace `std` with `core`), unless there's a good reason not to.
A more conservative approach would be to put them in `core.internal`. `core.internal.traits` is already there. My understanding is that anything in `core.internal` is not intended for public consumption, so we could start there without committing to a public interface that we'd have to maintain. It can always be exposed publicly later when the future becomes more clear. Mike
May 23
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 23 May 2019 at 07:40:20 UTC, Mike Franklin wrote:
 On Thursday, 23 May 2019 at 07:26:34 UTC, Mike Franklin wrote:
 On Thursday, 23 May 2019 at 07:17:19 UTC, Manu wrote:

 So, std.range.primitives -> core.range ?
Take your best guess, submit a PR and see what the reviewers say. If I had to make a decision, I'd just mimic what's in Phobos (i.e. replace `std` with `core`), unless there's a good reason not to.
A more conservative approach would be to put them in `core.internal`. `core.internal.traits` is already there. My understanding is that anything in `core.internal` is not intended for public consumption, so we could start there without committing to a public interface that we'd have to maintain. It can always be exposed publicly later when the future becomes more clear. Mike
Yes, let's go with core.internal.range. Motivation: when we can the package name, we have the unique opportunity of getting rid of some historic range missteps like auto-decoding for strings or using a special save() constructor for ForwardRanges to support class ranges.
May 23
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, May 23, 2019 1:45:23 AM MDT Seb via Digitalmars-d wrote:
 On Thursday, 23 May 2019 at 07:40:20 UTC, Mike Franklin wrote:
 On Thursday, 23 May 2019 at 07:26:34 UTC, Mike Franklin wrote:
 On Thursday, 23 May 2019 at 07:17:19 UTC, Manu wrote:
 So, std.range.primitives -> core.range ?
Take your best guess, submit a PR and see what the reviewers say. If I had to make a decision, I'd just mimic what's in Phobos (i.e. replace `std` with `core`), unless there's a good reason not to.
A more conservative approach would be to put them in `core.internal`. `core.internal.traits` is already there. My understanding is that anything in `core.internal` is not intended for public consumption, so we could start there without committing to a public interface that we'd have to maintain. It can always be exposed publicly later when the future becomes more clear. Mike
Yes, let's go with core.internal.range. Motivation: when we can the package name, we have the unique opportunity of getting rid of some historic range missteps like auto-decoding for strings or using a special save() constructor for ForwardRanges to support class ranges.
Anything along those lines that gets done in druntime at this point needs to not be part of the public API, because if we're really going to do some kind of phobos v2 and fix some of the mistakes with ranges, we want to get it right. And if we have something in core which is public, that will get in the way of that. Either way, it won't work to copy what we have in std.range.primitives right now (at least not for strings), because that relies on std.utf. However, since actually redesigning ranges is a much larger discussion, it really wouldn't be appropriate to just throw a new API in druntime either. So, I don't know that it's appropriate to put much in the way of range stuff in druntime at the moment. Not much there should even need it either. Regardless, anything that is done needs to be completely internal for now. - Jonathan M Davis
May 23
prev sibling parent reply Timo Sintonen <t.sintonen luukku.com> writes:
On Thursday, 23 May 2019 at 06:41:04 UTC, Mike Franklin wrote:

 Oh, and druntime must go.
 
 The whole distinction between the runtime library and the 
 standard library is clowny and has more to do with poorly 
 ported tradition from other languages, than with anything else.
 
 We need one standard library that is entirely pay-as-you-go 
 (e.g. empty main() means no library file is ever needed for 
 linking) and that offers an opt-in continuum.
--- So, the divide between druntime and phobos shouldn't even exist. But what does a united druntime/Phobos look like? How do we organize it logically, and modularly with clear boundaries, so it doesn't turn into a tangled mess and users only pay for what they use? I don't know, but I'm trying to figure that out.
We actually do need a runtime library. It should contain code for compiler generated runtime function calls. Nothing else. Users should have no need to import it or even have access to it. The rest should be moved. User callable code might go to phobos. Files that have only bindings and definitions might be in phobos or in utiliD. Then there is code and imports that are not for users but used internally in the library. I am not sure where they should be but utiliD might be a good place. What is left in runtime should not depend on phobos or utiliD. Some libc calls are always needed but they shoud be used as little as possible.
May 23
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 23 May 2019 at 09:46:14 UTC, Timo Sintonen wrote:

 We actually do need a runtime library. It should contain code 
 for compiler generated runtime function calls. Nothing else.
I agree. But consider that that runtime library is also written in D, and can have some complex constructs. That is why I want to build something like utiliD, so it can be imported by that runtime library and used to make the runtime library's implementation more idiomatic D. It can also be used by applications that aren't using any runtime features (e.g. some bare-metal platforms, and consuming D libraries from other languages such as C).
 Users should have no need to import it or even have access to 
 it.
This is where I disagree because I'm trying to make the language opt-in and pay-as-you-go. What I'm envisioning is a "Language as a Library". If one wants classes they `import classes;`. If one wants exceptions, they `import exceptions;` etc. The modules one imports contain the compiler lowerings and other constructs to support the feature. The compiler will emit an error if it can't find something that's not imported, including the compiler hooks, (just like it does today if one tries to use a symbol that isn't imported) Users opt-in to a feature by importing the feature. Note that this behavior is predicated on work being done to move all runtime lowerings into the semantic phase of the compiler instead of the IR stage like it is today. That work is currently underway. Walter and Andrei don't appear to like that idea. What they appear to propose is to make *all* runtime constructs templates. The runtime library is automatically imported, but because it's all templates, there's nothing to link. Users opt-in to a feature by simply using it. Either the instantiate one of the runtime templates explicitly, or by using the feature, the compiler lowers an expression to a runtime template implicitly. I'm skeptical of that. For example, do we need to declare `class Object() {}` instead of `class Object {}` in the runtime template library? And think how that would need to scale all the way down the call stack if each runtime implementation, and constructs it uses, and each construct that those constructs use all need to be templated. And what would that do to compile-times? Each module would have to re-compile all the runtime stuff they utilize. And what about the potential for template bloat? Mike
May 23
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 23 May 2019 at 11:00:13 UTC, Mike Franklin wrote:
 [snip]

 This is where I disagree because I'm trying to make the 
 language opt-in and pay-as-you-go.  What I'm envisioning is a 
 "Language as a Library".  If one wants classes they `import 
 classes;`.  If one wants exceptions, they `import exceptions;` 
 etc.  The modules one imports contain the compiler lowerings 
 and other constructs to support the feature.  The compiler will 
 emit an error if it can't find something that's not imported, 
 including the compiler hooks, (just like it does today if one 
 tries to use a symbol that isn't imported)  Users opt-in to a 
 feature by importing the feature.  Note that this behavior is 
 predicated on work being done to move all runtime lowerings 
 into the semantic phase of the compiler instead of the IR stage 
 like it is today.  That work is currently underway.
I like the sound of this, though I think it would be helpful to have some sense of the size of the impact (on compile-times, run-times, and memory, at least) when those features are used (and not just when they are not used).
May 23
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 23 May 2019 at 11:00:13 UTC, Mike Franklin wrote:
 On Thursday, 23 May 2019 at 09:46:14 UTC, Timo Sintonen wrote:

 Walter and Andrei don't appear to like that idea.  What they 
 appear to propose is to make *all* runtime constructs 
 templates.  The runtime library is automatically imported, but 
 because it's all templates, there's nothing to link.  Users 
 opt-in to a feature by simply using it.  Either the instantiate 
 one of the runtime templates explicitly, or by using the 
 feature, the compiler lowers an expression to a runtime 
 template implicitly.
I would also favour this approach.
 I'm skeptical of that.  For example, do we need to declare 
 `class Object() {}` instead of `class Object {}` in the runtime 
 template library?  And think how that would need to scale all 
 the way down the call stack if each runtime implementation, and 
 constructs it uses, and each construct that those constructs 
 use all need to be templated.  And what would that do to 
 compile-times? Each module would have to re-compile all the 
 runtime stuff they utilize.  And what about the potential for 
 template bloat?
These are good questions. I've never understood complaints about template bloat however. If the linker does its job, does it matter that individual object files are larger?
May 23
parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 23 May 2019 at 13:01:58 UTC, Atila Neves wrote:

 If the linker does its job, does it matter that individual 
 object files are larger?
Yes, my understanding is that the final binary would only have a single copy of whatever templates were instantiated. However, there is a danger for template bloat if their are too many template permutations based on type or other compile-time factor. For example, if the authors of druntime create large functions templated on type without factoring out the type-independent code into smaller functions. Then template bloat could become a problem. It's just something that needs to be carefully monitored if everything in the runtime is to be a template. Mike
May 23
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 5/23/19 5:46 AM, Timo Sintonen wrote:
 On Thursday, 23 May 2019 at 06:41:04 UTC, Mike Franklin wrote:
 
 Oh, and druntime must go.

 The whole distinction between the runtime library and the standard 
 library is clowny and has more to do with poorly ported tradition 
 from other languages, than with anything else.

 We need one standard library that is entirely pay-as-you-go (e.g. 
 empty main() means no library file is ever needed for linking) and 
 that offers an opt-in continuum.
--- So, the divide between druntime and phobos shouldn't even exist.  But what does a united druntime/Phobos look like?  How do we organize it logically, and modularly with clear boundaries, so it doesn't turn into a tangled mess and users only pay for what they use?  I don't know, but I'm trying to figure that out.
We actually do need a runtime library. It should contain code for compiler generated runtime function calls. Nothing else.
Ideally all of those would be small function templates.
May 23