www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [GSoC] Trying to find a good topic

reply Dan Printzell <xwildn00bx gmail.com> writes:
Hey everyone!
I have been meaning to send these questions earlier, but then 
school happened.

Small introduction about me:
I am Dan Printzell and I am currently studying for a master 
degree in computer
science in Sweden. I'm one of the maintainers for the D package 
on Arch Linux[1]
and I am also writing a operating system from scratch in D called 
PowerNex[2].
Development of it has slowed down due to school and bugs but I'm 
trying to get
out a new release this year.

I am planning on applying to GSoC but I'm having a bit of a 
problem trying to
decide on what I should do. My goal is to try and do something 
that I will
personally use in PowerNex and it is something that will help the 
whole D
community, this is why I am looking at dmd and betterC topics.

For dmd I am primarily interested in implementing DIP1014[3], as 
this will help
me with the development of PowerNex. In the past I had to 
workaround that this
feature was missing. From the outside is seems like this would 
not take too long
to implement, at least not a whole GSoC. But as I'm not too 
familiar with dmd's
source code, I only did a few small patches to allow dmd to 
support PowerNex as
a target[4].
So my question for this topic would be: How long time do you guys 
think that
this will take to implement, test and get it merged? Is a month 
about correct?

For the betterC topic I would be interested in implementing a 
bare metal D
runtime[5]. This is something I already have some experience as I 
have written a
betterC runtime for PowerNex, and I also used to update/maintain 
Adam D. Ruppe's
minimal.zip[6] when I used that as the runtime for PowerNex.
My question would be: What features should this runtime contain? 
 From the wiki I
see that basic libc functionality is requested, like the str* 
functions and
memory management. From what I can understand, the goals it to 
have a runtime
that can be used on multiple targeted environments, without any 
external
dependencies.

Other questions regarding this topic: how separate should it be 
from druntime?
Should this be a separate library, like a druntime-lite? or is 
the goal to allow
druntime to be used for a bare metal target? Also, should this 
only be betterC
or should it support regular D as well?


Depending on the implementation size of DIP1014 and the bare 
metal D runtime,
it seems like both could be implemented during a GSoC. But I am 
probably just
being time optimistic here.

Best Regards,
Dan Printzell

[1] https://www.archlinux.org/people/trusted-users/#wild
[2] https://github.com/PowerNex/PowerNex
[3] 
https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1014.md
[4] https://github.com/PowerNex/powernex-dmd I will probably 
submit some of
these upstream when PowerNex gets to a better, stable state
[5] https://wiki.dlang.org/GSOC_2019_Ideas#Baremetal_D_runtime
[6] http://arsdnet.net/dcode/minimal.zip
Mar 21
next sibling parent reply kinke <noone nowhere.com> writes:
I'm sure you'd be of great help. :)

On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:
 For dmd I am primarily interested in implementing DIP1014[3], 
 as this will help
 me with the development of PowerNex. In the past I had to 
 workaround that this
 feature was missing. From the outside is seems like this would 
 not take too long
 to implement, at least not a whole GSoC. But as I'm not too 
 familiar with dmd's
 source code
Wrt. DIP1014, I tried to point out in the discussion that it probably needs a D ABI change (I know that it definitely will for LDC). We're not talking about a trivial minor adaptation, but a change affecting all non-POD arguments passed by value - don't copy in registers or on the stack anymore and let callEE destruct it, but have callER create a temporary, pass it by ref and then destruct it by callER (also when callEE throws), just like C++. As a nice side-effect, this will also fix C++ interop in this regard. At least for LDC, there are some implicit moves emitted by the compiler in the glue-layer between front- and backend; I assume that's also the case for DMD. I don't think a full 3 months of GSoC would be needed for a working implementation, but the effort is easily underestimated IMO. And sadly, it's not a pure front-end thing, so other compilers will have to reimplement some/most? parts.
 For the betterC topic I would be interested in implementing a 
 bare metal D
 runtime[5]. This is something I already have some experience as 
 I have written a
 betterC runtime for PowerNex, and I also used to 
 update/maintain Adam D. Ruppe's
 minimal.zip[6] when I used that as the runtime for PowerNex.
 My question would be: What features should this runtime 
 contain? From the wiki I
 see that basic libc functionality is requested, like the str* 
 functions and
 memory management. From what I can understand, the goals it to 
 have a runtime
 that can be used on multiple targeted environments, without any 
 external
 dependencies.

 Other questions regarding this topic: how separate should it be 
 from druntime?
 Should this be a separate library, like a druntime-lite? or is 
 the goal to allow
 druntime to be used for a bare metal target? Also, should this 
 only be betterC
 or should it support regular D as well?
IMO, it should indeed be something like a druntime-lite without any external dependencies, and druntime build on top of it. Containing the 'absolutely essential runtime bits for the core features' (to be defined ;) - e.g., (most?) runtime functions implicitly invoked by the compiler, e.g., for array copying/initializing). As a goal, it should be fairly simple to add support for any new target, e.g., WebAssembly and a new OS like your PowerNex.
Mar 21
parent Dan Printzell <xwildn00bx gmail.com> writes:
On Thursday, 21 March 2019 at 21:01:51 UTC, kinke wrote:
 I'm sure you'd be of great help. :)
Thanks :)
 Wrt. DIP1014, I tried to point out in the discussion that it 
 probably needs a D ABI change (I know that it definitely will 
 for LDC). We're not talking about a trivial minor adaptation, 
 but a change affecting all non-POD arguments passed by value - 
 don't copy in registers or on the stack anymore and let callEE 
 destruct it, but have callER create a temporary, pass it by ref 
 and then destruct it by callER (also when callEE throws), just 
 like C++. As a nice side-effect, this will also fix C++ interop 
 in this regard.
I missed that it would require an ABI changes, thanks for mentioning this! Because of this I do not feel confrontable to try and implement this DIP and it is probably better to focus on the bare metal runtime topic.
 IMO, it should indeed be something like a druntime-lite without 
 any external dependencies, and druntime build on top of it. 
 Containing the 'absolutely essential runtime bits for the core 
 features' (to be defined ;) - e.g., (most?) runtime functions 
 implicitly invoked by the compiler, e.g., for array 
 copying/initializing). As a goal, it should be fairly simple to 
 add support for any new target, e.g., WebAssembly and a new OS 
 like your PowerNex.
Thanks, I added these requirements to my notes.
Mar 23
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 21 Mar 2019 19:50:49 +0000 schrieb Dan Printzell:

 For the betterC topic I would be interested in implementing a bare metal
 D runtime[5]. This is something I already have some experience as I have
 written a betterC runtime for PowerNex, and I also used to
 update/maintain Adam D. Ruppe's minimal.zip[6] when I used that as the
 runtime for PowerNex.
 My question would be: What features should this runtime contain?
  From the wiki I
 see that basic libc functionality is requested, like the str* functions
 and memory management. From what I can understand, the goals it to have
 a runtime that can be used on multiple targeted environments, without
 any external dependencies.
 
 Other questions regarding this topic: how separate should it be from
 druntime?
 Should this be a separate library, like a druntime-lite? or is the goal
 to allow druntime to be used for a bare metal target? Also, should this
 only be betterC or should it support regular D as well?
 
 
I think what we should do for betterC is to first replace the extern(C) hooks with a defined API. Right now, the compiler does not know if a runtime has feature X, it simply calls an extern(C) function _featureX(). If this functions is not in the runtime, this fails at link time only. A better approach would be to use a D function featureX in object.d. The compiler can then import object.d and see if there is a function featureX. If not, the runtime does not support the used feature and the compiler should emit an error. This way, we can implement a truly modular runtime. Just implement some feature and the compiler will allow using it, do not implement it and the compiler will error if you try to use it. In the extreme case, this means we can use a D compiler without any runtime and it will gracefully degrade to a reduced subset. Ideally, we can then implement different parts of the runtime in dub sub- packages. We could then ship a gdc for embedded targets without any default runtime and it would just work for some basic runtime. If you need classes, constructors, ... include microdrt:core. If you need unittests include microdrt:unittest (:exceptions, :gc, ...). For the C library, we should have some extra package stdc. We might also want a package for device-independent, portable code (volatile wrapper, traits, ...). We could then also use different configurations for microdrt:core, where the default 'ansic' would just use ansic features and be portable everywhere where a C lib is available, 'native' would work without a C lib, etc. https://wiki.dlang.org/GSOC_2019_Ideas#GDC_.28GCC-based_D_compiler.29 Whether this should replace druntime in the long term is probably a very difficult question. I'd say start small, with a minimal core runtime which is suitable even for the smallest 8bit microcontrollers. Getting the full featureset of druntime in there in a modular runtime is probably too much work for GSoC, but as a long-term goal it would be prefereable to only have one runtime. However, druntime is not the place for such radical restructurings. -- Johannes
Mar 21
next sibling parent reply kinke <noone nowhere.com> writes:
On Thursday, 21 March 2019 at 21:54:12 UTC, Johannes Pfau wrote:
 I think what we should do for betterC is to first replace the 
 extern(C) hooks with a defined API. Right now, the compiler 
 does not know if a runtime has feature X, it simply calls an 
 extern(C) function _featureX(). If this functions is not in the 
 runtime, this fails at link time only. A better approach would 
 be to use a D function featureX in object.d. The compiler can 
 then import object.d and see if there is a function featureX. 
 If not, the runtime does not support the used feature and the 
 compiler should emit an error.
These hooks are what I meant by 'functions implicitly invoked by the compiler', and the defined API being the (or a subset of) the public exports of the stripped-down runtime. How do you propose to solve the `object.d` issue of having to accomodate for absence/availability of all subpackages with a highly modular approach? I know that types (TypeInfo, ModuleInfo) are currently handled like this (compiler checks if they are defined in object.d and only emits those symbols if they are), but that only works because they reside themselves in an already way too bloated object.d.
Mar 21
parent reply kinke <noone nowhere.com> writes:
On Thursday, 21 March 2019 at 22:30:32 UTC, kinke wrote:
 How do you propose to solve the `object.d` issue of having to 
 accomodate for absence/availability of all subpackages with a 
 highly modular approach?
Ah, dub's predefined versions `Have_*` should work indeed.
Mar 21
parent Johannes Pfau <nospam example.com> writes:
Am Thu, 21 Mar 2019 22:46:14 +0000 schrieb kinke:

 On Thursday, 21 March 2019 at 22:30:32 UTC, kinke wrote:
 How do you propose to solve the `object.d` issue of having to
 accomodate for absence/availability of all subpackages with a highly
 modular approach?
Ah, dub's predefined versions `Have_*` should work indeed.
Something like that or we could do a conditional, public import (static if(__traits(compiles, import ...)). Alternatively we could of course split the API into different modules but that would be more difficult to get upstreamed. -- Johannes
Mar 22
prev sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 21 March 2019 at 21:54:12 UTC, Johannes Pfau wrote:

 I think what we should do for betterC is to first replace the 
 extern(C) hooks with a defined API. Right now, the compiler 
 does not know if a runtime has feature X, it simply calls an 
 extern(C) function _featureX(). If this functions is not in the 
 runtime, this fails at link time only. A better approach would 
 be to use a D function featureX in object.d. The compiler can 
 then import object.d and see if there is a function featureX. 
 If not, the runtime does not support the used feature and the 
 compiler should emit an error.

 This way, we can implement a truly modular runtime. Just 
 implement some feature and the compiler will allow using it, do 
 not implement it and the compiler will error if you try to use 
 it. In the extreme case, this means we can use a D compiler 
 without any runtime and it will gracefully degrade to a reduced 
 subset.
Yes, this is very similar to what I've been thinking, with regard to having the runtime inform the compiler about what features of the language are and aren't supported by the imported runtime. I've already implemented something very similar to that for `ModuleInfo`, `TypeInfo`, and `Exception` and that code in the compiler today. https://github.com/dlang/dmd/pull/7395 https://github.com/dlang/dmd/pull/7768 https://github.com/dlang/dmd/pull/7799 https://github.com/dlang/dmd/pull/7786 It allows the compiler to use design by introspection to determine what has and hasn't been imported, and generate code accordingly, or emit an error if something is required but can't be found. It's no different than how we work with code and imported modules in our editor. For example, if you write a call to a function, but you failed to import it; the compiler emits an error telling you it can't be found. If you try to use a language feature that isn't implemented and imported from druntime, the compiler can let you know, at compile-time, that you're trying to utilize a feature that doesn't have an implementation. But, it's more powerful than that. If the code passing through the compiler isn't utilizing certain features like `ModuleInfo`, `TypeInfo` and `Exceptions`, then there's no reason for the compiler to complain if that stuff doesn't exist in the imported runtime. That's what the above pull requests fixed. Prior to those PRs we had to implement a bunch unused boilerplate just to get a build, and none of that code ever ended up in the resulting binary anyway. Ridiculous. Best of all, Walter seemed to like it, which was a significant morale boost. https://github.com/dlang/dmd/pull/7395#issuecomment-349200847 Mike
Mar 21
prev sibling next sibling parent sarn <sarn theartofmachinery.com> writes:
On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:
 Other questions regarding this topic: how separate should it be 
 from druntime?
 Should this be a separate library, like a druntime-lite? or is 
 the goal to allow
 druntime to be used for a bare metal target? Also, should this 
 only be betterC
 or should it support regular D as well?
I'm honestly not sure what the scope of the project is, but one option is to make the normal D runtime work with Newlib or some other portable libc for embedded systems. That would create a lot of possibilities (even if not all D runtime features are supported). It would also help define the minimal requirements for the D runtime, making pure-D bare metal easier. BTW, PowerNex is a cool project.
Mar 21
prev sibling next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:

 I am planning on applying to GSoC but I'm having a bit of a 
 problem trying to
 decide on what I should do. My goal is to try and do something 
 that I will
 personally use in PowerNex and it is something that will help 
 the whole D
 community, this is why I am looking at dmd and betterC topics.
Hi Dan, I've been a fan of your work, benefit greatly from your Arch Linux contributions, and admire what you've accomplished so far. ( ̄^ ̄)ゞ <-- That's supposed to be a salute. I have been dreaming about this kind of work for years and did a lot of work in the 2017~2018 time-frame to move it along, but opportunity cost forced me to, at least temporarily, to focus my efforts elsewhere. You're probably a much better D programmer and software engineer than I am, but I believe I can help you quite a bit in your quest if you're willing to engage me and hear me out. Here are a few suggestions I have that I think will have the most beneficial impact on your existing goals and D as a whole # Suggestion 1 - Replace Runtime Hooks with Templates IMO the most important thing you can do to help D become more scalable to bare-metal-like platforms is to replace the runtime hooks [1] with templates. I think this is what Johannes was alluding to in his reply. This will not only remove the runtime/artificial dependency on `TypeInfo` (and therefore a dependency on classes as a whole), but it will also make druntime a "header-only" library that only requires libc as a dependency. Since -betterC only imports druntime, but doesn't link to it, it also will expand D language feature support for -betterC (though if this work were done, -betterC could even become obsolete). See my description at [2] and ask away if you'd like more details. I'm not really qualified to mentor such a project, but I believe I can significantly reduce the barrier to entry should you wish to pursue this. # Suggestion 2 - Create a No-dependency D Library There are a lot of great utilities in Phobos that do not require the druntime (e.g. std.meta, std.traits, std.conv, and probably more). What we need is a library that has NO DEPENDENCIES WHATSOEVER that can be imported into druntime and DMD to aid in their development. Manu hinted on this with a proposal for `core.traits` [3], but I don't think we should stop there, and I had more to say about in a bit of a passionate brainstorm [4]. This no-dependency-library, by its inherent nature, would bring a lot of the great features of Phobos and D to all kinds of programming, including bare-metal-like platforms. It would also make DMD and druntime development much more productive and idiomatic D. # Suggestion 3 - Port the Entire Runtime to ARM Cortex-M, RISCV, X64, {Insert your favorite hardware architecture} IMO, Suggestion 1 and 2 are prerequisites for this, but if you really want to, try to get druntime, in its entirety, ported to a bare-metal platform of your choice. You'll have to find some way to implement dynamic memory allocation and threads, but once that is done, I think it should be possible. Though I recommend making it easy on yourself and implement Suggestion 1 or 2 first. # Suggestion 4 - Replace Software Building Blocks with Implementations in D. By software building blocks, I mean `memcpy`, `memcmp`, `malloc`, `free` etc. This is a very unpopular topic, but if you were to work on Suggestion 1 and 2, you'd likely see this as a practical and worthwhile goal. If you're like most, however, you're probably thinking this is a waste of resources, so I'll defer elaborating on this for now. Though, if you do happen to see what I see, go for it. I am by no means an expert in DMD and druntime, but I did stumble around enough in the code in the 2017~2018 time-frame to know how to get started. I'm not qualified to mentor such a project but I do believe I can help reduce the barrier to entry. Let me know if you're interested in engaging with me and I'll do my best to help. You can reach me on Slack under the handle JinShil, and I think you're clever enough to get my e-mail through the same handle at GitHub, should you choose to contact me that way. Or we can just go nuts in this thread if you'd like. Good Luck! Mike [1] - Runtime Hooks on the Wiki - https://wiki.dlang.org/Runtime_Hooks [2] - Info on Replacing Runtime Hooks with Template - https://forum.dlang.org/post/hequvmostuhwnvcgrcge forum.dlang.org [3] - Move `std.traits` to druntime - https://forum.dlang.org/post/mailman.6241.1546722791.29801.digitalmars-d puremagic.com [4] - Me Dreaming About a More Scalable D - https://forum.dlang.org/post/dvbxqrcmyltgnikxdweb forum.dlang.org
Mar 21
parent reply Dan Printzell <xwildn00bx gmail.com> writes:
On Friday, 22 March 2019 at 02:03:05 UTC, Mike Franklin wrote:
 Hi Dan,
Hey Mike, Sorry about the late response, I had to finish a school assignment and I wanted to really go through everything you linked.
 I've been a fan of your work, benefit greatly from your Arch 
 Linux contributions, and admire what you've accomplished so 
 far.  ( ̄^ ̄)ゞ  <-- That's supposed to be a salute.
Thanks :)
 # Suggestion 1 - Replace Runtime Hooks with Templates
This is something I would like to do. As this would help with both writing a new bare metal runtime and it could help the current druntime. I can see that this idea would probably be the easiest to plan for as you could plan for it to be like X amount of runtime symbols per GSoC milestone, or something simular to this. This could be better than writing a new bare metal runtime as it would decrease the amount of code needed to get something to at least compile. I still have to think about if there is a smart way to allow "druntime-lite" to be used as a foundation for druntime. Because if the new runtime would only be a separate runtime it could become a annoyance if you need to update two runtimes for each runtime change.
 # Suggestion 2 - Create a No-dependency D Library
One question that need to be decided for this type of library would be where should it live inside the D ecosystem, and would this mean that the algorithm would live in two places, one copy inside phobos and one copy inside this library. It would be nice to try and design a module runtime, but the question is how this could be limited into something that I could do in about 3 months, with clearly defined milestones.
 # Suggestion 3 - Port the Entire Runtime to [...]
Yes I agree, this would require both suggestions or at least suggestion 2. The reason why I have not ported druntime to PowerNex yet is because I do not want to sit and add tons of stub & wrapper functions to get it to link. This would be a future goal and not a GSoC goal.
 # Suggestion 4 - Replace Software Building Blocks with
I guess I'm not like most (after all I'm writing my own OS from scratch :P), because I see the value of having D implementation of all these functions. Basically all bare metal code require these function in one way or another. I would say that these are a requirement for a bare metal runtime.

I've started to structure everything in my proposal draft to try 
and
get a better more clear ideas of all the ideas that have been
discussed so far. From one of your links I found this post[1] 
chain
which I felt gave a lot of good information when planning this
proposal, thanks for the links :).

 Let me know if you're interested in engaging with me and I'll 
 do my best to help.  You can reach me on Slack under the handle 
 JinShil, and I think you're clever enough to get my e-mail 
 through the same handle at GitHub, should you choose to contact 
 me that way.  Or we can just go nuts in this thread if you'd 
 like.
Sure I would love to talk more about ideas. For me it doesn't really matter where because I read / lurk basically everywhere. But maybe better to do it here on the forum in case other people have comments and ideas. [1] https://forum.dlang.org/post/mvbxrrpxquvsdjfvfhkc forum.dlang.org
Mar 24
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Sunday, 24 March 2019 at 23:26:05 UTC, Dan Printzell wrote:

 # Suggestion 1 - Replace Runtime Hooks with Templates
This is something I would like to do. As this would help with both writing a new bare metal runtime and it could help the current druntime. I can see that this idea would probably be the easiest to plan for as you could plan for it to be like X amount of runtime symbols per GSoC milestone, or something simular to this. This could be better than writing a new bare metal runtime as it would decrease the amount of code needed to get something to at least compile.
Indeed! There are some dragons to slay, though... 1) This will require familiarizing oneself with both DMD and druntime, as both will need to be modified simultaneously. If you intend to make pull requests to these repositories, please be aware that there currently isn't a great way to test two interdependent pull requests that reside in both DMD and druntime. I recommend the following procedure, but there may be a better way: 1. Add the template to druntime with unittests to verify the implementation. This PR should be merged first. 2. Modify DMD to lower expressions to the new template. Assuming there is already sufficient test cases added to the test suite, this should require additional tests; the real test is to ensure nothing breaks. This PR assumes 1. is already merged. 3. Submit a PR to druntime to remove the old runtime hook. 2) I described this in more detail in the last paragraph here (https://forum.dlang.org/post/hequvmostuhwnvcgrcge forum.dlang.org), that the compiler is currently lying to us about ` safe`, `pure`, and `nothrow` guarantees. When the runtime hook is converted to a template, the template will go through the semantic phase in the compiler, and the author will be forced to suddenly be honest. It could be quite difficult to make this work without breaking code or without hackish workarounds to avoid said breakage. As you alluded to, this can be done incrementally. It may be too large a project to tackle all of druntime, so maybe a good GSoC project would be, for example, to just get all array features finished. I recommend not worrying about getting things to work with -betterC at first; it just adds additional complexity and controversy. If you can first succeed without -betterC, you can then make additional changes to make the templates work in -betterC. If you submit PRs to druntime and DMD, this will also make the PR review go much smoother.
 I still have to think about if there is a smart way to allow
 "druntime-lite" to be used as a foundation for druntime. 
 Because if
 the new runtime would only be a separate runtime it could 
 become a
 annoyance if you need to update two runtimes for each runtime 
 change.
Annoying indeed! I don't think it's necessary to have two different runtimes with duplicate implementations in each if druntime is modularized and the dividing lines between modules are well thought out. Having worked with DMD and druntime a little and having given this some thought over the years and recently, I'm feeling more confident in something like the following: 1) - No-dependency D library (Basically, Suggestion 2 below) - I have more to say about this below. 2) - Level 0 druntime module(s) - Imports (1), but additionally adds compiler intrinsics and runtime features that don't require dynamic memory heap memory allocation. I envision these runtime features to be common to both bare-metal and OS platforms. 3) - Level 1 druntime module(s) - Imports (1) and (2), but additionally adds runtime features that require dynamic heap memory allocation and potentially runtime initialization and deinitialization, which seem to currently require dynamic heap memory allocation. This would still be quite suitable for some, if not most, bare-metal platforms. 4) - Level 2 druntime module(s) - Imports (1), (2), and (3), but finishes off the runtime with classes, GC, and threads. This could still be used for some bare-metal platforms, but would basically require implementing a mini OS inside the runtime itself. For OS platforms, one just needs to tie into the OS implementations as it is currently done. There are probably hundreds of different ways to organize this. For example, the compiler intrinsics and lowerings could be separate modules/libraries, threads and fibers could also be their own modules/libraries, etc.. The basic idea, though, is that users, and the compiler itself, can import just the language features they need or want, in a pay-as-you-go fashion. Also, with design by introspection, even imported modules can make decisions at compile-time about what is or isn't needed as is currently done with `ModuleInfo`, `TypeInfo`, and `Throwable`. It's important to do this tastefully, however, as it could easily turn into a monstrosity if taken too far. This still needs a lot more thought and work, but that's where I'm currently at. I'm just brainstorming, anyway. I'm not sure how something like this would integrate into the existing ecosystem. A good strategy that leverages incremental change will probably find more success. I think Suggestion 2 is a good start. It would help set a precedent and test the palatability for more modularity in the future.
 # Suggestion 2 - Create a No-dependency D Library
One question that need to be decided for this type of library would be where should it live inside the D ecosystem, and would this mean that the algorithm would live in two places, one copy inside phobos and one copy inside this library.
Yes, there are definitely architectural challenges here. And then, worst of all, you'd have to figure out what to name it. Deimos is taken, so if my astronomy is correct, we're out of moons. I suggest it be its own library independent from Phobos, druntime, and DMD; I think that would be the most scalable. Phobos, druntime, and DMD could all privately import and forward to it, and users wouldn't even know anything happened. It could also be published as a separate library on DUB for -betterC and bare-metal programmers to utilize without having to import Phobos and druntime. DUB could actually be utilized to great effect for all of D development if it could just be made into something that the entire community could get behind. In the end, DMD is just another D program, and Phobos and druntime are just D libraries.
 It would be nice to try and design a module runtime, but the 
 question
 is how this could be limited into something that I could do in 
 about 3
 months, with clearly defined milestones.
Understood. There's an awful lot to sort out here. I suggest just focussing on few modules from Phobos and/or druntime (e.g. std.meta, std.traits, core.attribute, etc...). Create a DUB library and GitHub repository for the GSoC project. After GSoC keep working on it. It will still be quite useful to -betterC and bare-metal programmers. In the longer-term, it could eventually be integrated into the larger D ecosystem and hopefully transferred to the D Language Foundation for inclusion in Phobos, druntime, and DMD and continued maintenance. I'm still not sure what that looks like, but regardless I think it is a worthwhile project in and of itself.
 # Suggestion 3 - Port the Entire Runtime to [...]
Yes I agree, this would require both suggestions or at least suggestion 2. The reason why I have not ported druntime to PowerNex yet is because I do not want to sit and add tons of stub & wrapper functions to get it to link. This would be a future goal and not a GSoC goal.
Yeah, Suggestion 1 should significantly reduce, if not eliminate, the need for all of those hacking workarounds that are currently required just to get a build. Take a look at the changelog for 2.079, though, to see what improvements you might be able to make today to your bare-metal runtime - https://dlang.org/changelog/2.079.0.html#minimal_runtime
 # Suggestion 4 - Replace Software Building Blocks with
I guess I'm not like most (after all I'm writing my own OS from scratch :P), because I see the value of having D implementation of all these functions. Basically all bare metal code require these function in one way or another. I would say that these are a requirement for a bare metal runtime.
Good! We can continue to utilize libc's implementations, but then we don't get all of the advantages (design-by introspection, CTFE, memory safety, etc...) that could potentially improve performance and reliability of the software as a whole. After the runtime hooks are implemented as templates, I would hope that we could one day use something like `memcpy(T)(scope T* dest, scope const T* source) safe pure nothrow` instead of `mempcy(void* dest, const void* source, size_t n)` in druntime's implementations. If it's not something you're interested in, I hope there are others that can see the value in such a thing and will find it to be a rewarding project for themselves. I think it would be a great project for a newcomer to D as it doesn't require mastery of the language, and the semantics of those building-blocks are well-understood. We won't see the benefit of using these D implementations, though, until the runtime hooks are converted to templates. Looking forward to hearing more about whatever it is you decide. Mike
Mar 25
parent reply Dan Printzell <xwildn00bx gmail.com> writes:
On Tuesday, 26 March 2019 at 04:15:54 UTC, Mike Franklin wrote:
 Looking forward to hearing more about whatever it is you decide.
I decided on doing the 'replacing runtime hooks with template' idea. After re-read everything you posted and took some time to think about it I concluded that this is the most sane idea for the time frame of a GSoC. I feel that this will be able to give the most improvement to the whole language, without the possibility of creating a library that may not be used or by trying to design a new druntime. You can also classify this idea as a continuation of previous work, that for example you have done, with replacing hooks with templates. And as you said wrapping/reimplementing the mem{cpy,set,move} functions will make more sense to do later on when more of these runtime hooks have been replaced with templates. I'm currently in the process of writing the first draft of this proposal and will post it here and on the GSoC application page, as soon as I get it into a more finished state, which should hopefully be later today. - Dan
Mar 27
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 28 March 2019 at 04:43:31 UTC, Dan Printzell wrote:
 On Tuesday, 26 March 2019 at 04:15:54 UTC, Mike Franklin wrote:
 Looking forward to hearing more about whatever it is you 
 decide.
I decided on doing the 'replacing runtime hooks with template' idea. After re-read everything you posted and took some time to think about it I concluded that this is the most sane idea for the time frame of a GSoC.
Thumbs up! This will be great for D in many ways. I wish you success. If there's anything I can do to help in any way, let me know. My skills and resources are quite limited, but I'll do my best. Mike
Mar 28
parent reply Dan Printzell <xwildn00bx gmail.com> writes:
On Thursday, 28 March 2019 at 13:00:51 UTC, Mike Franklin wrote:
 Thumbs up!  This will be great for D in many ways.  I wish you 
 success.  If there's anything I can do to help in any way, let 
 me know.  My skills and resources are quite limited, but I'll 
 do my best.
If you want to you (others are also welcomed) can look through my draft to make sure I have not written anything stupid and possibly suggest improvements. https://docs.google.com/document/d/15ba1vC1T7n2vX8kvQVikOj1Ip1-mt-zyiEwd0OYBoQA/edit?usp=sharing
Mar 29
next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 29 March 2019 at 15:35:10 UTC, Dan Printzell wrote:

 If you want to you (others are also welcomed) can look through
 my draft to make sure I have not written anything stupid and
 possibly suggest improvements.

 https://docs.google.com/document/d/15ba1vC1T7n2vX8kvQVikOj1Ip1-mt-zyiEwd0OYBoQA/edit?usp=sharing
I'm a bit rushed at the moment but this is what I have right now: I wouldn't put too much emphasis on betterC for the following reasons. 1. To make these templated runtime hooks work in betterC you will need to make it templates-all-the-way-down. That can easily turn into a huge PR that turns way too much of druntime into templates, and that will significantly reduce the chance of the PR being accepted. 2. Many of the current runtime hooks throw `Exception`s. `Exception`s are classes, so they can't be used in betterC. You *might* be able to work around that by replacing them with `assert`ions, but then again you need to decide if they should be `assert`ions for all builds or just betterC. If just betterC, then you'll have to `version(SupportsExceptions)` and provide two different implementations. I tried that and it turned into a headache especially when trying to dynamically create meaningful error messages. I recommend saving yourself the trouble for the first round; don't worry about betterC until you get the runtime hook working with non-betterC builds. You can revisit it with a followup PR to address betterC, after the non-betterC changes get accepted. 3. This work has the potential to make betterC obsolete. In general, the way betterC works, is it imports druntime, but doesn't link to it. Users can already achieve a similar result by building and linking separately especially with the work done in 2.079. Once all runtime hooks are replaced with templates, druntime becomes more of an "header-only" library for lack of better term. You may be able to achieve a similar result as betterC by simply not using certain features of D. If that day arrives, there will no longer be much of a reason for betterC. There are other benefits beyond betterC. 1. Lucia Cojocaru was the first to do some of this work; she did `__equals` and `__cmp`, I think. In her presentation about it at DConf2017 (https://www.youtube.com/watch?v=endKC3fDxqs), she demonstrated a 30% increase in runtime performance. It seems, at least to me, that we should be able to expect similar results by refactoring other runtime hooks. 2. Once converted to templates, we then have the ability to do compile-time microoptimizations base on type, size, alignment, and potentially other information available through design-by-introspection for additional exploitation. 3. As mentioned before, the compiler is currently bypassing all of D's compiler guarantees for `nothrow`, ` safe`, and `pure`. This, IMO, is a bug in the compiler. After converting runtime hooks to templates, the compiler will enforce these constraints, having the compiler automatically audit the implementation. That may prove to find and fix bugs we aren't currently aware of, just as Walter recently found and fixed a stack corruption bug when enabling -dip1000 in Phobos. The result is a verifiably more reliable implementation, one in which future bugs can't even be introduced because they won't build. 4. All the information the implementations need is available at compile-time. There is no reason to utilize runtime type information `TypeInfo`. `TypeInfo` is a class, so the current implementation has an intrinsic dependency on classes. Classes have an intrinsic dependency on the GC and other features. Replacing the runtime hooks with templates that parameterize on type, remove that dependency making the runtime more pay-as-you-go for implementations, like, but not limited to, bare-metal, that don't want or don't have the resources for the runtime overhead of classes and the GC. There still may be other classes that the implementation depends on (e.g. `Exception`s) but those dependencies could be refactored in followup PRs to completely eliminate the dependency on classes. That will bring more of D's features to the aforementioned implementations so users. Users will have a betterD rather than a betterC. Challenges to overcome. 1. Because ` safe`, `nothrow`, and `pure` will suddenly be enforced, the implementation will have to be refactored to abide by the compiler's enforcement of those constraints. Here are a few things I've learned. a. You may be able to use `pureMalloc` and friends (https://github.com/dlang/druntime/blob/29f495b5b3571484bcf4d0af2fe211d6b7d86830/src/core memory.d#L862-L891) to overcome the purity constraint. b. You should be able to use `trusted` to workaround the ` safe` constraint. c. You may be able to use `assert`ions instead of throwing `Exception`s to workaround the `nothrow` constraint. I have private e-mails from Walter approving, at least superficially, these techniques should you need them to gain leverage in the review process. It's not a guarantee, but it sounds promising. 2. One of the things Lucia and myself have not yet done is to measure the impact of converting these runtime hooks to templates on compile-time and template bloat. A more thorough and professional approach would be to provide data with each PR with such measurements. That's all I have time for at the moment. I'll make another pass later. Mike
Mar 29
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 29 March 2019 at 15:35:10 UTC, Dan Printzell wrote:
 On Thursday, 28 March 2019 at 13:00:51 UTC, Mike Franklin wrote:
 Thumbs up!  This will be great for D in many ways.  I wish you 
 success.  If there's anything I can do to help in any way, let 
 me know.  My skills and resources are quite limited, but I'll 
 do my best.
If you want to you (others are also welcomed) can look through my draft to make sure I have not written anything stupid and possibly suggest improvements. https://docs.google.com/document/d/15ba1vC1T7n2vX8kvQVikOj1Ip1-mt-zyiEwd0OYBoQA/edit?usp=sharing
I spent the day today trying to revive my work on `_d_arraycast` (https://github.com/dlang/dmd/pull/9516). I have not worked on D in a while, and I didn't realize that the druntime PR (https://github.com/dlang/druntime/pull/2264) was pulled, and I closed the original DMD PR (https://github.com/dlang/dmd/pull/8531). Since the druntime PR was merged, I think I'm on the hook to follow through and finish the job. I encountered an interesting issue that you may encounter in the GSoC project, so beware the following. Since the template will be processed through the semantic phase of the compiler, it will also be subject to CTFE. That's a good thing, but with `_d_arraycast` I discovered some casts are not supported in CTFE. The error being emitted is "reinterpreting cast from `void[]` to `Array*` is not supported in CTFE". I'm not sure what to do about it and I asked for some ideas. Perhaps I may be able to modify the runtime template with some `static if (__ctfe)` as a workaround. Anyway, it will be good to be aware of this issue. I had not previously anticipated it. Mike
Mar 30
prev sibling next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 29 March 2019 at 15:35:10 UTC, Dan Printzell wrote:
 If you want to you (others are also welcomed) can look through
 my draft to make sure I have not written anything stupid and
 possibly suggest improvements.

 https://docs.google.com/document/d/15ba1vC1T7n2vX8kvQVikOj1Ip1-mt-zyiEwd0OYBoQA/edit?usp=sharing
I've been spending some time trying to think about how to articulate the benefits of this work to D, and its community. For some reason, I'm finding it difficult to articulate, but I'll try. Hopefully, it will help you write a stronger proposal. Andei recently "hit the nail on the head" with this post - https://forum.dlang.org/post/q7j4sl$17pe$1 digitalmars.com
 Oh, and druntime must go.

 The whole distinction between the runtime library and the 
 standard library is clowny and has more to do with poorly 
 ported tradition from other languages, than with anything else.

 We need one standard library that is entirely pay-as-you-go 
 (e.g. empty main() means no library file is ever needed for 
 linking) and that offers an opt-in continuum.
IMO, that is *exactly* what we need. Some of D's features are extremely powerful assets in some problem domains (e.g. PC application software, Web services, etc.), but an unfortunate liability in other problem domains (e.g. bare-metal, resource-constrained applications, etc.) To make D scalable to all of these domains, it needs to be more pay-as-you-go and provide an opt-in continuum, as Andrei so eloquently stated. Converting the runtime hooks to templates provides an excellent first step towards reaching that goal. There are immediate benefits (I've already mentioned these, so sorry for the repetition.) * Some language features will no longer depend on `TypeInfo` (and therefore, classes, and the GC) so D will immediately become more scalable to bare-metal and other resource-constrained platforms that want features like arrays, but don't want the overhead classes and the GC (at least initially while they are bringing up a board, OS, or porting the D language to a new platform). * There is evidence in Lucia Cojocaru's work that this work has the potential to provide significant performance increases. * The new templates will finally run through the full semantic phase of the compiler enforce compiler guarantees such as memory safety, purity, etc. * This work makes druntime more of a "header-only" library. This is helpful in cross-compiling scenarios (e.g. microcontroller programming) because one will not need to cross-compile the runtime ahead of time to use those D features implemented by the templates. But the benefits will not stop there. Once this initial work is completed it will open up new opportunities for more performance optimizations and the aforementioned pay-as-you-go, opt-in continuum for expanding D to more platforms and problem domains. * The new templates will allow developers to make better decisions at design-time (design by introspection) and compile-time to make additional performance optimizations to the template implementations. I suspect significant exploitations will be discovered and implemented over time. * The templates can be further refactored with versioning and design by introspection to remove additional dependencies (i.e. `Exception`s). This will further bring the aforementioned pay-as-you-go, opt-in continuum to reality. Indirect benefits include more D features available to betterC, and consequently, more Phobos features available to betterC. In the longer term, however, if the pay-as-you-go, opt-in continuum becomes a reality, there will no longer be any reason for betterC; one just chooses not to use the D features they're not willing to pay for. * With the runtime hooks converted to templates, software building blocks like `memcpy`, `memcmp`, `malloc`, etc. can be re-written in D, extending the aforementioned design by introspection, compile-time optimizations, memory safety and other compiler enforced guarantees all the way down the call stack for fine-tuned, high-performance, reliable software. An exploration of `memcpy` in D can be found at https://github.com/JinShil/memcpyD * As the pay-as-you-go, opt-in continuum becomes more and more a reality, users will find D an attractive alternative for all kind of bare-metal, embedded, IoT, microcontroller, etc. problem domains competing with the likes of C and Rust that currently dominate those domains. One of the reasons Rust is taking off in the embedded space is due to it's "minimal runtime" philosophy. D can have that too, and converting the runtime hooks to templates is an important step in that direction. To keep the project within scope and on schedule, I recommend a direct translation first (to whatever extent possible). Avoid the temptation to do too much refactoring. Remember, you can always followup with more PRs to clean things up, optimize, whatever if time and energy remain. I hope I'm being helpful. Keep in mind I've never participated in GSoC or anything like it. I don't know much about writing proposals like this. Don't feel compelled to include any of it in your proposal. I'm also no expert with this stuff. I'm just someone who spent some time on it in 2018. I didn't submit many pull requests because I had to learn so much first about what needed to be done, and how. I'm trying to pass that on to you and others. I hope even after GSoC this kind of work will continue. I'll take another look at your proposal in a few days, assuming you do some polishing between now and then. Mike
Mar 31
parent Dan Printzell <xwildn00bx gmail.com> writes:
On Sunday, 31 March 2019 at 14:05:23 UTC, Mike Franklin wrote:
 To keep the project within scope and on schedule, I recommend a 
 direct translation first (to whatever extent possible).  Avoid 
 the temptation to do too much refactoring.  Remember, you can 
 always followup with more PRs to clean things up, optimize, 
 whatever if time and energy remain.
Yes this is what I'm targeting, a as direct translation as possible.
 I hope I'm being helpful.  Keep in mind I've never participated 
 in GSoC or anything like it.  I don't know much about writing 
 proposals like this.  Don't feel compelled to include any of it 
 in your proposal.
You are being really helpful <3. You mentioned topics and ideas that I forgot about. I forgot about Lucia Cojocarus talk, and I have now added that to the proposal.
 I'm also no expert with this stuff.  I'm just someone who spent 
 some time on it in 2018.  I didn't submit many pull requests 
 because I had to learn so much first about what needed to be 
 done, and how.  I'm trying to pass that on to you and others.
Thanks :)
 I hope even after GSoC this kind of work will continue.
I can not promise anything as I don't know what my future holds, but currently I could see myself continuing this work after GSoC.
 I'll take another look at your proposal in a few days, assuming 
 you do some polishing between now and then.
Sounds good, with all the comments from you and Kayomn I realized that I had to improve most of the text, and refocus a few parts. So the text is kind of messy right now.
Mar 31
prev sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Friday, 29 March 2019 at 15:35:10 UTC, Dan Printzell wrote:

 If you want to you (others are also welcomed) can look through
 my draft to make sure I have not written anything stupid and
 possibly suggest improvements.

 https://docs.google.com/document/d/15ba1vC1T7n2vX8kvQVikOj1Ip1-mt-zyiEwd0OYBoQA/edit?usp=sharing
We've finished the implementation of `_d_arraycast` as a template. I followed essentially the same procedure in your propose. 1) https://github.com/dlang/druntime/pull/2264 - Add new template to druntime with unittests verifying the implementation 2) https://github.com/dlang/dmd/pull/9516 - Update DMD to call the template and remove calls to the previous runtime hook 3) https://github.com/dlang/druntime/pull/2535#event-2241980654 - Remove the no-longer-used, old runtime hook from druntime I made the mistake of planning for betterC in (1). It made things a little too complicated which delayed the implementation and required me to have to do a little back-and-forth with Walter on how to not rock the boat too much. You've already stated you're going with a direct translation. That's good. This at least shows some of the changes that may be required to get a feature working in betterC in a followup PR. It also shows how complicated things can get when considering betterC. I still have to add DMD PR to verify the betterC implementation, but I'm a little on the fence about it. Anyway, in addition to Lucia's PRs... https://github.com/dlang/dmd/pulls?utf8=%E2%9C%93&q=is%3Apr+author%3Asomzzz+ https://github.com/dlang/druntime/pulls?utf8=%E2%9C%93&q=is%3Apr+author%3Asomzzz+ ... it serves as a good case study. Mike
Mar 31
parent Dan Printzell <xwildn00bx gmail.com> writes:
On Monday, 1 April 2019 at 02:59:09 UTC, Mike Franklin wrote:
 ... it serves as a good case study.
Thanks again for all the help and the links :) I'm happy with the proposal now, it took a bit longer than expected but it is much better than the old one. I only have some small nitpicks go though and fix. I will try to submit the proposal tomorrow. Feel free to look at it again if you want. [0] https://docs.google.com/document/d/15ba1vC1T7n2vX8kvQVikOj1Ip1-mt-zyiEwd0OYBoQA/edit?usp=sharing
Apr 06
prev sibling next sibling parent Mike Franklin <slavo5150 yahoo.com> writes:
On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:

I'll attempt to answer your specific questions below.

 For dmd I am primarily interested in implementing DIP1014[3], 
 as this will help
 me with the development of PowerNex. In the past I had to 
 workaround that this
 feature was missing. From the outside is seems like this would 
 not take too long
 to implement, at least not a whole GSoC. But as I'm not too 
 familiar with dmd's
 source code, I only did a few small patches to allow dmd to 
 support PowerNex as
 a target[4].
 So my question for this topic would be: How long time do you 
 guys think that
 this will take to implement, test and get it merged? Is a month 
 about correct?
I'm afraid I haven't looked into this and I don't have anything to share. Sorry.
 For the betterC topic I would be interested in implementing a 
 bare metal D
 runtime[5]. This is something I already have some experience as 
 I have written a
 betterC runtime for PowerNex, and I also used to 
 update/maintain Adam D. Ruppe's
 minimal.zip[6] when I used that as the runtime for PowerNex.
 My question would be: What features should this runtime contain?
As I see it there are 3 different levels: Level-0: No dependencies whatsoever. Level-0 would likely only include utilities, but more features could be added if you chose to import `memcpy`, `memset`, and `memcmp` from libc. However, I would prefer if those software building blocks were implemented in D. See https://github.com/JinShil/memcpyD for an unfinished exploration. Level-0 would not include dynamic heap memory allocation, but could utilize dynamic stack memory allocation. LDC and GDC have alloc out of the box, but DMD doesn't unfortunately, so you might have to hack something together with static arrays and runtime error-checking. See more about this at https://forum.dlang.org/post/yyylugjalnlawjvxaqfp forum.dlang.org Level-1: Imports Level-0, but also adds dynamic heap memory allocation and any D language feature implementations that depend on it. You could import libc for `malloc`, `free`, and friends, or they too could be implemented in D super extra credit. Level-2: Imports Level-1, but also adds threads. Once threads are implemented D features such as thread local storage and the GC become possible. Subsequently, with the GC, you can now implement classes and the rest of druntime. I think you could implement the GC in Level-1, but I haven't thought it through yet. If those three levels were distributed separately, a user could potentially opt into whatever level they desire for their application.
 From the wiki I
 see that basic libc functionality is requested, like the str* 
 functions and
 memory management. From what I can understand, the goals it to 
 have a runtime
 that can be used on multiple targeted environments, without any 
 external
 dependencies.
Yes, no dependencies on anything non-D is my preference. It is possible to replace the facilities currently being leveraged from libc, but will take a lot of work, and a significant project itself. For the short-term, however, we can tolerate a libc dependency, or you could make replacing these facilities your project.
 Other questions regarding this topic: how separate should it be 
 from druntime?
 Should this be a separate library, like a druntime-lite? or is 
 the goal to allow
 druntime to be used for a bare metal target? Also, should this 
 only be betterC
 or should it support regular D as well?
I don't really think about it as being separate from druntime. I see it as just an alternate implementation of druntime; one that is more scalable and pay-as-you-go than what we currently have.
 Depending on the implementation size of DIP1014 and the bare 
 metal D runtime,
 it seems like both could be implemented during a GSoC. But I am 
 probably just
 being time optimistic here.
Sound WAY too optimistic for me, but who am I to judge? I'll restate, briefly, what I said in my other post, that I think there are some fundamental changes to DMD and druntime (e.g. replacing runtime hooks with templates) that should probably be implemented first as prerequisites to all of this, and I can help with that. Mike
Mar 21
prev sibling parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Thursday, 21 March 2019 at 19:50:49 UTC, Dan Printzell wrote:
 Hey everyone!
 I have been meaning to send these questions earlier, but then 
 school happened.

 [...]
How about this - https://forum.dlang.org/thread/uqydrkgfqiehwugsnfxz forum.dlang.org
Mar 23