www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Phobos Action Items

reply Walter Bright <newshound2 digitalmars.com> writes:
Starting a new thread with this, as requested. There are things here for all 
skill levels and time commitments.

1. eliminate all gratuitous use of gc

2. review all Phobos modules for compatibility with ranges - std.zip, for 
example, was done before ranges and does not work with them

3. use -cov to improve code coverage of Phobos unittests

4. make sure every function in Phobos has an example

5. make sure every function in Phobos has Params: and Returns: sections
http://www.digitalmars.com/d/archives/digitalmars/D/Phobos_Documentation_-_call_to_action_258777.html

6. replace std.xml with something we can be proud of that is second to none in 
performance (Robert burner Schadek is mentoring on this 
https://github.com/lodo1995/experimental.xml)

7. compile a list of popular modules in other languages and see what we should
have

8. create a greenthreads module that works like Goroutines

9. create a module that enables code to be run on GPUs (John Colvin is doing 
work on this, ask him how to help!)

10. now that D can interface to C++ templates and exceptions, create the 
interface code to the C++ STL, put it in core.stdcpp

11. review all of Phobos for  safe compatibility - see some of the library 
issues here 
https://issues.dlang.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&component=druntime&component=phobos&keywords=safe&keywords_type=allwords&list_id=209021&query_format=advanced

12. remove dependency on autodecode from Phobos

13. there was recently a thread on Phobos desperately needing a BigDecimal 
module, someone helpfully posted links to a couple of free C versions, I
suggest 
that someone build a D wrapper for one or convert one to D.

14. write articles about D

15. pick a bugzilla issue and work on it 
https://issues.dlang.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&component=druntime&component=phobos&list_id=209020&query_format=advanced

16. this list is incomplete, it is off the top of my head

I've learned to not ask for a gui library :-)
Jun 18 2016
next sibling parent reply Jakob Bornecrantz <wallbraker gmail.com> writes:
On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:

 8. create a greenthreads module that works like Goroutines


Please don't use the term gorourines when describing a bunch of 
library functions. There is more to them then just functions. You 
will need compiler and GC support for the special stack that they 
have.

Cheers, Jakob.
Jun 18 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/18/2016 1:47 PM, Jakob Bornecrantz wrote:

 Please don't use the term gorourines when describing a bunch of library
 functions. There is more to them then just functions. You will need compiler
and
 GC support for the special stack that they have.


I know. I just didn't want to write a detailed list of requirements in what was 
just a bullet point intended as a starting point.
Jun 18 2016
prev sibling next sibling parent Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:

 Starting a new thread with this, as requested. There are things 
 here for all skill levels and time commitments.


There are a good number of things here that are simple to change. 
I might make a "How to Start Contributing to D" article with some 
examples of how to fix some of these.


 4. make sure every function in Phobos has an example

 5. make sure every function in Phobos has Params: and Returns: 
 sections


As a shameless self plug, I have two PRs open to fix some 
lack-luster docs: https://github.com/dlang/phobos/pull/4422 
https://github.com/dlang/phobos/pull/4320
Jun 18 2016
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:

 9. create a module that enables code to be run on GPUs (John 
 Colvin is doing work on this, ask him how to help!)


Now that I'm on my (southern hemisphere) winter break I will be 
working on getting LDC to emit PTX and SPRIV and if I get time 
whatever metal uses to be able to use D as the source language 
for CUDA, OpenCL and Metal respectively. This should hopefully 
enable seamless integration with the use of introspection and 
whatnot.

This would give us a huge advantage, being able to target the 
three major compute APIs.
Jun 18 2016
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/18/2016 6:01 PM, Nicholas Wilson wrote:

 On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:

 9. create a module that enables code to be run on GPUs (John Colvin is doing
 work on this, ask him how to help!)


 Now that I'm on my (southern hemisphere) winter break I will be working on
 getting LDC to emit PTX and SPRIV and if I get time whatever metal uses to be
 able to use D as the source language for CUDA, OpenCL and Metal respectively.
 This should hopefully enable seamless integration with the use of introspection
 and whatnot.

 This would give us a huge advantage, being able to target the three major
 compute APIs.


Yup. Fire away!
Jun 18 2016
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Sunday, 19 June 2016 at 01:01:30 UTC, Nicholas Wilson wrote:

 On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:

 9. create a module that enables code to be run on GPUs (John 
 Colvin is doing work on this, ask him how to help!)


 Now that I'm on my (southern hemisphere) winter break I will be 
 working on getting LDC to emit PTX and SPRIV and if I get time 
 whatever metal uses to be able to use D as the source language 
 for CUDA, OpenCL and Metal respectively. This should hopefully 
 enable seamless integration with the use of introspection and 
 whatnot.

 This would give us a huge advantage, being able to target the 
 three major compute APIs.


This would be awesome. I would love to get my hands on prototypes 
of this, no matter how buggy and unfinished they are at any given 
point.
Jun 19 2016
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 19 June 2016 at 09:50:42 UTC, John Colvin wrote:

 On Sunday, 19 June 2016 at 01:01:30 UTC, Nicholas Wilson wrote:

 On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:

 [...]


 Now that I'm on my (southern hemisphere) winter break I will 
 be working on getting LDC to emit PTX and SPRIV and if I get 
 time whatever metal uses to be able to use D as the source 
 language for CUDA, OpenCL and Metal respectively. This should 
 hopefully enable seamless integration with the use of 
 introspection and whatnot.

 This would give us a huge advantage, being able to target the 
 three major compute APIs.


 This would be awesome. I would love to get my hands on 
 prototypes of this, no matter how buggy and unfinished they are 
 at any given point.


Glad you're enthusiastic! I've made a new thread in general about 
this. All testing appreciated.
Jun 19 2016
prev sibling next sibling parent reply Suliman <evermind live.ru> writes:
 8. create a greenthreads module that works like Goroutines


But we already have fibers, I thought that they are same with 
Goroutines
Jun 18 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/18/2016 10:22 PM, Suliman wrote:

 8. create a greenthreads module that works like Goroutines


 But we already have fibers, I thought that they are same with Goroutines


Fibers are thread local. Goroutines are distributed among fibers and threads, 
and can switch from one thread to another.
Jun 18 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
On Sunday, 19 June 2016 at 05:37:01 UTC, Walter Bright wrote:

 On 6/18/2016 10:22 PM, Suliman wrote:

 8. create a greenthreads module that works like Goroutines


 But we already have fibers, I thought that they are same with 
 Goroutines


 Fibers are thread local. Goroutines are distributed among 
 fibers and threads, and can switch from one thread to another.


Which is inherently suboptimal and is a part of Go marketing 
bullshit not worth spending time on. It also requires heavy 
runtime modifications (because TLS) unless one wants to totally 
screw plain fibers.

Proper action item instead would be providing standard event loop 
and task system in Phobos.
Jun 18 2016
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Sunday, 19 June 2016 at 06:19:34 UTC, Dicebot wrote:

 Which is inherently suboptimal and is a part of Go marketing 
 bullshit not worth spending time on. It also requires heavy 
 runtime modifications (because TLS) unless one wants to totally 
 screw plain fibers.

 Proper action item instead would be providing standard event 
 loop and task system in Phobos.


I'm really not much of a fibers user, but I have read a bit on 
how Chapel handles parallelism and found that very interesting.
Jun 19 2016
parent reply Dicebot <public dicebot.lv> writes:
On Sunday, 19 June 2016 at 13:32:02 UTC, jmh530 wrote:

 On Sunday, 19 June 2016 at 06:19:34 UTC, Dicebot wrote:

 Which is inherently suboptimal and is a part of Go marketing 
 bullshit not worth spending time on. It also requires heavy 
 runtime modifications (because TLS) unless one wants to 
 totally screw plain fibers.

 Proper action item instead would be providing standard event 
 loop and task system in Phobos.


 I'm really not much of a fibers user, but I have read a bit on 
 how Chapel handles parallelism and found that very interesting.


Fibers are more of a concurrency than a parallelism tool. Do you 
have a link to relevant Chapel description? I am not familiar 
with it.
Jun 19 2016
parent reply Lodovico Giaretta <lodovico giaretart.net> writes:
On Sunday, 19 June 2016 at 15:40:09 UTC, Dicebot wrote:

 On Sunday, 19 June 2016 at 13:32:02 UTC, jmh530 wrote:

 On Sunday, 19 June 2016 at 06:19:34 UTC, Dicebot wrote:

 Which is inherently suboptimal and is a part of Go marketing 
 bullshit not worth spending time on. It also requires heavy 
 runtime modifications (because TLS) unless one wants to 
 totally screw plain fibers.

 Proper action item instead would be providing standard event 
 loop and task system in Phobos.


 I'm really not much of a fibers user, but I have read a bit on 
 how Chapel handles parallelism and found that very interesting.


 Fibers are more of a concurrency than a parallelism tool. Do 
 you have a link to relevant Chapel description? I am not 
 familiar with it.


Just found this, although there's probably other material out 
there:

http://chapel.cray.com/tutorials/SC10/M10-4-TaskPar.pdf

Lodovico Giaretta
Jun 19 2016
parent jmh530 <john.michael.hall gmail.com> writes:
On Sunday, 19 June 2016 at 16:21:36 UTC, Lodovico Giaretta wrote:

 Fibers are more of a concurrency than a parallelism tool. Do 
 you have a link to relevant Chapel description? I am not 
 familiar with it.


 Just found this, although there's probably other material out 
 there:

 http://chapel.cray.com/tutorials/SC10/M10-4-TaskPar.pdf

 Lodovico Giaretta


Thanks, that would be what I would have directed him to. They 
kind of use task parallelism in a way similar to concurrency. 
There is more here
http://chapel.cray.com/docs/latest/users-guide/taskpar/taskParallelismOverview.html
as well as in the language spec
http://chapel.cray.com/docs/latest/language/spec.html
Jun 19 2016
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Sunday, 19 June 2016 at 05:37:01 UTC, Walter Bright wrote:

 On 6/18/2016 10:22 PM, Suliman wrote:

 8. create a greenthreads module that works like Goroutines


 But we already have fibers, I thought that they are same with 
 Goroutines


 Fibers are thread local. Goroutines are distributed among 
 fibers and threads, and can switch from one thread to another.


Alright. What is the goal here ?

First, Fiber are doing some synchronized work. This must go. They 
can't be exchanged across thread safely so this is only make 
performance worse without any benefit.

Second, while go move goroutine from one thread to another, the 
go scheduler goes to great length to avoid doing it. There is all 
kind of bad side effects coming from this and you don't want 
them. HHVM does not move requests from one thread to another.

It is to be noted that go to not provide any control over 
scheduling, while D does, doesn't have thread local storage and 
the type system doesn't provide any sharing constraints.

This is important because we do. If we want to make that work, 
there are implications. First, it become impossible to cache the 
TLS segment address across function calls as any function call 
can mean the thread has changed. Even with that deoptimization 
(that all code will need to pay, not only the code using these 
goroutines) that means the type system is now completely broken. 
Any goroutine can take the address of a thread local, store it on 
the stack and voila, it is now shared across threads. So that 
means you need in addition DIP30 or something alike to make sure 
that the only things passed to the goroutine is shared. In 
addition, you want to make sure these do not touch the TLS, so 
you either need some kind of new attribute, like pure_tls, or 
just make them pure altogether (which would reduce their utility 
greatly).

TL;DR : It sucks, there are all kind of bad side effects. The 
only reason go does it, is because they goroutine everything and 
can only have an adaptive strategy as this is done in the runtime.

Now on the rant. We don't need more stuff. We have plenty of 
stuff. We have plenty of unfinished stuff and this is what we 
should focus on. We have plenty of stuff that try to be 2 things 
at once and does both badly. We don't need anymore of this.

Just remove all synchronization from fiber and be done with it.
Jun 18 2016
next sibling parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Sunday, 19 June 2016 at 06:28:45 UTC, deadalnix wrote:

 On Sunday, 19 June 2016 at 05:37:01 UTC, Walter Bright wrote:

 On 6/18/2016 10:22 PM, Suliman wrote:

 8. create a greenthreads module that works like Goroutines


 But we already have fibers, I thought that they are same with 
 Goroutines


 Fibers are thread local. Goroutines are distributed among 
 fibers and threads, and can switch from one thread to another.


 Alright. What is the goal here ?

 First, Fiber are doing some synchronized work. This must go. 
 They can't be exchanged across thread safely so this is only 
 make performance worse without any benefit.

 Second, while go move goroutine from one thread to another, the 
 go scheduler goes to great length to avoid doing it. There is 
 all kind of bad side effects coming from this and you don't 
 want them. HHVM does not move requests from one thread to 
 another.

 It is to be noted that go to not provide any control over 
 scheduling, while D does, doesn't have thread local storage and 
 the type system doesn't provide any sharing constraints.

 This is important because we do. If we want to make that work, 
 there are implications. First, it become impossible to cache 
 the TLS segment address across function calls as any function 
 call can mean the thread has changed. Even with that 
 deoptimization (that all code will need to pay, not only the 
 code using these goroutines) that means the type system is now 
 completely broken. Any goroutine can take the address of a 
 thread local, store it on the stack and voila, it is now shared 
 across threads. So that means you need in addition DIP30 or 
 something alike to make sure that the only things passed to the 
 goroutine is shared. In addition, you want to make sure these 
 do not touch the TLS, so you either need some kind of new 
 attribute, like pure_tls, or just make them pure altogether 
 (which would reduce their utility greatly).

 TL;DR : It sucks, there are all kind of bad side effects. The 
 only reason go does it, is because they goroutine everything 
 and can only have an adaptive strategy as this is done in the 
 runtime.

 Now on the rant. We don't need more stuff. We have plenty of 
 stuff. We have plenty of unfinished stuff and this is what we 
 should focus on. We have plenty of stuff that try to be 2 
 things at once and does both badly. We don't need anymore of 
 this.

 Just remove all synchronization from fiber and be done with it.


I agree. BTW, what do you think about improving the compiler's 
understanding of Fibers (AFAIU they're purely a runtime thing), 
so e.g. they're usable at CTFE. See this for more details:

http://lists.llvm.org/pipermail/llvm-dev/2016-June/100838.html
Jun 18 2016
parent deadalnix <deadalnix gmail.com> writes:
On Sunday, 19 June 2016 at 06:37:03 UTC, ZombineDev wrote:

 I agree. BTW, what do you think about improving the compiler's 
 understanding of Fibers (AFAIU they're purely a runtime thing), 
 so e.g. they're usable at CTFE. See this for more details:

 http://lists.llvm.org/pipermail/llvm-dev/2016-June/100838.html


I saw that but I'm unclear what is the cost benefit at this 
point. Which is an elaborate way to say I don't know.
Jun 19 2016
prev sibling parent qznc <qznc web.de> writes:
On Sunday, 19 June 2016 at 06:28:45 UTC, deadalnix wrote:

 On Sunday, 19 June 2016 at 05:37:01 UTC, Walter Bright wrote:

 On 6/18/2016 10:22 PM, Suliman wrote:

 8. create a greenthreads module that works like Goroutines


 But we already have fibers, I thought that they are same with 
 Goroutines


 Fibers are thread local. Goroutines are distributed among 
 fibers and threads, and can switch from one thread to another.


 Alright. What is the goal here ?


Do people really want the threads? You don't get them with gccgo 
either, do you?

Isn't the common request more about the channels Go provides?
Jun 19 2016
prev sibling next sibling parent reply Andrea Fontana <nospam example.com> writes:
On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:


 6. replace std.xml with something we can be proud of that is 
 second to none in performance (Robert burner Schadek is 
 mentoring on this https://github.com/lodo1995/experimental.xml)


I hope it will support html and xpath too.

Andrea
Jun 20 2016
parent Lodovico Giaretta <lodovico giaretart.net> writes:
On Monday, 20 June 2016 at 07:48:49 UTC, Andrea Fontana wrote:

 On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:


 6. replace std.xml with something we can be proud of that is 
 second to none in performance (Robert burner Schadek is 
 mentoring on this https://github.com/lodo1995/experimental.xml)


 I hope it will support html and xpath too.

 Andrea


The low-level parser is explicitly written to support any format 
which resembles XML, and will have no problem handling HTML. 
Regarding higher level APIs: with validations turned on, HTML 
would surely be rejected (it breaks many XML well-formedness 
constraints); with validations turned off, it may or may not 
work, because higher level APIs implicitly assume that their 
input is well-formed.

As for XPath, I probably won't have time to work on it during 
GSoC, but if everything goes well I may work on it after the end 
of the project.

Lodovico Giaretta
Jun 21 2016
prev sibling next sibling parent qznc <qznc web.de> writes:
On Saturday, 18 June 2016 at 20:04:50 UTC, Walter Bright wrote:

 Starting a new thread with this, as requested. There are things 
 here for all skill levels and time commitments.

 3. use -cov to improve code coverage of Phobos unittests

 4. make sure every function in Phobos has an example

 5. make sure every function in Phobos has Params: and Returns: 
 sections
 http://www.digitalmars.com/d/archives/digitalmars/D/Phobos_Documentation_-_call_to_action_258777.html


Maybe someone would like to go a step meta and look into 
automatically tracking and checking things first? Here is what 
autotester (or some other tool) could do:

* Check for a lower bound of coverage, so we never regress.

* Check documentation for Example, Params:, Returns:

* Check for broken links

By tracking I think: Count the stuff above and plot over time. 
Somebody did this for the number of pull requests, for example. 
This in itself does not fix anything, but it motivates. "Measure 
it and it will improve" –Jack Welch
Jun 20 2016
prev sibling parent Kagamin <spam here.lot> writes:
https://lexi-lambda.github.io/blog/2016/06/12/four-months-with-haskell/#documentation
is-nearly-worthless - found this on reddit, criticizes haskell docs and praises
racket docs:

 they are so effective because of how the prose explains what 
 each function does, when to use it, why you’d use it, and why 
 you wouldn’t use it.
Jun 22 2016