www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - State of the Compiler

reply Walter Bright <newshound2 digitalmars.com> writes:
I'm pretty unhappy with the state of dmd. I think the best that can be said for 
it at the moment is that it works.

1. every time we fix something, there are unexpected regressions.
2. the code is a hopeless tangle.
3. too much global state (like global.errors, which is itself a source of 
repeated regressions)
4. it's slowed down substantially.
5. near complete lack of encapsulation
6. my ideas on how software should be written have evolved a lot since it was 
laid out

This cannot be fixed overnight, but I've been working on it off and on in 
between doing other more urgent things, and submitting PRs for them. The goals 
of all this are (and this includes the back end):

1. elimination of all conditional compilation, and macros. All that code should 
be encapsulated and banished to Port or Target.

2. encapsulation of all memory management. For example, StringExp's instance of 
string data is allocated and free'd all over the place. I've made some progress 
in encapsulating it. Getting it properly encapsulated means we can do Small 
String Optimization in it, which should get us a nice boost. SSO for all these 
sorts of structures needs to be done.

3. retrofit in const as much as possible, and later try to use pure.

4. get rid of all global state variables, except for configuration stuff set by 
command line switches

5. convert back end to D.

6. use nested functions to improve encapsulation.

7. remove C++ artifacts, like the fake delegates, and replace them with real 
delegates

8. replace symbol tables and custom hash code with builtin hashes, and fix the 
builtin hashes if they aren't as fast

9. change to "full lazy" semantic analysis of imports, i.e. only parse them far 
enough to build the symbol table. Do semantic analysis of the symbols only on 
demand. I did this for enums a while back, and the results have been very
pleasing.

10. use rc memory for CTFE, or even stack memory. this is already done to a 
small extent, but should go much further

11. encapsulate, encapsulate, encapsulate

12. start retrofitting with phobos algorithms

13. use ranges


One benefit of all this is we could start using multicores to compile!

This will all take years, but we've already made good progress such as the 
conversion of the source code to D.
Feb 28 2016
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 29/02/16 1:10 PM, Walter Bright wrote:

 I'm pretty unhappy with the state of dmd. I think the best that can be
 said for it at the moment is that it works.

 1. every time we fix something, there are unexpected regressions.
 2. the code is a hopeless tangle.
 3. too much global state (like global.errors, which is itself a source
 of repeated regressions)
 4. it's slowed down substantially.
 5. near complete lack of encapsulation
 6. my ideas on how software should be written have evolved a lot since
 it was laid out

 This cannot be fixed overnight, but I've been working on it off and on
 in between doing other more urgent things, and submitting PRs for them.
 The goals of all this are (and this includes the back end):

 1. elimination of all conditional compilation, and macros. All that code
 should be encapsulated and banished to Port or Target.

 2. encapsulation of all memory management. For example, StringExp's
 instance of string data is allocated and free'd all over the place. I've
 made some progress in encapsulating it. Getting it properly encapsulated
 means we can do Small String Optimization in it, which should get us a
 nice boost. SSO for all these sorts of structures needs to be done.

 3. retrofit in const as much as possible, and later try to use pure.

 4. get rid of all global state variables, except for configuration stuff
 set by command line switches

 5. convert back end to D.

 6. use nested functions to improve encapsulation.

 7. remove C++ artifacts, like the fake delegates, and replace them with
 real delegates

 8. replace symbol tables and custom hash code with builtin hashes, and
 fix the builtin hashes if they aren't as fast

 9. change to "full lazy" semantic analysis of imports, i.e. only parse
 them far enough to build the symbol table. Do semantic analysis of the
 symbols only on demand. I did this for enums a while back, and the
 results have been very pleasing.

 10. use rc memory for CTFE, or even stack memory. this is already done
 to a small extent, but should go much further

 11. encapsulate, encapsulate, encapsulate

 12. start retrofitting with phobos algorithms

 13. use ranges


 One benefit of all this is we could start using multicores to compile!

 This will all take years, but we've already made good progress such as
 the conversion of the source code to D.



I have to suggest this, but how much work do you believe it would take 
to switch over to sdc's frontend instead? (Keeping in mind it isn't 
compiling much right now).
Feb 28 2016
parent Chris Wright <dhasenan gmail.com> writes:
On Mon, 29 Feb 2016 15:13:28 +1300, Rikki Cattermole wrote:


 I have to suggest this, but how much work do you believe it would take
 to switch over to sdc's frontend instead? (Keeping in mind it isn't
 compiling much right now).


Not impossible, just difficult.

The first step is to catch SDC up to the latest DMD release. That's for 
what it compiles, what it errors about, what it warns about, and ddoc, at 
a minimum. That's for FreeBSD, Linux, OSX, and Windows.

After that, we keep SDC up to date with DMD for some amount of time. We 
announce that we're cutting over and people should migrate their PRs to 
SDC.

In the meantime, the SDC team is just porting changes from DMD to SDC.

Refactoring in place means your PR might need changes to apply correctly, 
but it won't need to be rewritten from scratch. It's far less of an 
interruption.
Feb 28 2016
prev sibling next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright wrote:

 12. start retrofitting with phobos algorithms


Other DMD devs have taken a hard line stance that Phobos code 
cannot and should not be used in ddmd. What's your opinion on 
that?
Feb 28 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/28/2016 8:53 PM, Jack Stouffer wrote:

 On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright wrote:

 12. start retrofitting with phobos algorithms


 Other DMD devs have taken a hard line stance that Phobos code cannot and should
 not be used in ddmd. What's your opinion on that?


There are good reasons for that. But there's no reason why std.algorithm.sort, 
for example, cannot still be used (cut & paste).
Feb 28 2016
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-02-29 05:57, Walter Bright wrote:


 There are good reasons for that. But there's no reason why
 std.algorithm.sort, for example, cannot still be used (cut & paste).


Unfortunately Phobos has quite a complex dependency graph. Pulling in 
std.algorithm.sort will most likely pull in half of Phobos.

-- 
/Jacob Carlborg
Feb 28 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 02/28/2016 11:57 PM, Walter Bright wrote:

 There are good reasons for that. But there's no reason why
 std.algorithm.sort, for example, cannot still be used (cut & paste).


How would cutting and pasting from phobos be better than using it? -- Andrei
Mar 01 2016
next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Tuesday, 1 March 2016 at 13:48:18 UTC, Andrei Alexandrescu 
wrote:

 How would cutting and pasting from phobos be better than using 
 it? -- Andrei


There wouldn't be downward pressure on Phobos devs from DMD devs 
to not change anything.

One of the things that contributed to my decision to switch to D 
from Python is that D is:

1. Willing to admit that it made a mistake
2. Willing to fix it through deprecation -> un-documentation -> 
removal

Python is no longer willing to make backwards incompatible 
changes with with it's bad experience with Python 3. Therefore 
one of two things will happen. Either the language will stagnate, 
or the bad ideas will remain when new ideas are added, bloating 
the language.

If you have more pressure not to not change anything, I think 
this will have negative consequences. Bad ideas must be removed, 
and removed as soon as possible, if the language is to thrive.
Mar 01 2016
next sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Tue, 01 Mar 2016 16:23:37 +0000, Jack Stouffer wrote:


 On Tuesday, 1 March 2016 at 13:48:18 UTC, Andrei Alexandrescu wrote:

 How would cutting and pasting from phobos be better than using it? --
 Andrei


 
 There wouldn't be downward pressure on Phobos devs from DMD devs to not
 change anything.


Specifically not to add anything, since we want to ensure that DMD 2.73.0 
compiles with DMD 2.70.0 through DMD 2.72.x.
Mar 01 2016
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 1 March 2016 at 16:57:41 UTC, Chris Wright wrote:

 Specifically not to add anything, since we want to ensure that 
 DMD 2.73.0 compiles with DMD 2.70.0 through DMD 2.72.x.


Why is this important? Cant you lock DMD2.x to DMD2.70.x 
(back-porting bug fixes)?

Then you push for changes with future upgrades in mind, so when 
you have a significant improvement overall you switch to the next 
major stable release?
Mar 01 2016
next sibling parent Chris Wright <dhasenan gmail.com> writes:
On Tue, 01 Mar 2016 17:06:21 +0000, Ola Fosheim Grøstad wrote:


 Cant you lock DMD2.x to DMD2.70.x (back-porting bug fixes)?


Yes. If the build system uses dmd from $PATH, that would be pretty 
annoying. However, if the build system downloads and unpacks the correct 
dmd inside the build directory and uses that instead, that would work 
pretty well.

This would be a reasonable thing to add even if we're not going to allow 
dmd to depend on phobos. It lets people know what version of dmd should 
be able to compile the current dmd, defending against "it compiles on my 
machine" issues. It also prevents you from using newer language features 
in dmd before it's been officially determined that that's acceptable.

Backporting all bugfixes is a nontrivial cost. Backporting only as 
necessary isn't, and it's just as straightforward with this strategy as 
with copying phobos modules into dmd's source tree.
Mar 01 2016
prev sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Tuesday, 1 March 2016 at 17:06:21 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 1 March 2016 at 16:57:41 UTC, Chris Wright wrote:

 Specifically not to add anything, since we want to ensure that 
 DMD 2.73.0 compiles with DMD 2.70.0 through DMD 2.72.x.


 Why is this important? Cant you lock DMD2.x to DMD2.70.x 
 (back-porting bug fixes)?


Back-porting bug fixes requires resources/manpower the compiler 
team doesn't have. They don't have enough manpower to keep with 
with the current bugs, let alone adding extra work to each fix by 
targeting two code bases.
Mar 01 2016
parent tsbockman <thomas.bockman gmail.com> writes:
On Tuesday, 1 March 2016 at 20:57:39 UTC, Jack Stouffer wrote:

 They don't have enough manpower to keep with with the current 
 bugs, let alone adding extra work to each fix by targeting two 
 code bases.


Speaking of which... could I persuade anyone to take a crack at 
this DMD codegen bug I found back in January?

https://issues.dlang.org/show_bug.cgi?id=15573

It breaks the test suite for checkedint on DMD.
Mar 01 2016
prev sibling next sibling parent Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Tuesday, 1 March 2016 at 16:23:37 UTC, Jack Stouffer wrote:

 On Tuesday, 1 March 2016 at 13:48:18 UTC, Andrei Alexandrescu 
 wrote:

 How would cutting and pasting from phobos be better than using 
 it? -- Andrei


 There wouldn't be downward pressure on Phobos devs from DMD 
 devs to not change anything.

 One of the things that contributed to my decision to switch to 
 D from Python is that D is:

 1. Willing to admit that it made a mistake
 2. Willing to fix it through deprecation -> un-documentation -> 
 removal

 Python is no longer willing to make backwards incompatible 
 changes with with it's bad experience with Python 3. Therefore 
 one of two things will happen. Either the language will 
 stagnate, or the bad ideas will remain when new ideas are 
 added, bloating the language.

 If you have more pressure not to not change anything, I think 
 this will have negative consequences. Bad ideas must be 
 removed, and removed as soon as possible, if the language is to 
 thrive.


I agree - that's exactly right (I don't mean about python, as I'm 
not involved there).  Balancing between change for the sake of it 
and stasis requires good judgment, and not everybody who voices 
loudly an opinion on the topic excels at this.

As Walter says, you should listen to the people who already like 
you and use your product, and spend less time worrying about 
those who don't.  Because it tends to be the case that should you 
change what they say, they'll find something else to complain 
about.

Of the 'serious developers' who have the most skin in the game 
commercially, it doesn't seem like they are upset with the 
present way things are done.  If anything, Sociomantic - to take 
the most prominent example - have asked for a bit more breakage 
to do things right.

Also, life involves Type 1 and Type 2 errors.  If over time you 
don't have some people unhappy that there is too much breakage 
and some unhappy that you seem to be stuck with bad legacy 
choices then you probably aren't striking the right balance, and 
it's worth bearing that in mind, I think.
Mar 02 2016
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 1 March 2016 at 16:23:37 UTC, Jack Stouffer wrote:

 If you have more pressure not to not change anything, I think 
 this will have negative consequences. Bad ideas must be 
 removed, and removed as soon as possible, if the language is to 
 thrive.


You cannot delete things from a standard library, that makes it 
non-standard. You need to add stuff, then have a looooong 
deprecation period before removing the old stuff.

It is actually easier to change the language than to change the 
semantics of a standard library.
Mar 04 2016
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, 1 March 2016 at 13:48:18 UTC, Andrei Alexandrescu 
wrote:

 On 02/28/2016 11:57 PM, Walter Bright wrote:

 There are good reasons for that. But there's no reason why
 std.algorithm.sort, for example, cannot still be used (cut & 
 paste).


 How would cutting and pasting from phobos be better than using 
 it? -- Andrei


You avoid a circular dependency. If it's just stuff like the 
functions in std.algorithm, it's probably not too bad, but if 
anything pulls in std.conv, std.format, std.stdio, etc., you end 
up with a bunch of extra stuff, whereas what the compiler really 
needs is probably only a small piece of that. And if there's a 
bug in any of the Phobos code that the compiler uses, you end up 
with a bug in the compiler, which in turn causes bugs when 
compiling Phobos...

Copying specific pieces of Phobos and pasting them into dmd does 
allow for stuff that's more specific to what dmd is doing and 
avoids dependency issues - but it does mean copying, which means 
code duplication and can cause it's own set of problems.

I don't know if it's ultimately better to have dmd use Phobos or 
not, but it does avoid some potentially weird dependency issues 
with bugs if we avoid it. Things already get weird enough when 
Phobos depends on itself. Locking the version down like Ola 
Fosheim Grøstad is talking about definitely would mitigate the 
problem, but it would just be simpler to avoid it. Regardless, 
I'll leave it up to the compiler devs to come to a decision on 
that one.

- Jonathan M Davis
Mar 01 2016
prev sibling parent reply Suliman <evermind live.ru> writes:
On Monday, 29 February 2016 at 04:53:16 UTC, Jack Stouffer wrote:

 On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright 
 wrote:

 12. start retrofitting with phobos algorithms


 Other DMD devs have taken a hard line stance that Phobos code 
 cannot and should not be used in ddmd. What's your opinion on 
 that?


Please, explain the situation...
Feb 28 2016
parent reply Chris Wright <dhasenan gmail.com> writes:
On Mon, 29 Feb 2016 07:33:51 +0000, Suliman wrote:


 On Monday, 29 February 2016 at 04:53:16 UTC, Jack Stouffer wrote:

 On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright wrote:

 12. start retrofitting with phobos algorithms


 Other DMD devs have taken a hard line stance that Phobos code cannot
 and should not be used in ddmd. What's your opinion on that?


 
 Please, explain the situation...


DMD doesn't depend on Phobos. It will not depend on Phobos. There are a 
number of reasons for this.

One of the important ones is: let's say DMD depended on Phobos. I'm 
developing Phobos and DMD side by side. I need a new function for DMD, 
and it would fit nicely in std.algorithm. So I add it to Phobos in a PR, 
add my code to DMD the next day -- and suddenly compiling DMD is a mess.

And that situation is worse. If DMD added a new feature earlier that 
release cycle and Phobos started depending on it, now you can't compile 
Phobos with the prior version of DMD, and you can't compile DMD with the 
prior version of Phobos, so you need to check out potentially several 
different revisions of DMD and Phobos iteratively to build their next 
versions.

Also, by not including all of Phobos, there's less code to reason about 
if you're trying to verify DMD's correctness. That's a bit marginal, but 
it's there.
Feb 29 2016
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 29 February 2016 at 16:22:41 UTC, Chris Wright wrote:

 One of the important ones is: let's say DMD depended on Phobos. 
 I'm developing Phobos and DMD side by side. I need a new 
 function for DMD, and it would fit nicely in std.algorithm. So 
 I add it to Phobos in a PR, add my code to DMD the next day -- 
 and suddenly compiling DMD is a mess.


That would be the sign of a _completely_ messed up development 
process which would suggest problems way beyond using a standard 
library for writing a self-hosting compiler...


 And that situation is worse. If DMD added a new feature earlier 
 that release cycle and Phobos started depending on it, now you 
 can't compile Phobos with the prior version of DMD, and you 
 can't compile DMD with the prior version of Phobos, so you need 
 to check out potentially several different revisions of DMD and 
 Phobos iteratively to build their next versions.


Err... no, you use a preselected _stable_ version for the current 
non-stable build and archive stable builds you depend on. Of 
course, to get there you need a stable branch in the first place.

Basically all the rational arguments for not using phobos in the 
front end can be turned into arguments for not relying on phobos 
in any major project.
Feb 29 2016
next sibling parent reply asdf <a b.c> writes:
On Monday, 29 February 2016 at 16:57:57 UTC, Ola Fosheim Grøstad 
wrote:

 Basically all the rational arguments for not using phobos in 
 the front end can be turned into arguments for not relying on 
 phobos in any major project.


Nobody *ever* bootstraps a compiler using all it's features.
http://www.standardpascal.com/CDC6000pascal.html
http://www.lysator.liu.se/c/dmr-on-histories.html
http://www.forth.com/resources/evolution/evolve_6.html
http://www.doc.gold.ac.uk/~mas01cr/papers/s32008/sbcl.pdf <-- 
actually hard to bootstrap, makes gentle fun of lisp500.c
Feb 29 2016
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 29 February 2016 at 21:09:22 UTC, asdf wrote:

 On Monday, 29 February 2016 at 16:57:57 UTC, Ola Fosheim 
 Grøstad wrote:

 Basically all the rational arguments for not using phobos in 
 the front end can be turned into arguments for not relying on 
 phobos in any major project.


 Nobody *ever* bootstraps a compiler using all it's features.


Completely bogus argument. What was common was to implement a 
small subset in another language and use that subset for the main 
compiler, to simplify porting to new architectures.

This argument has no useful value since D that would make D stuck 
on the C++ version of DMD. And it is pointless when you have 
cross compilers and VMs.
Feb 29 2016
prev sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Mon, 29 Feb 2016 16:57:57 +0000, Ola Fosheim Grøstad wrote:


 On Monday, 29 February 2016 at 16:22:41 UTC, Chris Wright wrote:

 One of the important ones is: let's say DMD depended on Phobos.
 I'm developing Phobos and DMD side by side. I need a new function for
 DMD, and it would fit nicely in std.algorithm. So I add it to Phobos in
 a PR, add my code to DMD the next day -- and suddenly compiling DMD is
 a mess.


 
 That would be the sign of a _completely_ messed up development process
 which would suggest problems way beyond using a standard library for
 writing a self-hosting compiler...


You might be complaining about a single person doing this, but it could 
be several different people instead of just one.

Continuous integration would catch this type of problem, and I think that 
would be good enough. Forbidding any dependency is the nuclear option.


 Basically all the rational arguments for not using phobos in the front
 end can be turned into arguments for not relying on phobos in any major
 project.


The first part is an argument against developing against a development 
version of phobos when people need to be able to compile with the latest 
release. I'd strongly recommend that everyone develop against released 
versions of phobos.

The second part is an argument against circular dependencies between 
projects. Phobos depends only on system libraries, libcurl, druntime, and 
dmd, and it's unlikely to add your project written in D as a new 
dependency.

I'm not sure how that translates to "never use phobos".
Feb 29 2016
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 29 February 2016 at 23:25:45 UTC, Chris Wright wrote:

 The second part is an argument against circular dependencies 
 between projects. Phobos depends only on system libraries, 
 libcurl, druntime, and dmd, and it's unlikely to add your 
 project written in D as a new dependency.

 I'm not sure how that translates to "never use phobos".


It translates to the same thing for any major project as it does 
for the compiler. If phobos isn't stable enough for the compiler, 
then it isn't stable enough for any major project.

All we need is a stable release, lock the compiler development to 
that release (add a conditional check for it). For archival 
purposes all that is needed is to ensure that the executable 
image of this stable release can be run in an x86 emulator.
Feb 29 2016
next sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 1 March 2016 at 07:00:43 UTC, Ola Fosheim Grøstad 
wrote:

 If phobos isn't stable enough for the compiler, then it isn't 
 stable enough for any major project.


Not supported by evidence. Unstable ecosystems are successfully 
used in major projects as long as they don't have such circular 
dependencies.
Mar 01 2016
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 1 March 2016 at 10:11:03 UTC, Kagamin wrote:

 On Tuesday, 1 March 2016 at 07:00:43 UTC, Ola Fosheim Grøstad 
 wrote:

 If phobos isn't stable enough for the compiler, then it isn't 
 stable enough for any major project.


 Not supported by evidence. Unstable ecosystems are successfully 
 used in major projects as long as they don't have such circular 
 dependencies.


No circular dependency.

Most _serious_ developers go for mature frameworks and libraries 
for major projects.
Mar 01 2016
next sibling parent reply Luis <luis.panadero gmail.com> writes:
On Tuesday, 1 March 2016 at 11:22:50 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 1 March 2016 at 10:11:03 UTC, Kagamin wrote:

 On Tuesday, 1 March 2016 at 07:00:43 UTC, Ola Fosheim Grøstad 
 wrote:

 If phobos isn't stable enough for the compiler, then it isn't 
 stable enough for any major project.


 Not supported by evidence. Unstable ecosystems are 
 successfully used in major projects as long as they don't have 
 such circular dependencies.


 No circular dependency.

 Most _serious_ developers go for mature frameworks and 
 libraries for major projects.


So, for example,  a C++ compiler can't use the standard C++ lib ?
Mar 01 2016
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 1 March 2016 at 12:57:43 UTC, Luis wrote:

 So, for example,  a C++ compiler can't use the standard C++ lib 
 ?


I don't follow your logic here. Clang uses std.
Mar 01 2016
parent reply Luis <luis.panadero gmail.com> writes:
On Tuesday, 1 March 2016 at 13:14:20 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 1 March 2016 at 12:57:43 UTC, Luis wrote:

 So, for example,  a C++ compiler can't use the standard C++ 
 lib ?


 I don't follow your logic here. Clang uses std.


There is some discussion if dmd should not use phobos because 
there would be a circular relation. Well, as you pointed CLang 
uses std, so where is the problem ? I think that not should be 
any problem if dmd always uses the previous version of phobos to 
bootstrap itself.
Mar 01 2016
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 1 March 2016 at 13:19:29 UTC, Luis wrote:

 There is some discussion if dmd should not use phobos because 
 there would be a circular relation. Well, as you pointed CLang 
 uses std, so where is the problem ?


I don't think there is a problem, but if there is a problem then 
that problem would transfer to all other major projects using 
phobos.

Either phobos is considered alpha (unstable api) or not.  If it 
is ok to use it for a database engine, then it is ok to use it 
for the compiler as well. If not then people should be given fair 
warning of what is and isn't considered alpha.


 I think that not should be any problem if dmd always uses the 
 previous version of phobos to bootstrap itself.


Yes, perhaps not the previous version, but a specified stable 
version (same version for compiler, runtime and library).

There is no "circular relation" in this as long as you use a 
compiler that is pretty solid.
Mar 01 2016
prev sibling parent reply Mathias Lang via Digitalmars-d <digitalmars-d puremagic.com> writes:
2016-03-01 12:22 GMT+01:00 Ola Fosheim Gr=C3=B8stad via Digitalmars-d <
digitalmars-d puremagic.com>:


 On Tuesday, 1 March 2016 at 10:11:03 UTC, Kagamin wrote:


 On Tuesday, 1 March 2016 at 07:00:43 UTC, Ola Fosheim Gr=C3=B8stad wrote=




:

 If phobos isn't stable enough for the compiler, then it isn't stable
 enough for any major project.


 Not supported by evidence. Unstable ecosystems are successfully used in
 major projects as long as they don't have such circular dependencies.


 No circular dependency.

 Most _serious_ developers go for mature frameworks and libraries for majo=


r

 projects.


https://yourlogicalfallacyis.com/no-true-scotsman

IMO this discussion belongs to dmd-internal. And there are a lot of low
hanging fruits to harvest before Phobos, some of them don't even require
compiler knowledge.
Mar 01 2016
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 1 March 2016 at 21:01:13 UTC, Mathias Lang wrote:

Most _serious_ developers go for mature frameworks and
 libraries for major projects.


 https://yourlogicalfallacyis.com/no-true-scotsman


Here I thought I stated one of the most obvious empirical fact, 
but in the D community there is no shortage of butt-hurt 
defensiveness which means that the even the most obvious facts 
have to be contradicted. That's the kind of attitude which will 
make professional look elsewhere.


 IMO this discussion belongs to dmd-internal.


This is a general end-user forum, are you saying that people 
should not be allowed to discuss things openly and freely when 
the lead developer brings the topic to the table?

Nobody has said that this discussions cannot be discussed 
elsewhere. I would expect that there has been internal 
discussions on this topic before it was published here in the 
_General_ forum.


 And there are a lot of low hanging fruits to harvest before 
 Phobos, some of them don't even require compiler knowledge.


Focusing on "Low hanging fruits" has been costly for this project.

Walter is getting it exactly right on the 
encapsulation/refactoring issue. This is currently discouraging 
people from digging into the compiler internals. Combine 
refactoring with documentation and you get more people working on 
the compiler.

Making it more _fun_ and easier for new people to jump into the 
code base would b a very important strategy. Making the codebase 
more idiomatic would be an important step.
Mar 02 2016
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Tuesday, 1 March 2016 at 21:01:13 UTC, Mathias Lang wrote:

 2016-03-01 12:22 GMT+01:00 Ola Fosheim Grøstad via 
 Digitalmars-d < digitalmars-d puremagic.com>:


 On Tuesday, 1 March 2016 at 10:11:03 UTC, Kagamin wrote:


 On Tuesday, 1 March 2016 at 07:00:43 UTC, Ola Fosheim Grøstad 
 wrote:


 If phobos isn't stable enough for the compiler, then it 
 isn't stable enough for any major project.


 Not supported by evidence. Unstable ecosystems are 
 successfully used in major projects as long as they don't 
 have such circular dependencies.


 No circular dependency.

 Most _serious_ developers go for mature frameworks and 
 libraries for major projects.


 https://yourlogicalfallacyis.com/no-true-scotsman

 IMO this discussion belongs to dmd-internal. And there are a 
 lot of low hanging fruits to harvest before Phobos, some of 
 them don't even require compiler knowledge.


https://www.youtube.com/watch?v=jmRNmGhdYnU
Mar 02 2016
prev sibling next sibling parent reply Temtaime <temtaime gmail.com> writes:
IMHO Phobos must be used in DMD. It can reduce dmd's code so 
it'll be easier to maintain it. Also phobos is time-proven, so 
less bugs will be in the compiler.
And forcément using phobos can test both phobos and dmd to find 
the regressions in the development stage.
Mar 01 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 03/01/2016 05:19 AM, Temtaime wrote:

 IMHO Phobos must be used in DMD.


Agreed. It will happen in time, -- Andrei
Mar 01 2016
prev sibling parent Chris Wright <dhasenan gmail.com> writes:
On Tue, 01 Mar 2016 07:00:43 +0000, Ola Fosheim Grøstad wrote:


 On Monday, 29 February 2016 at 23:25:45 UTC, Chris Wright wrote:

 The second part is an argument against circular dependencies between
 projects. Phobos depends only on system libraries, libcurl, druntime,
 and dmd, and it's unlikely to add your project written in D as a new
 dependency.

 I'm not sure how that translates to "never use phobos".


 
 It translates to the same thing for any major project as it does for the
 compiler. If phobos isn't stable enough for the compiler,
 then it isn't stable enough for any major project.
 
 All we need is a stable release, lock the compiler development to that
 release (add a conditional check for it). For archival purposes all that
 is needed is to ensure that the executable image of this stable release
 can be run in an x86 emulator.


In other words, in mars.d we add:
  static assert(__VERSION__ == 2068);
And occasionally we update that to something not bleeding edge. That's 
pretty extreme. Kinda hostile toward packagers and developers.

I'd set up continuous integration for each dmd release I want dmd to 
compile with, but that's just me.

Anyway, since dmd and phobos share a lot of top contributors and phobos 
tends to be an early adopter for new language features, they're often 
developed together. Plus they both have a lot more contributors than 
anything I've worked on outside a professional capacity. So we want 
either automated checking or a very simple rule. Walter went with a very 
simple rule. That doesn't protect from using newer language features, 
though, so you still need continuous integration. Less surface area for 
problems to crop up, though.
Mar 01 2016
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright wrote:

 11. encapsulate, encapsulate, encapsulate


Good point, this will make it easier for new people to contribute!


 One benefit of all this is we could start using multicores to 
 compile!


What is the payoff for having multithreading in the front end? In 
order to benefit much from it you either need cache-locality, a 
small working set, waiting for I/O and the frontend being a 
bigger bottle neck than the backend optimizer.

Seems to me you can get more out of it by focusing on:

1. moving as much as possible to the back end and making the 
front end simpler

2. focus on caching-techniques between compiles

3. add full separation between frontend and backend and keep the 
backend in a separate process using shared memory for 
communicating the IR. That way you can run the backend on 
multiple machines, CPUs or cores + reuse backends written in any 
language.
Feb 29 2016
prev sibling next sibling parent reply Mathias Lang via Digitalmars-d <digitalmars-d puremagic.com> writes:
2016-02-29 1:10 GMT+01:00 Walter Bright via Digitalmars-d <
digitalmars-d puremagic.com>:


 I'm pretty unhappy with the state of dmd. I think the best that can be
 said for it at the moment is that it works.
 [...]



Any plan for modularization ? Currently we have massive modules living in
src, and it's pretty hard to get the architecture when you want to start
contributing.
Feb 29 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/29/2016 1:47 AM, Mathias Lang via Digitalmars-d wrote:

 Any plan for modularization ? Currently we have massive modules living in src,
 and it's pretty hard to get the architecture when you want to start
contributing.


I would call that "encapsulation".
Feb 29 2016
prev sibling next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright wrote:

 5. convert back end to D.


What's the status of this? When the ddmd switch happened, Daniel 
Murphy was saying that a similar transition in the back end would 
take about six to eight months.
Feb 29 2016
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/29/2016 12:18 PM, Jack Stouffer wrote:

 On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright wrote:

 5. convert back end to D.


 What's the status of this? When the ddmd switch happened, Daniel Murphy was
 saying that a similar transition in the back end would take about six to eight
 months.


There's progress, but much more needs to be done. Still too much preprocessor
use.
Feb 29 2016
prev sibling parent Daniel Murphy <yebbliesnospam gmail.com> writes:
On 1/03/2016 7:18 AM, Jack Stouffer wrote:

 On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright wrote:

 5. convert back end to D.


 What's the status of this? When the ddmd switch happened, Daniel Murphy
 was saying that a similar transition in the back end would take about
 six to eight months.


The actual conversion process is pretty straightforward.  The big issue 
is that converting would mean losing the dmc test suite which is our 
best way to test the backend.

The plan is to dump the test suite to IR, then set up a way to run the 
IR through the backend and check the output hasn't changed.  Then we can 
start actually converting to D.  I have a PR for some of this open.
Feb 29 2016
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, 29 February 2016 at 00:10:33 UTC, Walter Bright wrote:

 This will all take years, but we've already made good progress 
 such as the conversion of the source code to D.


You should probably turn at least some of this into a checklist 
somewhere so that it doesn't get lost in the shuffle. Also, it 
could be used as a source of suggestions for stuff for folks to 
work on in the compiler if they want suggestions or want guidance 
on which sorts of things that they're thinking about doing should 
be done first.

Maybe the wiki should have pages for dmd and Phobos (and possible 
druntime) with stuff like this which indicates what you'd prefer 
that folks do. They're still free to do whatever they want and 
try and get it merged, and most of the core developers already 
have a fair idea of what they want to work on (frequently more 
than they can get to in a sane time frame), but it might better 
inform which things get done first as well as server as 
suggestions for new folks who don't know what they can or should 
do to help.

- Jonathan M Davis
Mar 02 2016
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Wednesday, 2 March 2016 at 22:12:11 UTC, Jonathan M Davis 
wrote:

 Maybe the wiki should have pages for dmd and Phobos (and 
 possible druntime) with stuff like this which indicates what 
 you'd prefer that folks do.


I'd think that bugzilla is a much better mechanism for this, as 
you can mark internal dmd enhancement requests as such and people 
already browse bugzilla for things to work on, where as the wiki 
tends to not get maintained.
Mar 02 2016
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 3 March 2016 at 00:21:48 UTC, Jack Stouffer wrote:

 On Wednesday, 2 March 2016 at 22:12:11 UTC, Jonathan M Davis 
 wrote:

 Maybe the wiki should have pages for dmd and Phobos (and 
 possible druntime) with stuff like this which indicates what 
 you'd prefer that folks do.


 I'd think that bugzilla is a much better mechanism for this, as 
 you can mark internal dmd enhancement requests as such and 
 people already browse bugzilla for things to work on, where as 
 the wiki tends to not get maintained.


*warning*: off-topic!

Tagging them with an according priority or maybe "help wanted" on 
their issue tracker is what most project that I know do.

As a newcomer to DLang I would like to point out that such a 
roadmap about the project's future is nice to have. I know you 
have those nice visions [1, 2], but there are quite hard to find 
and aimed at newcomers.
(update: I edited the main wiki page to link to [2] instead of 
[1]).

I __really like___ Mozilla's Contributing guide, especially 
because they have three categories (Good first bugs [7], mentored 
bugs [8], student projects [9]).

Other ideas
------------

* dedicated section at [4] (e.g. Good areas to start with or Help 
needed)
* the idea of bountysource is pretty awesome [4] - I can see that 
it hasn't been that active over time. What is your experience 
with that? Maybe tagging such issues with 10-20$ could help to 
attract people if announced properly.

Apart from that I think that your DIP [5] list is pretty sweat, 
so putting them in the wiki could work too ;-)

[1] https://wiki.dlang.org/Vision/2015H2
[2] https://wiki.dlang.org/Vision/2016H1
[3] https://wiki.dlang.org/Get_involved
[4] https://www.bountysource.com/teams/d/issues
[5] http://wiki.dlang.org/List_of_DIP
[6] 
https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Introduction
[7] https://bugzil.la/sw:%22[good%20first%20bug]%22&limit=0
[8] http://www.joshmatthews.net/bugsahoy/
[9] https://bugzil.la/kw:student-project
Mar 04 2016
parent Kagamin <spam here.lot> writes:
On Friday, 4 March 2016 at 10:56:47 UTC, Seb wrote:

 I __really like___ Mozilla's Contributing guide, especially 
 because they have three categories (Good first bugs [7], 
 mentored bugs [8], student projects [9]).


SDC and LDC do something like that: 
https://github.com/ldc-developers/ldc/issues?utf8=%E2%9C%93&q=label%3AC-junior-job
Mar 04 2016