www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A betterC modular standard library?

reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
Hi,

Who is interested in betterC _modular_* standard library?
I am planing to make libmir org a community for it.
Thought and concerns?

We already have better `cpuid` and better `random` packages. The 
betterC std.range and std.algorithm analogs would be released 
with new ndslice implementation. Mir's algorithm would be faster 
then Phobos and will generate less template bloat. Then 
lightweight threads, multithread GLAS, matrix inversion. Fastest 
I/O and http2 ...

(they all are betterC modular libraries)

Andrei and probably 90% of existing D users don't want Phobos to 
move this direction.

In other hand I need a commercial attractive D infrastructure for 
large and heavy system projects. There is no commercial 
perspective for me to contribute to Phobos because:

1. Phobos version depends on compiler version. Delay with LDC 
release is too large. It should / can be one day.

2. We can not do hot fixes in Phobos without compiling and 
deploying patched libphobos. Mir's DUB package are much more 
flexible.

3. Dependencies should be clear. Modularity is a proper way for 
large std library. In phobos everything integrated with 
everything. DRuntime -> Phobos abstraction is weird for betterC 
because system modules can depends universal algorithms, but 
universal algorithm are more portable if they have not system 
dependencies.

4. Phobos does not provide and will not provide betterC 
guaranties. If something works for betterC mode now it may not 
work in the future release.

[1] https://github.com/libmir/
* - separate github/DUB libraries with clear dependencies.

Best regards,
Ilya
Dec 18 2016
next sibling parent reply Temtaime <temtaime gmail.com> writes:
On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko 
wrote:

 Hi,

 Who is interested in betterC _modular_* standard library?
 I am planing to make libmir org a community for it.
 Thought and concerns?

 We already have better `cpuid` and better `random` packages. 
 The betterC std.range and std.algorithm analogs would be 
 released with new ndslice implementation. Mir's algorithm would 
 be faster then Phobos and will generate less template bloat. 
 Then lightweight threads, multithread GLAS, matrix inversion. 
 Fastest I/O and http2 ...

 (they all are betterC modular libraries)

 Andrei and probably 90% of existing D users don't want Phobos 
 to move this direction.

 In other hand I need a commercial attractive D infrastructure 
 for large and heavy system projects. There is no commercial 
 perspective for me to contribute to Phobos because:

 1. Phobos version depends on compiler version. Delay with LDC 
 release is too large. It should / can be one day.

 2. We can not do hot fixes in Phobos without compiling and 
 deploying patched libphobos. Mir's DUB package are much more 
 flexible.

 3. Dependencies should be clear. Modularity is a proper way for 
 large std library. In phobos everything integrated with 
 everything. DRuntime -> Phobos abstraction is weird for betterC 
 because system modules can depends universal algorithms, but 
 universal algorithm are more portable if they have not system 
 dependencies.

 4. Phobos does not provide and will not provide betterC 
 guaranties. If something works for betterC mode now it may not 
 work in the future release.

 [1] https://github.com/libmir/
 * - separate github/DUB libraries with clear dependencies.

 Best regards,
 Ilya


Let's write a D, phobos.
Then we drop all the features and use a BetterC mode.
Then we write all modules against betterC mode.
Dec 18 2016
parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 18 December 2016 at 11:06:58 UTC, Temtaime wrote:

 Let's write a D, phobos.
 Then we drop all the features and use a BetterC mode.
 Then we write all modules against betterC mode.


Sounds like a plan! --Ilya
Dec 18 2016
prev sibling next sibling parent reply Radu <void null.pt> writes:
On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko 
wrote:

 Hi,

 Who is interested in betterC _modular_* standard library?
 I am planing to make libmir org a community for it.
 Thought and concerns?

 We already have better `cpuid` and better `random` packages. 
 The betterC std.range and std.algorithm analogs would be 
 released with new ndslice implementation. Mir's algorithm would 
 be faster then Phobos and will generate less template bloat. 
 Then lightweight threads, multithread GLAS, matrix inversion. 
 Fastest I/O and http2 ...

 (they all are betterC modular libraries)

 Andrei and probably 90% of existing D users don't want Phobos 
 to move this direction.

 In other hand I need a commercial attractive D infrastructure 
 for large and heavy system projects. There is no commercial 
 perspective for me to contribute to Phobos because:

 1. Phobos version depends on compiler version. Delay with LDC 
 release is too large. It should / can be one day.

 2. We can not do hot fixes in Phobos without compiling and 
 deploying patched libphobos. Mir's DUB package are much more 
 flexible.

 3. Dependencies should be clear. Modularity is a proper way for 
 large std library. In phobos everything integrated with 
 everything. DRuntime -> Phobos abstraction is weird for betterC 
 because system modules can depends universal algorithms, but 
 universal algorithm are more portable if they have not system 
 dependencies.

 4. Phobos does not provide and will not provide betterC 
 guaranties. If something works for betterC mode now it may not 
 work in the future release.

 [1] https://github.com/libmir/
 * - separate github/DUB libraries with clear dependencies.

 Best regards,
 Ilya


I hope this doesn't turn into a new Tango vs. Phobos war.

If you have ideas on how to improve things without messing the 
entire community with ego wars and such, then it sounds good. 
Otherwise thumbs down!

The first sign of faith will be that your modules will always 
work with druntime and phobos - i.e. you wont require a special 
runtime for your stuff to work.

As for Andrei and Walter - I think you need to see why people 
(talented people!) want to side walk the phobos & druntime normal 
contribution routes. Something clearly is wrong there!
Dec 18 2016
next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 18 December 2016 at 14:37:55 UTC, Radu wrote:

 On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko 
 wrote:

 Hi,

 Who is interested in betterC _modular_* standard library?
 I am planing to make libmir org a community for it.
 Thought and concerns?

 We already have better `cpuid` and better `random` packages. 
 The betterC std.range and std.algorithm analogs would be 
 released with new ndslice implementation. Mir's algorithm 
 would be faster then Phobos and will generate less template 
 bloat. Then lightweight threads, multithread GLAS, matrix 
 inversion. Fastest I/O and http2 ...

 (they all are betterC modular libraries)

 Andrei and probably 90% of existing D users don't want Phobos 
 to move this direction.

 In other hand I need a commercial attractive D infrastructure 
 for large and heavy system projects. There is no commercial 
 perspective for me to contribute to Phobos because:

 1. Phobos version depends on compiler version. Delay with LDC 
 release is too large. It should / can be one day.

 2. We can not do hot fixes in Phobos without compiling and 
 deploying patched libphobos. Mir's DUB package are much more 
 flexible.

 3. Dependencies should be clear. Modularity is a proper way 
 for large std library. In phobos everything integrated with 
 everything. DRuntime -> Phobos abstraction is weird for 
 betterC because system modules can depends universal 
 algorithms, but universal algorithm are more portable if they 
 have not system dependencies.

 4. Phobos does not provide and will not provide betterC 
 guaranties. If something works for betterC mode now it may not 
 work in the future release.

 [1] https://github.com/libmir/
 * - separate github/DUB libraries with clear dependencies.

 Best regards,
 Ilya


 I hope this doesn't turn into a new Tango vs. Phobos war.

 If you have ideas on how to improve things without messing the 
 entire community with ego wars and such, then it sounds good. 
 Otherwise thumbs down!


Maybe I will be able to propose them on DConf.


 The first sign of faith will be that your modules will always 
 work with druntime and phobos - i.e. you wont require a special 
 runtime for your stuff to work.


Yes/No in the same time.

1. It would be always possible to use Mir and Phobos together, 
100% guarantee. Furthermore, some Mir libraries are going to be 
infrastructure for _all_, not only for D. For example Mir GLAS 
will target the Julia language after D.

2. In the same time many thing is not possible to create with 
Phobos/Druntime. The simplest example is numeric libraries. I can 
create a multithread GLAS with core.thread, but if I do so GLAS 
will never be accepted as default BLAS implementation for Julia. 
So, i need to create my own low level thread library. Intel MKL, 
OpenBLAS have their own thread management: this is common 
practice.

Cheers,
Ilya
Dec 18 2016
parent reply Radu <void null.pt> writes:
On Sunday, 18 December 2016 at 15:01:55 UTC, Ilya Yaroshenko 
wrote:

 On Sunday, 18 December 2016 at 14:37:55 UTC, Radu wrote:

 [...]


 Maybe I will be able to propose them on DConf.


 [...]


 Yes/No in the same time.

 1. It would be always possible to use Mir and Phobos together, 
 100% guarantee. Furthermore, some Mir libraries are going to be 
 infrastructure for _all_, not only for D. For example Mir GLAS 
 will target the Julia language after D.

 2. In the same time many thing is not possible to create with 
 Phobos/Druntime. The simplest example is numeric libraries. I 
 can create a multithread GLAS with core.thread, but if I do so 
 GLAS will never be accepted as default BLAS implementation for 
 Julia. So, i need to create my own low level thread library. 
 Intel MKL, OpenBLAS have their own thread management: this is 
 common practice.

 Cheers,
 Ilya


I get point 2, but I should be able to use your new modules 
without giving up druntime/phobos, right?. They should not be 
mutual exclusive!
Dec 18 2016
next sibling parent bachmeier <no spam.net> writes:
On Sunday, 18 December 2016 at 15:20:48 UTC, Radu wrote:

 I get point 2, but I should be able to use your new modules 
 without giving up druntime/phobos, right?. They should not be 
 mutual exclusive!


I think that being able to use them the same as any other C 
library should be sufficient.
Dec 18 2016
prev sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 18 December 2016 at 15:20:48 UTC, Radu wrote:

 2. In the same time many thing is not possible to create with 
 Phobos/Druntime. The simplest example is numeric libraries. I 
 can create a multithread GLAS with core.thread, but if I do so 
 GLAS will never be accepted as default BLAS implementation for 
 Julia. So, i need to create my own low level thread library. 
 Intel MKL, OpenBLAS have their own thread management: this is 
 common practice.

 Cheers,
 Ilya


 I get point 2, but I should be able to use your new modules 
 without giving up druntime/phobos, right?. They should not be 
 mutual exclusive!


Yes, you are able to use any Mir libs with Phobos/DRuntime --Ilya
Dec 18 2016
prev sibling parent qznc <qznc web.de> writes:
On Sunday, 18 December 2016 at 14:37:55 UTC, Radu wrote:

 On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko

 Who is interested in betterC _modular_* standard library?
 I am planing to make libmir org a community for it.
 Thought and concerns?


 I hope this doesn't turn into a new Tango vs. Phobos war.


My first thought as well. Probably an irrational fear, but the 
question will pop up often.
Dec 18 2016
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/18/16 4:26 AM, Ilya Yaroshenko wrote:

 Hi,

 Who is interested in betterC _modular_* standard library?
 I am planing to make libmir org a community for it.
 Thought and concerns?

 We already have better `cpuid` and better `random` packages. The betterC
 std.range and std.algorithm analogs would be released with new ndslice
 implementation. Mir's algorithm would be faster then Phobos and will
 generate less template bloat. Then lightweight threads, multithread
 GLAS, matrix inversion. Fastest I/O and http2 ...

 (they all are betterC modular libraries)

 Andrei and probably 90% of existing D users don't want Phobos to move
 this direction.


Please do not misconstrue what I said.

You have my support to improve the state of affairs with the D core 
runtime and standard library, and have had it from day one.

With regard to new functionality that supplants existing pieces such as 
cpuid_v2 or random_v2, definitely. We can and should add that and many 
other great things to phobos. Go for it.

With regard to avoiding to link druntime, we should build the argument 
better. Is all of druntime bad, or just parts of it? E.g. if something 
does not link the GC but does use vtables, is it good/bad, and to what 
extent? These issues are not entirely clear to me and it would be great 
if you explained them better. I definitely need to be better educated on 
this. What I do know is making "don't use druntime" a goal in and of 
itself is not the most useful way to frame the problem.

With regard to changes that break the entire fabric of Phobos (such as: 
we must eliminate random-access ranges), this is definitely more like 
simple survival rather than things I want.

With regard to properly attributing credit for good work, I believe this 
is important and I commit to do anything reasonable for making it happen.

I kindly advise you to better understand what you are trying to do, and 
how to do it to maximize impact. It seems you are not entirely clear on 
what you want to do and what the best route is, as shown by you retiring 
your std.experimental.ndslice work after you've had full freedom on what 
should go in it.


 In other hand I need a commercial attractive D infrastructure for large
 and heavy system projects. There is no commercial perspective for me to
 contribute to Phobos because:

 1. Phobos version depends on compiler version. Delay with LDC release is
 too large. It should / can be one day.


Then a good way to help this is improve the LDC lag.


 2. We can not do hot fixes in Phobos without compiling and deploying
 patched libphobos. Mir's DUB package are much more flexible.


The reverse of that is less stability. The good part about merging your 
work in Phobos is you maximize impact and visibility.

These seem to me artificial arguments. Yes, being preoccupied over quick 
deployment may make sense in the initial stages when there's no 
stability and the design may take any direction. That should get less 
necessary once some abstractions are in place.

Organizations that are interested in patching minute fixes may build 
their own standard library (and sometimes compiler). This is routinely 
the case with C++ at Facebook, Google, and in all likelihood other 
large-scale tech shops. It would be tenuous to argue that this should be 
the default way to go about things for everyone. I don't understand why 
you consider it an essential matter.


 3. Dependencies should be clear. Modularity is a proper way for large
 std library. In phobos everything integrated with everything. DRuntime
 -> Phobos abstraction is weird for betterC because system modules can
 depends universal algorithms, but universal algorithm are more portable
 if they have not system dependencies.


I read this three times and cannot understand it. At any rate: DIP 1005, 
which is partially motivated by our discussions, should allow us to make 
dependencies as simple and clean as function-level.


 4. Phobos does not provide and will not provide betterC guaranties. If
 something works for betterC mode now it may not work in the future release.


Yes, we can definitely improve this.

We (the Foundation) are all for making things better for achieving the 
things you want to achieve. At this point D needs unity and rallying 
under a shared vision, not balkanization in several special interest 
groups. There are so many great things you can do starting literally 
today to make things better for everybody using D, and why in the world 
would anyone want to stop you. I strongly believe your participation to 
the Standard D language and library will be vastly better for you 
personally and professionally.


Andrei
Dec 18 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 18 December 2016 at 15:18:37 UTC, Andrei Alexandrescu 
wrote:

 On 12/18/16 4:26 AM, Ilya Yaroshenko wrote:

 Hi,

 Who is interested in betterC _modular_* standard library?
 I am planing to make libmir org a community for it.
 Thought and concerns?

 We already have better `cpuid` and better `random` packages. 
 The betterC
 std.range and std.algorithm analogs would be released with new 
 ndslice
 implementation. Mir's algorithm would be faster then Phobos 
 and will
 generate less template bloat. Then lightweight threads, 
 multithread
 GLAS, matrix inversion. Fastest I/O and http2 ...

 (they all are betterC modular libraries)

 Andrei and probably 90% of existing D users don't want Phobos 
 to move
 this direction.


 Please do not misconstrue what I said.

 You have my support to improve the state of affairs with the D 
 core runtime and standard library, and have had it from day one.

 With regard to new functionality that supplants existing pieces 
 such as cpuid_v2 or random_v2, definitely. We can and should 
 add that and many other great things to phobos. Go for it.

 With regard to avoiding to link druntime, we should build the 
 argument better. Is all of druntime bad, or just parts of it?


DRuntime is good for applications. But writing multi-language 
libraries with DRuntime is bad idea. The last paragraph in [1] 
describes why DRuntime can not be used for GLAS.

I never said we should "remove GC" or "remove runtime". We just 
need to provide a clear way for betterC subset and modularise 
library. DRuntime and Phobos are well designed hight level 
libraries.


 E.g. if something does not link the GC but does use vtables, is 
 it good/bad, and to what extent?


Very good point. First, I don't see a real reason why we need 
DRuntime for vtables (classes and interfaces). LDC allows to use 
vtables without DRuntime (only with extern(C++), hehe). D classes 
can be constructed on stack or with allocators. The only problem 
is vtable initlization, which requires `typeid` (TypeInfo 
object). I am not a OOP expert, but it looks like we can have 
extern(C++) classes without DRuntime if we solve problem with 
initilization.


 These issues are not entirely clear to me and it would be great 
 if you explained them better. I definitely need to be better 
 educated on this. What I do know is making "don't use druntime" 
 a goal in and of itself is not the most useful way to frame the 
 problem.


0. Linking compatibility. Assume A has builded a betterC library 
AA using GDC version XXX and B has builded a betterC library BB 
using LDC version YYY.

Question: how mister C can use BB and AA together in his 
C-library if both this libraries depends on different DRuntimes 
(compiler and version)?

1. Different mangling names for DMD and LDC. mir-cpuid provides 
extern(C) interface for all functionality except few very low 
level struct declarations. The general problem is that we do not 
have binary compatibility where we may have it. And it is not 
clear where we already have it.

2. Any function may throw an Error.

3. Any throwable is not a part of betterC. Phobos core team 
rejected approach to replace RangeErrors with asserts or another 
function declaration. Possible alternative is __range_assert() or 
a library function.

4. Threads are GC oriented:
     1. They are classes (they can be structs like allocators)
     2. They are integrated with GC. This requires additional 
computation time if I am not wrong.
     3. They are not replaceable. Phobos multithread modules can 
not use something else.

5. Mutexes are GC oriented:
    ditto 1. and 3.

6. Intrinsics are compiler magic without specification. It is not 
clear what it is intrinsic, are they inlined, where their bodies 
are defined, how/if they works in CTFE. LLVM provides a list of 
intrinsics [3], this is very useful to write a betterC libraries.

... and many other ... They all are not clear for me too.

-------------
A solution for mister C:
Mister A and mister B should use a runtime which has either a 
small stable extern(C) interface or all its functions are 
inlined. All other functions can be a dub packages.
-------------

Proof:
cpuid: extern(C) interface with aliases.
a fork of aligned  mallocator for GLAS: all functions are always 
inlined.


 With regard to changes that break the entire fabric of Phobos 
 (such as: we must eliminate random-access ranges), this is 
 definitely more like simple survival rather than things I want.


I don't want eliminate R-A-R now. But we can change our Range 
concept and provide alternative module as dub package, which will 
be announced as a future replacement (say, after 5 years). But 
duplicated sorting implementation with cursors (lightweight 
iterators) in the same module is weird. It is better to continue 
this discussion after the cursor based implementation will be 
ready for your review.


 With regard to properly attributing credit for good work, I 
 believe this is important and I commit to do anything 
 reasonable for making it happen.

 I kindly advise you to better understand what you are trying to 
 do, and how to do it to maximize impact. It seems you are not 
 entirely clear on what you want to do and what the best route 
 is, as shown by you retiring your std.experimental.ndslice work 
 after you've had full freedom on what should go in it.


 In other hand I need a commercial attractive D infrastructure 
 for large
 and heavy system projects. There is no commercial perspective 
 for me to
 contribute to Phobos because:

 1. Phobos version depends on compiler version. Delay with LDC 
 release is
 too large. It should / can be one day.


 Then a good way to help this is improve the LDC lag.


I mean lag between new Phobos PR and LDC release. This includes 
time for PR review, DMD release, LDC release. Libraries should be 
separated from a compiler, IMHO.


 2. We can not do hot fixes in Phobos without compiling and 
 deploying
 patched libphobos. Mir's DUB package are much more flexible.


 The reverse of that is less stability.


A hot fix can be added in local repository without creating a PR. 
DUB allows to do it.


 The good part about merging your work in Phobos is you maximize 
 impact and visibility.


Yes, but merging existing libraries to Phobos will slowdown other 
projects as DCV. Also  I will spend a lot efforts on stuff like 
repositories synchronisation. Plus it makes my users unhappy: 
they will need to switch from Phobos to Mir if they want to use 
recent addition.


 These seem to me artificial arguments. Yes, being preoccupied 
 over quick deployment may make sense in the initial stages when 
 there's no stability and the design may take any direction. 
 That should get less necessary once some abstractions are in 
 place.


C++ adds new APIs one time per few years. D add few changes every 
month.

Phobos and DRuntime does not behave as traditional standard 
system language libraries. They are more like a scripting 
language heaps which are grows very fast (like scipy).

I wan't a D Standard Library to be exist too, but it should be 
slim, has better backward compatibility, ideal backward binary 
compatibility, has good low level specification and 
documentation. Large hight level library like Phobos can not do 
it with community support only.


 Organizations that are interested in patching minute fixes may 
 build their own standard library (and sometimes compiler). This 
 is routinely the case with C++ at Facebook, Google, and in all 
 likelihood other large-scale tech shops. It would be tenuous to 
 argue that this should be the default way to go about things 
 for everyone. I don't understand why you consider it an 
 essential matter.


Good point. Why not to be more user friendly for large-scale tech 
companies then?

If we can show that merging too LDC requires less efforts then 
upgrading from g++ to clang++ ...


 3. Dependencies should be clear. Modularity is a proper way 
 for large
 std library. In phobos everything integrated with everything. 
 DRuntime
 -> Phobos abstraction is weird for betterC because system 
 modules can
 depends universal algorithms, but universal algorithm are more 
 portable
 if they have not system dependencies.


 I read this three times and cannot understand it. At any rate: 
 DIP 1005, which is partially motivated by our discussions, 
 should allow us to make dependencies as simple and clean as 
 function-level.


The  DIP 1005 idea is good. But this will fix few percentage of 
existing dependency ambiguity. Plus it does not solve the 
dependency problem, it just makes it slightly more visible.


 4. Phobos does not provide and will not provide betterC 
 guaranties. If
 something works for betterC mode now it may not work in the 
 future release.


 Yes, we can definitely improve this.

 We (the Foundation) are all for making things better for 
 achieving the things you want to achieve. At this point D needs 
 unity and rallying under a shared vision, not balkanization in 
 several special interest groups. There are so many great things 
 you can do starting literally today to make things better for 
 everybody using D, and why in the world would anyone want to 
 stop you. I strongly believe your participation to the Standard 
 D language and library will be vastly better for you personally 
 and professionally.


 Andrei


I will continue in the answer to Walter.

Thank you for the answer (it is hard to understand me because 
English and other reasons),

Ilya

[1] 
http://forum.dlang.org/post/jwmzvfashovivqzamxrq forum.dlang.org
[2] 
http://forum.dlang.org/post/jneokbosxeatvkppbuvf forum.dlang.org
[3] http://llvm.org/docs/LangRef.html#intrinsic-functions
Dec 18 2016
parent reply safety0ff <safety0ff.dev gmail.com> writes:
On Sunday, 18 December 2016 at 18:02:58 UTC, Ilya Yaroshenko 
wrote:

 Thank you for the answer (it is hard to understand me because 
 English and other reasons),

 Ilya


It was difficult to understand your vision until this post, now I 
think I grasp it.

Let me try to summarize what I've understood:

D as it stands, is not suitable for writing low level libraries 
as well as large scale software development because of compiler 
dependence of compiled code.

Examples:
If you have two software teams, and team A's software dependent 
on compiler X (e.g requires newer feature, requires 
bug/regression fix, etc) and team B's software depends on 
compiler Y to meet performance requirements, they get stuck.

Also you want to create a low level library than can be easily 
distributed and linked from other languages (e.g. GLAS) extern 
(C) is the only viable option, but that can still lock in the D 
compiler used if you depend on phobos/druntime.


So the proposal is to make binary compatibility possible in the 
near future by implementing "betterC" which provides a bare-bones 
language and removes the greatest sources of incompatibilities.

Once this is done a community can form around it and create 
completely modular libraries. These can be used by all D and 
non-D users alike without compatibility problems.


Since this is all predicated on "betterC", which isn't 
implemented. I think it is imperative to create a full 
specification.

I look forward to seeing where this initiative goes.

P.S.: I think Ilya writes "evaluates" where he means "evolves"
Dec 19 2016
parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 18:04:14 UTC, safety0ff wrote:

 On Sunday, 18 December 2016 at 18:02:58 UTC, Ilya Yaroshenko 
 wrote:

 Thank you for the answer (it is hard to understand me because 
 English and other reasons),

 Ilya


 It was difficult to understand your vision until this post, now 
 I think I grasp it.

 Let me try to summarize what I've understood:
 [...]


Yes, thank you for the clarification!


 P.S.: I think Ilya writes "evaluates" where he means "evolves"


Oh, yes! I mean "evolves"!
Dec 19 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote:

 We already have better `cpuid` and better `random` packages.


Great! Please PR them for Phobos.



 The betterC
 std.range and std.algorithm analogs would be released with new ndslice
 implementation. Mir's algorithm would be faster then Phobos and will generate
 less template bloat. Then lightweight threads, multithread GLAS, matrix
 inversion. Fastest I/O and http2 ...

 (they all are betterC modular libraries)


Propose them for Phobos!



 Andrei and probably 90% of existing D users don't want Phobos to move this
 direction.


I don't understand that point. What direction?



 In other hand I need a commercial attractive D infrastructure for large and
 heavy system projects. There is no commercial perspective for me to contribute
 to Phobos because:

 1. Phobos version depends on compiler version. Delay with LDC release is too
 large. It should / can be one day.


The language evolves over time, and the standard library must, too. It happens 
with every language.



 2. We can not do hot fixes in Phobos without compiling and deploying patched
 libphobos. Mir's DUB package are much more flexible.

 3. Dependencies should be clear. Modularity is a proper way for large std
 library. In phobos everything integrated with everything. DRuntime -> Phobos
 abstraction is weird for betterC because system modules can depends universal
 algorithms, but universal algorithm are more portable if they have not system
 dependencies.


Not sure what you mean. Algorithms in Phobos are not dependent on system 
idiosyncracies.



 4. Phobos does not provide and will not provide betterC guaranties. If
something
 works for betterC mode now it may not work in the future release.


I've done a fair amount of work to remove such things. Clearly more can and 
should be done.

Generally, we do have a goal of making Phobos entirely "pay as you go" instead 
of being so interconnected (which is what betterC is really all about). We can 
definitely use help in improving that and helping it along.
Dec 18 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote:

 On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote:

 We already have better `cpuid` and better `random` packages.


 Great! Please PR them for Phobos.


cpuid is used in Mir GLAS and it should be a betterC library. 
Without DRuntime dependency because DRuntime has not backward 
binary compatibility and has not compatibility between different 
compilers.

 From my reply to Andrei:

----
Linking compatibility. Assume A has builded a betterC library AA 
using GDC version XXX and B has builded a betterC library BB 
using LDC version YYY.

Question: how mister C can use BB and AA together in his 
C-library if both this libraries depends on different DRuntimes 
(compiler and version)?
----

mir-glas and mir-cpuid are libraries with extern(C) interface for 
all languages. It is silly to build a BLAS implementation only 
for one language.


 The betterC
 std.range and std.algorithm analogs would be released with new 
 ndslice
 implementation. Mir's algorithm would be faster then Phobos 
 and will generate
 less template bloat. Then lightweight threads, multithread 
 GLAS, matrix
 inversion. Fastest I/O and http2 ...

 (they all are betterC modular libraries)


 Propose them for Phobos!


First I need to implement them and announce for the community. 
Then community will say "What he is doing??? C++ iterators, 
really?" Hehe

This is reason why ndslice is deprecated in Phobos. It requires 
another std.algorithm and std.range. I am not able to implement 
all this ideas in Phobos PRs: it is requires too much time. In 
addition, I can not use Phobos for other reasons.


 Andrei and probably 90% of existing D users don't want Phobos 
 to move this
 direction.


 I don't understand that point. What direction?


1. Modularity: D should provide a very slim library (like 
std.traits and may be default GC with old core, which can be 
turned off or replaced). Other parts or Phobos/Druntime should be 
split into parts and the parts should have their own 
repositories. They should be dub packages with D Foundation 
review / control.

2. Low-level API first! A high level OOP/GC-oriented API can be 
provided later or can be provided in DUB libraries.

3. betterC guaranties / No problems with backward binary 
compatibility for betterC libs. This means that code should be 
templated / inlined. Some parts, like Exceptions and GC may have 
this problems, but Exceptions and GC should be optional language 
features and they are not a part of betterC mode.


 In other hand I need a commercial attractive D infrastructure 
 for large and
 heavy system projects. There is no commercial perspective for 
 me to contribute
 to Phobos because:

 1. Phobos version depends on compiler version. Delay with LDC 
 release is too
 large. It should / can be one day.


 The language evolves over time, and the standard library must, 
 too. It happens with every language.


Agreed. Because that it is more flexible to have a very slim std 
library with good set of community supported libraries.

This is a reason why almost all portable multi-language numeric 
libraries are written in C. C runtime is slim and evaluates 
very-very slow.


 2. We can not do hot fixes in Phobos without compiling and 
 deploying patched
 libphobos. Mir's DUB package are much more flexible.

 3. Dependencies should be clear. Modularity is a proper way 
 for large std
 library. In phobos everything integrated with everything. 
 DRuntime -> Phobos
 abstraction is weird for betterC because system modules can 
 depends universal
 algorithms, but universal algorithm are more portable if they 
 have not system
 dependencies.


 Not sure what you mean. Algorithms in Phobos are not dependent 
 on system idiosyncracies.


They are, for example they depends on RangeErrors. See
https://github.com/dlang/phobos/pull/4937


 4. Phobos does not provide and will not provide betterC 
 guaranties. If something
 works for betterC mode now it may not work in the future 
 release.


 I've done a fair amount of work to remove such things. Clearly 
 more can and should be done.

 Generally, we do have a goal of making Phobos entirely "pay as 
 you go" instead of being so interconnected (which is what 
 betterC is really all about). We can definitely use help in 
 improving that and helping it along.


Great. Why not to go to the proposed direction:
     1 Modularity,
     2 Low-level API first,
     3 betterC guaranties?

Thank you,
Ilya
Dec 18 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote:

 On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote:

 On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote:

 We already have better `cpuid` and better `random` packages.


 Great! Please PR them for Phobos.


 cpuid is used in Mir GLAS and it should be a betterC library.


It's this kind of imaginary dialog that I don't quite grok:

Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my own."

Andrei Alexandrescu: "OK, so what's different?"

IY: "Mine has a few engineering improvements."

AA: "Cool, why don't you merge them into core.cpuid?"

IY: "Well mine doesn't have a shared static constructor, so it doesn't 
need a runtime to automatically call that library initialization 
function. User need to explicitly call an init function before using it."

AA: "I understand. Great, so how about this - we add your code to 
core.cpuid_v2 to druntime. Then we make all in core.cpuid to forward to 
it so there's no duplication. It all works out!"

IY: "No, I don't want to do that. It's still in druntime and I don't 
want druntime. I want betterC."

AA: "But it will be compilable with betterC and we can add unittests to 
make that happen. YOU HAVE MY SUPPORT. Let's do it."

IY: "No, I want to change it often. The deployment schedule of druntime 
is too slow."

AA: "How often do you need to change it? Is it that unstable?"

IY: "Um, not too often."

AA: "Then what is the matter? Are you worried about the IP of the 
engineering improvements you are making? Are you licensing this 
differently?"

IY: "No, it's for the most part similar to core.cpuid and the license 
will be also Boost."

AA: "Then what is the matter? Do you want me to wait until you release a 
stable mir.cpuid and copy it over with credit, per the Boost license, to 
core.cpuid_v2?"

IY: "..."

It's this kind of stuff I need to have a better understanding of. Some 
technical arguments are meaningful, some others point to problems with 
obvious solutions that are somehow shunned, and yet others are like a 
hidden Markov model - I see the effects, but I don't see the causes.


Andrei
Dec 18 2016
next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 18 December 2016 at 19:28:24 UTC, Andrei Alexandrescu 
wrote:

 On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote:

 On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright 
 wrote:

 On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote:

 We already have better `cpuid` and better `random` packages.


 Great! Please PR them for Phobos.


 cpuid is used in Mir GLAS and it should be a betterC library.


 It's this kind of imaginary dialog that I don't quite grok:

 Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my 
 own." [...]
 IY: "..."

 It's this kind of stuff I need to have a better understanding 
 of. Some technical arguments are meaningful, some others point 
 to problems with obvious solutions that are somehow shunned, 
 and yet others are like a hidden Markov model - I see the 
 effects, but I don't see the causes.


 Andrei



My last argument was:
----
Linking compatibility. Assume A has builded a betterC library AA 
using GDC version XXX and B has builded a betterC library BB 
using LDC version YYY.

Question: how mister C can use BB and AA together in his 
C-library if both this libraries depends on different DRuntimes 
(compiler and version)?
----

DRuntime is large and evaluates fast. We have not backward binary 
compatibility in DRuntime and we have not binary compatibility 
between different compiler versions.

mir-glas is a betterC library for all languages.  It depends in 
mir-cpuid. I want mir-glas to replace OpenBLAS. To do it I need 
the same portability.

OK, assume we already have DRuntime for all targets including 
MIPS, POWER, armv7, aarch64, Loongson.

Does this solves problem? No.

The reason is very simple. All non-betterC D libraries must be 
compiled in lockstep: with the same compiler vendor and druntime 
version!

This will just kill GLAS. How this can be packed for Ubuntu for 
example? It will depends on LDC libphobos == XXX. Another numeric 
library, say FFT, will depend on libphobos == YYY by GDC. How 
they can be linked if they require different DRuntime from 
different compilers? Or different druntime versions for the same 
compiler?

Do you want to constraint D users to do not distribute binary 
libraries in popular repositories like Debian/Ubuntu?

DRuntime incompatibility is a huge constraints for business. Do 
you want to force all companies to use identical version of 
compiler and identical vendor?

Yes, C++ changes their API, but it few orders of magnitude slower 
then D. And they support multiple branches with bugs fixes. D 
supports only the recent one.

This is why GLAS/cpuid need to be a betterC library.

Ilya
Dec 18 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/18/2016 03:10 PM, Ilya Yaroshenko wrote:

 On Sunday, 18 December 2016 at 19:28:24 UTC, Andrei Alexandrescu wrote:

 On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote:

 On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright wrote:

 On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote:

 We already have better `cpuid` and better `random` packages.


 Great! Please PR them for Phobos.


 cpuid is used in Mir GLAS and it should be a betterC library.


 It's this kind of imaginary dialog that I don't quite grok:

 Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my own."
 [...]
 IY: "..."

 It's this kind of stuff I need to have a better understanding of. Some
 technical arguments are meaningful, some others point to problems with
 obvious solutions that are somehow shunned, and yet others are like a
 hidden Markov model - I see the effects, but I don't see the causes.


 Andrei



 My last argument was:
 ----
 Linking compatibility. Assume A has builded a betterC library AA using
 GDC version XXX and B has builded a betterC library BB using LDC version
 YYY.

 Question: how mister C can use BB and AA together in his C-library if
 both this libraries depends on different DRuntimes (compiler and version)?
 ----


My understanding is that if cpuid_v2.d does not pull any dependency, it 
doesn't matter whether it sits in druntime or elsewhere. Is that the case?


 DRuntime is large and evaluates fast.


So is it moving too slow or too fast?


 We have not backward binary
 compatibility in DRuntime and we have not binary compatibility between
 different compiler versions.


Even if we don't, we can guarantee that core.cpuid_v2 adds no link-time 
dependency. We may offer it as a pure template library, or as C APIs. 
Whatever floats your boat. Is this enough for ensuring compatibility?



Andrei
Dec 18 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Sunday, 18 December 2016 at 20:16:11 UTC, Andrei Alexandrescu 
wrote:

 On 12/18/2016 03:10 PM, Ilya Yaroshenko wrote:

 On Sunday, 18 December 2016 at 19:28:24 UTC, Andrei 
 Alexandrescu wrote:

 On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote:

 On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright 
 wrote:

 On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote:

 We already have better `cpuid` and better `random` 
 packages.


 Great! Please PR them for Phobos.


 cpuid is used in Mir GLAS and it should be a betterC library.


 It's this kind of imaginary dialog that I don't quite grok:

 Ilya Yaroshenko: "I can't use druntime's cpuid so I defined 
 my own."
 [...]
 IY: "..."

 It's this kind of stuff I need to have a better understanding 
 of. Some
 technical arguments are meaningful, some others point to 
 problems with
 obvious solutions that are somehow shunned, and yet others 
 are like a
 hidden Markov model - I see the effects, but I don't see the 
 causes.


 Andrei



 My last argument was:
 ----
 Linking compatibility. Assume A has builded a betterC library 
 AA using
 GDC version XXX and B has builded a betterC library BB using 
 LDC version
 YYY.

 Question: how mister C can use BB and AA together in his 
 C-library if
 both this libraries depends on different DRuntimes (compiler 
 and version)?
 ----


 My understanding is that if cpuid_v2.d does not pull any 
 dependency, it doesn't matter whether it sits in druntime or 
 elsewhere. Is that the case?


 DRuntime is large and evaluates fast.


 So is it moving too slow or too fast?


It is too slow as library to contribute to and too fast as 
standard library to depend on.


 We have not backward binary
 compatibility in DRuntime and we have not binary compatibility 
 between
 different compiler versions.


 Even if we don't, we can guarantee that core.cpuid_v2 adds no 
 link-time dependency. We may offer it as a pure template 
 library, or as C APIs. Whatever floats your boat. Is this 
 enough for ensuring compatibility?


I need to understand how it is can be done because I did not find 
a valid solution.

mir-cpuid has global symbols, they are accessed using extern(C) 
API. If multiple libraries trying to initialize it than it will 
be initialized only once.

extern C API does not solve a problem with DRuntime because 
current DRuntime is not a betterC library and it is huge. A huge 
betterC library is a problem too because it is less portable.

Anyway, if you find a proper solution please share it. How it can 
be implemented as template library? Does different libraries will 
have their own cpuid instantiations and __gshared symbols?

How modularity can break existing Phobos? Why not to allow new 
packages like mir-cpuid and std.allocator to be a betterC library 
with their own repository? This will not break existing Phobos 
and Druntime but this will allow me to write server and numeric 
software without copy-pasting Phobos codes. I prefer to fork 
something rather then copypast.

My plans to work on:
1. Low-level numeric libraries
2. Low-level fast async I/O adaptive generic event loop, 
asymmetric green threads, http2
3. Microelectromechanical systems

They will require new threads, new timers, new mutexes, new event 
loop (existing C libs are slow), to be used.

Existing one does not satisfy my requirements.

I do not earn any money for this work.

I am working on this projects because I believe in D and believe 
that my current and future clients will believe in D much more 
then now. I know what they need.

Ilya
Dec 18 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/18/2016 1:02 PM, Ilya Yaroshenko wrote:

 I need to understand how it is can be done because I did not find a valid
solution.

 mir-cpuid has global symbols, they are accessed using extern(C) API. If
multiple
 libraries trying to initialize it than it will be initialized only once.

 extern C API does not solve a problem with DRuntime because current DRuntime is
 not a betterC library and it is huge. A huge betterC library is a problem too
 because it is less portable.


cpuid can be written in a manner that it is self-contained. If so, it doesn't 
matter how large druntime is, as it won't have references to any of the rest of 
it and so the rest won't be pulled in by the linker.

https://github.com/dlang/druntime/blob/master/src/core/cpuid.d

Currently, core.cpuid has no imports. It does have a static constructor,
though, 
which references the startup code in druntime. This can be changed so it uses 
the static construction support in the C runtime library. Then it will be 
independent of druntime.
Dec 20 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Tuesday, 20 December 2016 at 12:00:43 UTC, Walter Bright wrote:

 On 12/18/2016 1:02 PM, Ilya Yaroshenko wrote:

 I need to understand how it is can be done because I did not 
 find a valid solution.

 mir-cpuid has global symbols, they are accessed using 
 extern(C) API. If multiple
 libraries trying to initialize it than it will be initialized 
 only once.

 extern C API does not solve a problem with DRuntime because 
 current DRuntime is
 not a betterC library and it is huge. A huge betterC library 
 is a problem too
 because it is less portable.


 cpuid can be written in a manner that it is self-contained. If 
 so, it doesn't matter how large druntime is, as it won't have 
 references to any of the rest of it and so the rest won't be 
 pulled in by the linker.


This is true for application but not for libraries. mir-glas 
depends on mir-cpuid. If I want GLAS to be ported for a new 
target then I will release a new CPUID version and will be able 
to use it the same day. With DRuntime (as solid project), I need 
to wait up to 6 months before it will be released with DMD and 
then with LDC. Workaround are possible, but they requires more 
efforts, more time, more code.

Please do not force Mir to use DRuntime. We need a solid solution

I do not force anyone to drop DRuntime: Mir libraries can be used 
with DRuntime/Phobos. We have full backward compatibility. 
DRuntime does not satisfy portability requirements (which was 
repeated multiple times in this thread). DRuntime (at least for 
now and future few years) is a big constraint for betterC libs to 
replace existing C analogs. Please do not force me to use 
DRuntime.
Dec 20 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/20/2016 4:30 AM, Ilya Yaroshenko wrote:

 This is true for application but not for libraries. mir-glas depends on
 mir-cpuid. If I want GLAS to be ported for a new target then I will release a
 new CPUID version and will be able to use it the same day. With DRuntime (as
 solid project), I need to wait up to 6 months before it will be released with
 DMD and then with LDC. Workaround are possible, but they requires more efforts,
 more time, more code.

 Please do not force Mir to use DRuntime. We need a solid solution

 I do not force anyone to drop DRuntime: Mir libraries can be used with
 DRuntime/Phobos. We have full backward compatibility. DRuntime does not satisfy
 portability requirements (which was repeated multiple times in this thread).
 DRuntime (at least for now and future few years) is a big constraint for
betterC
 libs to replace existing C analogs.


Here's how anyone can override druntime's cpuid for their application without 
touching druntime:

     dmd -c mycpuid
     dmd myapp mycpuid.o

I.e. the way linkers work is if the symbol's definition exists, it is not
pulled 
in from the library.


 Please do not force me to use DRuntime.


I don't propose forcing anyone to use Druntime. What I do propose is reducing 
interdependencies among the object files in Druntime. Druntime is not a 
monolithic chunk of code, it is a library of object files.

I don't see the need to rewrite cpuid because its sole dependency on other 
object files in Druntime is the static constructor, because the static 
constructor can be adjusted to not rely on anything else in Druntime.

If there are other issues with cpuid, please elucidate.
Dec 20 2016
next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Tuesday, 20 December 2016 at 12:49:53 UTC, Walter Bright wrote:


 I don't see the need to rewrite cpuid because its sole 
 dependency on other object files in Druntime is the static 
 constructor, because the static constructor can be adjusted to 
 not rely on anything else in Druntime.

 If there are other issues with cpuid, please elucidate.


1. https://issues.dlang.org/show_bug.cgi?id=16028
2. The code is very hard to maintain, fix, extend. This is a 
reason why mir-cpuid was written from scratch without any 
copy-pasting.
3. Library is x86 oriented. Some OSs has interface to fetch 
unified CPU information.
4. No TLB information provided.
5. Number of CPUs per cache is not know. ARM CPUs may have 
complex topology (plus 3.)
Dec 20 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/20/2016 5:35 AM, Ilya Yaroshenko wrote:

 On Tuesday, 20 December 2016 at 12:49:53 UTC, Walter Bright wrote:


 I don't see the need to rewrite cpuid because its sole dependency on other
 object files in Druntime is the static constructor, because the static
 constructor can be adjusted to not rely on anything else in Druntime.

 If there are other issues with cpuid, please elucidate.


 1. https://issues.dlang.org/show_bug.cgi?id=16028
 2. The code is very hard to maintain, fix, extend. This is a reason why
 mir-cpuid was written from scratch without any copy-pasting.
 3. Library is x86 oriented. Some OSs has interface to fetch unified CPU
 information.
 4. No TLB information provided.
 5. Number of CPUs per cache is not know. ARM CPUs may have complex topology
 (plus 3.)


Thank you, but I meant issues that would require it to be separate from 
druntime, not bug fixes or enhancements to account for ever more things being 
added to the CPUID instruction.

BTW, of course it is x86 oriented. Any use of cpuid must be. After all, what 
does "SSSE3.1" mean on ARM?
Dec 20 2016
prev sibling parent Mike <none none.com> writes:
On Tuesday, 20 December 2016 at 12:49:53 UTC, Walter Bright wrote:

 I don't propose forcing anyone to use Druntime.


Perhaps this is not part of Ilya's concern, but druntime is 
required to get a build.

   //----test.d---------
   module test;

   void main() { }

   dmd -m64 -defaultlib= -debuglib= -conf= test.d

   Error: cannot find source code for runtime library file 
'object.d'
        dmd might not be correctly installed. Run 'dmd -man' for 
installation
        instructions.
        config file: (null)
   Specify path to file 'object.d' with -I switch


There's nothing here that really requires druntime, unless you 
want to count the plumbing required to call DMain.  In that case, 
we can just call CMain directly:

   module test;

   extern(C) void main() { }

   Same Error:
   Error: cannot find source code for runtime library file 
'object.d'


Try with the -betterC switch:

   dmd -m64 -defaultlib= -debuglib= -conf= -betterC test.d

   Same Error:
   Error: cannot find source code for runtime library file 
'object.d'

Ok, fine, let's add an empty object.d

   //object.d
   module object;

   //test.d
   module test;

   extern(C) void main() { }

   dmd -m64 -defaultlib= -debuglib= -conf= -betterC object.d test.d

   Error: undefined identifier 'Error'
   Error: undefined identifier 'Error'

Ok, now I need something called 'Error'.  I don't see anywhere in 
my code where that's necessary, but it must be important because 
it told me twice.

   //object.d
   module object;

   class Throwable { }

   class Error : Throwable { }

   //test.d
   module test;

   extern(C) void main() { }

   dmd -m64 -defaultlib= -debuglib= -conf= -betterC object.d test.d

   object.d(3): Error: class object.Throwable missing or corrupt 
object.d

Ok, that's weird. I just added it.  Luckily I've done this 
before, so I know what to do.

   //object.d
   module object;

   class Object { }

   class Throwable { }

   class Error : Throwable
   {
       this(string x) { }
   }

   //test.d
   module test;

   extern(C) void main() { }

   dmd -m64 -defaultlib= -debuglib= -conf= -betterC object.d test.d

   Error: no constructor for Error
   Error: no constructor for Error

Hmm. Again with the double error messages.  Ok, let's add a 
constructor.

   //object.d
   module object;

   class Object { }

   class Throwable { }

   class Error : Throwable
   {
       this(string x) { }
   }

   //test.d
   module test;

   extern(C) void main() { }

   dmd -m64 -defaultlib= -debuglib= -conf= -betterC object.d test.d

   object.d(9): Error: undefined identifier 'string'

At least its only telling me once.  Easy fix.

   //object.d
   module object;

   alias immutable(char)[] string;

   class Object { }

   class Throwable { }

   class Error : Throwable
   {
       this(string x) { }
   }

   //test.d
   module test;

   extern(C) void main() { }

   dmd -m64 -defaultlib= -debuglib= -conf= -betterC object.d test.d

   Error: TypeInfo not found. object.d may be incorrectly 
installed or corrupt,
     compile with -v switch

I could go on, but I'm not going to.  The final result to get a 
build looks like this, but you have to link with --gc-sections to 
avoid having to implement a buch of other stuff.

   module object;

   extern(C) void __dmd_personality_v0() { }

   extern(C) void _d_dso_registry() { }

   alias immutable(char)[] string;

   class Object { }

   class TypeInfo { }

   class TypeInfo_Class : TypeInfo
   {
       ubyte[136] ignore;
   }

   alias TypeInfo_Class ClassInfo;

   class Throwable { }

   class Error : Throwable
   {
       this(string x) { }
   }

   //test.d
   module test;

   extern(C) void main() { }

   dmd -m64 -defaultlib= -debuglib= -conf= -betterC object.d 
test.d -L=--gc-sections

If you add structs or classes to your code, you have to add even 
more druntime stuff.

The point I'm trying to make here is druntime is required just to 
get a build, even though your code doesn't need it.

Mike
Dec 20 2016
prev sibling parent Andrey <avraliov gmail.com> writes:
On Sunday, 18 December 2016 at 19:28:24 UTC, Andrei Alexandrescu 
wrote:

 On 12/18/2016 01:49 PM, Ilya Yaroshenko wrote:

 On Sunday, 18 December 2016 at 17:41:31 UTC, Walter Bright 
 wrote:

 On 12/18/2016 1:26 AM, Ilya Yaroshenko wrote:

 We already have better `cpuid` and better `random` packages.


 Great! Please PR them for Phobos.


 cpuid is used in Mir GLAS and it should be a betterC library.


 It's this kind of imaginary dialog that I don't quite grok:

 Ilya Yaroshenko: "I can't use druntime's cpuid so I defined my 
 own."

 Andrei Alexandrescu: "OK, so what's different?"

 IY: "Mine has a few engineering improvements."

 AA: "Cool, why don't you merge them into core.cpuid?"

 IY: "Well mine doesn't have a shared static constructor, so it 
 doesn't need a runtime to automatically call that library 
 initialization function. User need to explicitly call an init 
 function before using it."

 AA: "I understand. Great, so how about this - we add your code 
 to core.cpuid_v2 to druntime. Then we make all in core.cpuid to 
 forward to it so there's no duplication. It all works out!"

 IY: "No, I don't want to do that. It's still in druntime and I 
 don't want druntime. I want betterC."

 AA: "But it will be compilable with betterC and we can add 
 unittests to make that happen. YOU HAVE MY SUPPORT. Let's do 
 it."

 IY: "No, I want to change it often. The deployment schedule of 
 druntime is too slow."

 AA: "How often do you need to change it? Is it that unstable?"

 IY: "Um, not too often."

 AA: "Then what is the matter? Are you worried about the IP of 
 the engineering improvements you are making? Are you licensing 
 this differently?"

 IY: "No, it's for the most part similar to core.cpuid and the 
 license will be also Boost."

 AA: "Then what is the matter? Do you want me to wait until you 
 release a stable mir.cpuid and copy it over with credit, per 
 the Boost license, to core.cpuid_v2?"

 IY: "..."

 It's this kind of stuff I need to have a better understanding 
 of. Some technical arguments are meaningful, some others point 
 to problems with obvious solutions that are somehow shunned, 
 and yet others are like a hidden Markov model - I see the 
 effects, but I don't see the causes.


 Andrei


I think, the reason of misunderstanding is different skills in a 
subject. And even different subjects. Some experts are compilers 
developers, some are library developers. They all in different 
subject. In c++ community there is a group of compiler developers 
experts who desides, how to implement this or that feature, and 
the ability of implementation. They know how to develop 
compilers. It doesn't mean they are more clever than library 
developers, just different skills. And when Andrei asking to 
explain the reason of some proposal, he waits arguments in 
_terminology_ of comliler developer. _terminology_ understanding 
(in wide) means _experience_ (compiler developing in this case). 
If no experience in the subject, there is a way to get it, and 
the other way to ask a recommendation of experienced colleague.
Dec 18 2016
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Sunday, 18 December 2016 at 18:49:33 UTC, Ilya Yaroshenko 
wrote:

 1. Modularity: D should provide a very slim library (like 
 std.traits and may be default GC with old core, which can be 
 turned off or replaced). Other parts or Phobos/Druntime should 
 be split into parts and the parts should have their own 
 repositories. They should be dub packages with D Foundation 
 review / control.


This would be awesome to have!!

tl:dr: Phobos is very bloated. Let's do something about it!

I absolutely agree that there's a lot of stuff that doesn't 
belong into Phobos.
Also many parts are outdated or have ugly APIs, but they can only 
be replaced if a superior community solution has evolved.
So why not "dubify" the optional parts of Phobos?

Solved Problem: Versioning
--------------------------

This also solves the versioning problem Andrei discussed last 
month nicely. For example when std.allocator was merged into 
Phobos a lot of builds broke because packages did can their 
dependency from dub to std.x.allocator, but many people were 
still compiling with an older compiler.

Solved Problem: High quality community libraries
------------------------------------------------

Moreover, at the last DConf there were many voices that D is 
lacking good community libraries (aka the gold libraries). So the 
Phobos collection could be one.

Btw one shouldn't forgot that people have already written a lot 
of replacements of Phobos modules and imho in most cases it made 
sense!

A quick list
------------

(this list is incomplete and intended as an example to show that 
for most parts of Phobos better solutions co-exists)

etc.c.sqlite (aka std.database)

-> https://github.com/buggins/ddbc
(and: https://wiki.dlang.org/Database_Libraries)

etc.c.zlib / std.zip

-> https://github.com/rcythr/archive

std.base64

-> https://code.dlang.org/packages/base-d

std.bigint

-> 
https://github.com/andersonpd/eris/blob/master/integer/extended.d

std.csv

-> https://github.com/eBay/tsv-utils-dlang

std.container

-> https://github.com/economicmodeling/containers (uses 
allocators!)

std.complex

I am pretty sure this is on Ilya's list as well ;-)

std.encoding:

-> https://github.com/e10s/d-base32

std.getopt

-> https://github.com/jasonwhite/darg
-> https://github.com/SirTony/commando
etc.

std.json:

-> https://github.com/s-ludwig/std_data_json

std.math:

-> https://code.dlang.org/packages/ctstdmath
-> https://github.com/libmir/mir-math

std.net.curl:

-> https://github.com/ikod/dlang-requests

std.net.isemail:

-> https://github.com/anton-dutov/mail
-> http://vibed.org/api/vibe.mail.smtp/Mail

std.signals:

-> https://code.dlang.org/packages/phobosx
-> https://code.dlang.org/packages/observe

std.socket:

-> http://vibed.org/api/vibe.http.websockets/
-> http://vibed.org/api/vibe.core.net/TCPConnection

std.stdio:

-> https://github.com/jasonwhite/io
-> https://github.com/schveiguy/iopipe
-> 
https://github.com/rejectedsoftware/vibe.d/blob/master/stream/vibe/stream/stdio.d

std.uri:

-> 
https://github.com/rikkimax/alphaPhobos/blob/master/source/std/experimental/uri.d
-> http://vibed.org/api/vibe.inet.url/
-> https://github.com/adamdruppe/arsd/blob/master/http2.d

std.variant

-> https://github.com/s-ludwig/taggedalgebraic

std.xml

-> https://github.com/lodo1995/experimental.xml
-> https://github.com/jacob-carlborg/orange


Also a lot of proposed Phobos modules already exist in the 
community:

std.color

https://github.com/TurkeyMan/color

std.decimal

https://github.com/andersonpd/eris/
https://github.com/jaypha/fixed

std.events (planned?)

-> https://github.com/etcimon/libasync


Of course many of these libraries are one man projects and aren't 
considered "high-quality", but the reason they exist in the first 
place shows that a lot of things just shouldn't be 
"standardized", but more provided with a "practical variant". We 
should focus solely on modules that can be standardized easily 
(std.traits, std.ranges, ...) or are required to create a unified 
experience (e.g. a common allocation/logging interface). Imho 
there's really no need to ship sth. like SQLite as part of the 
standard library.

Final remarks
-------------

- From a technical point of view there shouldn't be a problem to 
have a unified documentation experience etc.
- Thanks to Martin's awesome work, we already test a couple of 
"selected" DUB packages on every DMD/Druntime/Phobos commit
Dec 18 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/18/16 7:31 PM, Seb wrote:

 tl:dr: Phobos is very bloated.


How does the D standard library compare in size with some other 
languages (C++, Rust, Go, Java, Python)?


 Let's do something about it!


I've said this several times, and it bears repeating. We want to make D 
entirely pay-as-you go, and offer full support to anyone working on it. 
The best "something" anyone can do about it is do good engineering work 
in the D mainline. That will automatically benefit all users of the D 
language, instead of balkanizing the community. Granted, the lure of the 
clean slate is strong, and more care is needed to mind backwards 
compatibility, but that's where the real impact is.


Andrei
Dec 18 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 01:52:21 UTC, Andrei Alexandrescu 
wrote:

 On 12/18/16 7:31 PM, Seb wrote:

 tl:dr: Phobos is very bloated.


 How does the D standard library compare in size with some other 
 languages (C++, Rust, Go, Java, Python)?


C++ and Rust are the best from this list in terms of low level 
programming. They have smaller libraries comparing with D.

Go, Java, Python --- If I need a language like them I will you 
them, not D.

Go - awesome GC, commercially supported, easy to write your own 
libs
Java - awesome GC, commercially supported, Scala(!!!)
Python - awesome scripting, commercially supported

D - ??? awesome what ? What do we have better then Scala?

Performance? NO.
BRandomAccessRanges makes our performance less then C/C++.

Clean and clever API in standard library? NO.
We are placing our Range API whenever possible including RNGs. We 
have not a language standard for I/O. Half of Phobos is outdated 
to language standards.

Best generics? NO.
Scala is more powerful.

----------
What do we have awesome? Where D is the best?
----------
D is awesome for low-level system libraries in betterC mode.
(and small text processing routines)
This will make D great in other fields.
----------


 Let's do something about it!


 I've said this several times, and it bears repeating. We want 
 to make D entirely pay-as-you go, and offer full support to 
 anyone working on it. The best "something" anyone can do about 
 it is do good engineering work in the D mainline.


National D Foundation Party.


 That will automatically benefit all users of the D language, 
 instead of balkanizing the community.


...


 Granted, the lure of the clean slate is strong, and more care 
 is needed to mind backwards compatibility, but that's where the 
 real impact is.


 Andrei


How Mir projects breaks backward compatibility ?

How proposed direction breaks backward compatibility?

Lets use other prefix for the new libraries. Mir uses `mir.` And 
allow Phobos to be a sandbox for C++ as it is now.

Ilya
Dec 18 2016
next sibling parent yawniek <yawniek srtnwz.com> writes:
On Monday, 19 December 2016 at 05:59:07 UTC, Ilya Yaroshenko 
wrote:

 D - ??? awesome what ? What do we have better then Scala?

 Performance? NO.
 BRandomAccessRanges makes our performance less then C/C++.

 Clean and clever API in standard library? NO.
 We are placing our Range API whenever possible including RNGs. 
 We have not a language standard for I/O. Half of Phobos is 
 outdated to language standards.

 Best generics? NO.
 Scala is more powerful.

 ----------
 What do we have awesome? Where D is the best?
 ----------
 D is awesome for low-level system libraries in betterC mode.
 (and small text processing routines)
 This will make D great in other fields.


while i mostly use D as a awesome high level language (with 
mediocre libs, and an inconsistent stdlib) i also think that D 
has an edge when it comes to building fast system level tools 
that used to be written in c: microservices, datatabases etc.
however for that, as ilya mentioned it needs to be super user 
friendly to compile down a single static object/binary with no 
dependencies and a clear line on when phobos can/should/must be 
used and when not (e.g. create a view on phobos documentation 
that only shows things that work without druntime. add some 
official docs on how to use it)

people turn to C to extend their language of choice because they 
need speed in a very narrow area. if that becomes as easy (or 
even easier, since there is dub!) as with C it would be a huge 
win for D, because people would start to extend their stuff with 
D. No other language is in a such good position as D, from what i 
see not that much needs to be done to make that work.
i see a huge opportunity there
Dec 19 2016
prev sibling parent reply lurker <lurk lurking.org> writes:
On Monday, 19 December 2016 at 05:59:07 UTC, Ilya Yaroshenko 
wrote:

 On Monday, 19 December 2016 at 01:52:21 UTC, Andrei 
 Alexandrescu wrote:

 On 12/18/16 7:31 PM, Seb wrote:

 tl:dr: Phobos is very bloated.


 How does the D standard library compare in size with some 
 other languages (C++, Rust, Go, Java, Python)?


 C++ and Rust are the best from this list in terms of low level 
 programming. They have smaller libraries comparing with D.

 Go, Java, Python --- If I need a language like them I will you 
 them, not D.

 Go - awesome GC, commercially supported, easy to write your own 
 libs
 Java - awesome GC, commercially supported, Scala(!!!)
 Python - awesome scripting, commercially supported

 D - ??? awesome what ? What do we have better then Scala?

 Performance? NO.
 BRandomAccessRanges makes our performance less then C/C++.

 Clean and clever API in standard library? NO.
 We are placing our Range API whenever possible including RNGs. 
 We have not a language standard for I/O. Half of Phobos is 
 outdated to language standards.

 Best generics? NO.
 Scala is more powerful.

 ----------
 What do we have awesome? Where D is the best?
 ----------
 D is awesome for low-level system libraries in betterC mode.
 (and small text processing routines)
 This will make D great in other fields.
 ----------


 Let's do something about it!


 I've said this several times, and it bears repeating. We want 
 to make D entirely pay-as-you go, and offer full support to 
 anyone working on it. The best "something" anyone can do about 
 it is do good engineering work in the D mainline.


 National D Foundation Party.


 That will automatically benefit all users of the D language, 
 instead of balkanizing the community.


 ...


 Granted, the lure of the clean slate is strong, and more care 
 is needed to mind backwards compatibility, but that's where 
 the real impact is.


 Andrei


 How Mir projects breaks backward compatibility ?

 How proposed direction breaks backward compatibility?

 Lets use other prefix for the new libraries. Mir uses `mir.` 
 And allow Phobos to be a sandbox for C++ as it is now.

 Ilya


<delurk>
Thats why I asked for moderation in forums.
<lurk>
Dec 19 2016
parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 21:00:57 UTC, lurker wrote:

 On Monday, 19 December 2016 at 05:59:07 UTC, Ilya Yaroshenko 
 wrote:

 [...]


 <delurk>
 Thats why I asked for moderation in forums.
 <lurk>


+1, and ability to edit own posts.
Dec 19 2016
prev sibling next sibling parent reply Mike <none none.com> writes:
On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko 
wrote:


 Who is interested in betterC _modular_* standard library?


What exactly do you mean by "betterC"?  As far as I know, the 
`-betterC` switch just removes ModuleInfo generation.

Do you actually mean, a standard library that has fewer 
dependencies on druntime features?

Mike
Dec 18 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 01:33:42 UTC, Mike wrote:

 On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko 
 wrote:


 Who is interested in betterC _modular_* standard library?


 What exactly do you mean by "betterC"?  As far as I know, the 
 `-betterC` switch just removes ModuleInfo generation.


Because -betterC was not implemented yet.


 Do you actually mean, a standard library that has fewer 
 dependencies on druntime features?


I mean this too

Ilya
Dec 18 2016
parent Mike <none none.com> writes:
On Monday, 19 December 2016 at 05:22:10 UTC, Ilya Yaroshenko 
wrote:

 What exactly do you mean by "betterC"?  As far as I know, the 
 `-betterC` switch just removes ModuleInfo generation.


 Because -betterC was not implemented yet.


I'm curious to know what a complete implementation entails.


 Do you actually mean, a standard library that has fewer 
 dependencies on druntime features?


 I mean this too


I've been wanting to decouple from druntime since I started 
learning D in late 2013, though probably for different reasons 
than yours.  There have been two recent posts [1] [2] wanting to 
do the same thing, and now your post envisioning a more modular 
druntime-less phobos.

[1] - 
https://forum.dlang.org/post/drqdrobcoaanenocycgp forum.dlang.org
[2] - 
https://forum.dlang.org/post/tmofjecvnqdthvetezfp forum.dlang.org

It's encouraging to see this trend.  I am interested in seeing D 
become more modular and pay-as-you-go, though I'd prefer to see 
changes in the compiler and druntime (e.g. [3] [4] [5]) before 
phobos.

[3] - http://forum.dlang.org/post/mr71kt$2rgi$1 digitalmars.com
[4] - 
http://forum.dlang.org/post/eiwalbqlbkipdrmsrfoh forum.dlang.org
[5] - https://issues.dlang.org/show_bug.cgi?id=11666

Mike
Dec 18 2016
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko 
wrote:

 Hi,

 Who is interested in betterC _modular_* standard library?
 I am planing to make libmir org a community for it.
 Thought and concerns?

 We already have better `cpuid` and better `random` packages. 
 The betterC std.range and std.algorithm analogs would be 
 released with new ndslice implementation. Mir's algorithm would 
 be faster then Phobos and will generate less template bloat. 
 Then lightweight threads, multithread GLAS, matrix inversion. 
 Fastest I/O and http2 ...

 (they all are betterC modular libraries)

 Andrei and probably 90% of existing D users don't want Phobos 
 to move this direction.

 In other hand I need a commercial attractive D infrastructure 
 for large and heavy system projects. There is no commercial 
 perspective for me to contribute to Phobos because:


If I understand correctly, the only reaon not to disable the 
runtime instead of going betterC (what I call runtime-free vs 
runtime-less) is because Phobos and druntime aren't ported to 
every platform you need, but the libC is.

That it calling for the question: "Isn't porting Phobos and 
druntime to those new platforms your clients need less work?"

Maybe they only need ARM and x86-64, the effort should be minimal 
then? And with the runtime disabled, things like the GC just need 
to link, not actually work :)
Tha makes porting a bit simpler.

Things like std.algorithm, std.range, traits etc... took many 
man-years to do and can work as is.
core.cpuid can work with little modifications.

I'm not saying you can use a lot of Phobos and druntime without 
using the runtime, because currently you can't.
But think about it: when not linking the runtime, there isn't 
even a way to call "emplace" or "destroy", dynamic casts won't 
work, etc.

In the future more and more things will work without the runtime, 
exceptions, class creation and then you'll don't have to maintain 
a stdlib.

Please reconsider reconsidering.
Dec 19 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 19:31:24 UTC, Guillaume Piolat 
wrote:

 On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko 
 wrote:

 [...]


 If I understand correctly, the only reaon not to disable the 
 runtime instead of going betterC (what I call runtime-free vs 
 runtime-less) is because Phobos and druntime aren't ported to 
 every platform you need, but the libC is.


No,
1. DRuntime has not fixed API
2. It has not backward binary compatibility
3. It has not cross-compiler binary compatibility.

So, it can not be used.
Dec 19 2016
next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 19 December 2016 at 19:42:42 UTC, Ilya Yaroshenko 
wrote:

 No,
 1. DRuntime has not fixed API


Can't you static link with it? If you static link with it, your 
client don't deal with DRuntime.



 2. It has not backward binary compatibility


Ditto. If you static link with it, your client don't deal with 
binary compatibility.



 3. It has not cross-compiler binary compatibility.


Mandate that every D code in the project is built with the same 
compiler, which is realistic.
Dec 19 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 19:48:39 UTC, Guillaume Piolat 
wrote:

 On Monday, 19 December 2016 at 19:42:42 UTC, Ilya Yaroshenko 
 wrote:

 No,
 1. DRuntime has not fixed API


 Can't you static link with it? If you static link with it, your 
 client don't deal with DRuntime.



 2. It has not backward binary compatibility


 Ditto. If you static link with it, your client don't deal with 
 binary compatibility.



 3. It has not cross-compiler binary compatibility.


 Mandate that every D code in the project is built with the same 
 compiler, which is realistic.


How this can work for libraries? Please read this thread.
This is not realistic. This a huge constraint for D libraries to 
be modular, replaceable and distributed in binary form.
Dec 19 2016
next sibling parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 19:52:43 UTC, Ilya Yaroshenko 
wrote:

 On Monday, 19 December 2016 at 19:48:39 UTC, Guillaume Piolat 
 wrote:

 On Monday, 19 December 2016 at 19:42:42 UTC, Ilya Yaroshenko 
 wrote:

 No,
 1. DRuntime has not fixed API


 Can't you static link with it? If you static link with it, 
 your client don't deal with DRuntime.



 2. It has not backward binary compatibility


 Ditto. If you static link with it, your client don't deal with 
 binary compatibility.



 3. It has not cross-compiler binary compatibility.


 Mandate that every D code in the project is built with the 
 same compiler, which is realistic.


 How this can work for libraries? Please read this thread.
 This is not realistic. This a huge constraint for D libraries 
 to be modular, replaceable and distributed in binary form.


To be more precise: I do not build library exclusively for D. 
They should be easily interacted with other languages.
Dec 19 2016
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 19 December 2016 at 19:54:54 UTC, Ilya Yaroshenko 
wrote:

 To be more precise: I do not build library exclusively for D. 
 They should be easily interacted with other languages.


Non sequitur, you can enable the runtime/attah threads withing 
callbacks.
Dec 19 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 19:59:01 UTC, Guillaume Piolat 
wrote:

 On Monday, 19 December 2016 at 19:54:54 UTC, Ilya Yaroshenko 
 wrote:

 To be more precise: I do not build library exclusively for D. 
 They should be easily interacted with other languages.


 Non sequitur, you can enable the runtime/attah threads withing 
 callbacks.


No, DRuntime based numerical libraries will never replace C 
analogs.
Dec 19 2016
parent Guillaume Piolat <first.last gmail.com> writes:
On Monday, 19 December 2016 at 20:10:02 UTC, Ilya Yaroshenko 
wrote:

 On Monday, 19 December 2016 at 19:59:01 UTC, Guillaume Piolat 
 wrote:

 On Monday, 19 December 2016 at 19:54:54 UTC, Ilya Yaroshenko 
 wrote:

 To be more precise: I do not build library exclusively for D. 
 They should be easily interacted with other languages.


 Non sequitur, you can enable the runtime/attah threads withing 
 callbacks.


 No, DRuntime based numerical libraries will never replace C 
 analogs.


Going to bed. Please don't take it as attacks, I'm juste trying 
to help find the best possible solution, with the least amount of 
pain.
Dec 19 2016
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 19 December 2016 at 19:52:43 UTC, Ilya Yaroshenko 
wrote:

 How this can work for libraries? Please read this thread.
 This is not realistic. This a huge constraint for D libraries 
 to be modular, replaceable and distributed in binary form.


They can be at the MIR level.

How this can work: Mir libraries are distributed as static and 
dynamic libs, whatever.
Each one of these links statically with the same druntime and 
libphobos. When the user replace one MIR library it also replaces 
the others.

I don't see what isn't realistic here.

See also: CUDA, Intel Performance Primitives...
People update CUDA in bunch, they don't use cuDNN from release 
6.15 with cuRuntime from release 7.8
Dec 19 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 19:57:45 UTC, Guillaume Piolat 
wrote:

 On Monday, 19 December 2016 at 19:52:43 UTC, Ilya Yaroshenko 
 wrote:

 How this can work for libraries? Please read this thread.
 This is not realistic. This a huge constraint for D libraries 
 to be modular, replaceable and distributed in binary form.


 They can be at the MIR level.

 How this can work: Mir libraries are distributed as static and 
 dynamic libs, whatever.
 Each one of these links statically with the same druntime and 
 libphobos. When the user replace one MIR library it also 
 replaces the others.

 I don't see what isn't realistic here.

 See also: CUDA, Intel Performance Primitives...
 People update CUDA in bunch, they don't use cuDNN from release 
 6.15 with cuRuntime from release 7.8


Mir CPUID was created for Mir GLAS, which is BLAS implementation.

How Mir GLAS be a default BLAS implementation in linux 
distributives if it is depends on DRuntime?

What about other non Mir numerical library written in Dlang, say 
for FFT?

How they should be linked together.

This example is already in this thread multiple times.

Do you propose to constraint Dlang library distribution in binary 
form?

I do not write D libraries for Dlang. I wrote them for people, 
which uses different languages. Most of them do not know about D.
Dec 19 2016
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 19 December 2016 at 20:05:20 UTC, Ilya Yaroshenko 
wrote:

 What about other non Mir numerical library written in Dlang, 
 say for FFT?

 How they should be linked together.


If Mir libs are static libs this can indeed be a problem.
If someone has made another D library like this, it will either 
have to be its own dynlib or come with a runtime with the same 
API. You can manage to use very little of druntime such as 
"emplace"/"destroy" and that's about it when it comes to API.


I'll concede that my solution will not get you packaged on 
official package lists for Debian/Ubuntu, because shared 
libraries will get preferred there.

But I just checked and libcublas is just put in the non-free 
packages.
https://packages.debian.org/jessie/libcublas6.0
If it's good enough for CUDA then maybe it's good enough for MIR?

Note that with MIR being several dynlibs everyone of Mir library 
can come with any runtime (and then use it or not) without 
interaction of these runtimes.

When you use a CUDA dynamic library you don't know if it has a C 
runtime inside (or a X runtime, whatever).



 Do you propose to constraint Dlang library distribution in 
 binary form?


No. Not sure how that question ensues from the premises.
Dec 19 2016
parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 20:24:21 UTC, Guillaume Piolat 
wrote:

 On Monday, 19 December 2016 at 20:05:20 UTC, Ilya Yaroshenko 
 wrote:

 [...]


 If Mir libs are static libs this can indeed be a problem.
 If someone has made another D library like this, it will either 
 have to be its own dynlib or come with a runtime with the same 
 API. You can manage to use very little of druntime such as 
 "emplace"/"destroy" and that's about it when it comes to API.

 [...]


No, Mir is not a hardware manufacturer which forces its users to 
do something.


 Note that with MIR being several dynlibs everyone of Mir 
 library can come with any runtime (and then use it or not) 
 without interaction of these runtimes.

 [...]


Looks like bad design. Existing C libs would be better then.
Dec 19 2016
prev sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 19 December 2016 at 20:42, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 19 December 2016 at 19:31:24 UTC, Guillaume Piolat wrote:

 On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko wrote:

 [...]



 If I understand correctly, the only reaon not to disable the runtime
 instead of going betterC (what I call runtime-free vs runtime-less) is
 because Phobos and druntime aren't ported to every platform you need, but
 the libC is.



 No,
 1. DRuntime has not fixed API
 2. It has not backward binary compatibility
 3. It has not cross-compiler binary compatibility.

 So, it can not be used.


You're confusing druntime with phobos.  I haven't changed any bindings
to druntime in years...
Dec 19 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 19 December 2016 at 20:22:29 UTC, Iain Buclaw wrote:

 On 19 December 2016 at 20:42, Ilya Yaroshenko via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 On Monday, 19 December 2016 at 19:31:24 UTC, Guillaume Piolat 
 wrote:

 On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko 
 wrote:

 [...]



 If I understand correctly, the only reaon not to disable the 
 runtime instead of going betterC (what I call runtime-free vs 
 runtime-less) is because Phobos and druntime aren't ported to 
 every platform you need, but the libC is.



 No,
 1. DRuntime has not fixed API
 2. It has not backward binary compatibility
 3. It has not cross-compiler binary compatibility.

 So, it can not be used.


 You're confusing druntime with phobos.  I haven't changed any 
 bindings to druntime in years...


If someone build a library with GCC and another one with Clang 
they will work together. What about LDC and GDC?
Dec 19 2016
parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 19 December 2016 at 21:29, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 19 December 2016 at 20:22:29 UTC, Iain Buclaw wrote:

 On 19 December 2016 at 20:42, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Monday, 19 December 2016 at 19:31:24 UTC, Guillaume Piolat wrote:

 On Sunday, 18 December 2016 at 09:26:09 UTC, Ilya Yaroshenko wrote:

 [...]




 If I understand correctly, the only reaon not to disable the runtime
 instead of going betterC (what I call runtime-free vs runtime-less) is
 because Phobos and druntime aren't ported to every platform you need, but
 the libC is.




 No,
 1. DRuntime has not fixed API
 2. It has not backward binary compatibility
 3. It has not cross-compiler binary compatibility.

 So, it can not be used.



 You're confusing druntime with phobos.  I haven't changed any bindings to
 druntime in years...



 If someone build a library with GCC and another one with Clang they will
 work together. What about LDC and GDC?


Clang developers decided that they should be ABI compatible with GCC.
I'm not sure if the LDC developers are doing the same thing. :-)
Dec 19 2016
prev sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: A betterC modular standard library?
References: <phexetutyelrssyrucvw forum.dlang.org>
In-Reply-To: <phexetutyelrssyrucvw forum.dlang.org>

--qhnfOq2CEv5mloVi7rH0OwT2qiGQnsjT4
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 12/18/2016 11:26 AM, Ilya Yaroshenko wrote:

 Hi,
=20
 Who is interested in betterC _modular_* standard library?
 I am planing to make libmir org a community for it.
 Thought and concerns?


I also consider Phobos a lost cause only suitable for scripting
purposes. However what you try to do won't be acceptable for me either
(see below).


 We already have better `cpuid` and better `random` packages. The better=


C

 std.range and std.algorithm analogs would be released with new ndslice
 implementation. Mir's algorithm would be faster then Phobos and will
 generate less template bloat. Then lightweight threads, multithread
 GLAS, matrix inversion. Fastest I/O and http2 ...


I consider only algorithms and interfaces suitable for standard library.
GLAS, I/O, HTTP - this is exactly all the stuff I want to see removed
into separate dub packages and maintained as such.


 (they all are betterC modular libraries)


=2E. and this "betterC" weirdness you use with your own meaning is
definitely something I don't want to _ever_ see as standard library
approach. It is similar to `malloc` vs `GC` debate for Phobos, because
correct answer tends to be "neither", with rationale that any library
utility that has to care about the difference is hardly generic enough
for unconditional distribution.

------------------------

My own dream path for fixing Phobos may look like this:

1) Start with separating environment-agnostic core (algorithms and
API's) from utilities
2) Rewrite all higher level utilities in a way that they don't have any
inter-dependencies and only allowed to use algorithm/API modules
3) Move higher level stuff into own dub packages with a special "Phobos"
category
4) When compiler is released, provide snapshot of latest up to date
packages in that category as part of the distribution for those who
don't want to bother with fetching manually

This actually may be even possible to do within a linear deprecation
process. On the other hand I doubt such drastic change to structure is
possible to be sold to community.


--qhnfOq2CEv5mloVi7rH0OwT2qiGQnsjT4--
Dec 20 2016
parent reply Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Tuesday, 20 December 2016 at 08:15:35 UTC, Dicebot wrote:

 On 12/18/2016 11:26 AM, Ilya Yaroshenko wrote:>

 This actually may be even possible to do within a linear 
 deprecation process. On the other hand I doubt such drastic 
 change to structure is possible to be sold to community.


Consider that plan sold to me...

---
Paolo
Dec 20 2016
parent reply Timothee Cour via Digitalmars-d <digitalmars-d puremagic.com> writes:
 ilya: regarding -betterC 's stated goal to bypass druntime:

besides static linking (which can be inefficient), why not use runtime
shared libraries to distribute mir (eg GLAS) to C clients ?

IIRC, from a single C++ program, I am able to dlopen & dlsym & run D
functions defined in shared libraries built with different compilers (eg
dmd and ldc) even if they use GC (and therefore dmd's and ldc's druntime) ;
pseudocode:

// main.cpp
```
void main(){
  dlopen, dlsym, and run "void fun1()" from libfun_dmd.so
  dlopen, dlsym, and run "void fun2()" from libfun_ldc.so
}
```

That would provide the needed isolation without suffering from
disadvantages of static libraries (eg bloat); compared to the betterC
alternative, you'd have access to all of druntime/phobos and not have to
reinvent the wheel


On Tue, Dec 20, 2016 at 1:06 AM, Paolo Invernizzi via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Tuesday, 20 December 2016 at 08:15:35 UTC, Dicebot wrote:


 On 12/18/2016 11:26 AM, Ilya Yaroshenko wrote:>

 This actually may be even possible to do within a linear deprecation
 process. On the other hand I doubt such drastic change to structure is
 possible to be sold to community.


 Consider that plan sold to me...

 ---
 Paolo
Dec 20 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Tuesday, 20 December 2016 at 09:12:55 UTC, Timothee Cour wrote:

  ilya: regarding -betterC 's stated goal to bypass druntime:

 besides static linking (which can be inefficient), why not use 
 runtime
 shared libraries to distribute mir (eg GLAS) to C clients ?

 IIRC, from a single C++ program, I am able to dlopen & dlsym & 
 run D functions defined in shared libraries built with 
 different compilers (eg dmd and ldc) even if they use GC (and 
 therefore dmd's and ldc's druntime) ; pseudocode:

 // main.cpp
 ```
 void main(){
   dlopen, dlsym, and run "void fun1()" from libfun_dmd.so
   dlopen, dlsym, and run "void fun2()" from libfun_ldc.so
 }
 ```

 That would provide the needed isolation without suffering from 
 disadvantages of static libraries (eg bloat); compared to the 
 betterC alternative, you'd have access to all of 
 druntime/phobos and not have to reinvent the wheel


Because I do not have control on phobos, i can not distribute my 
copy of phobos with required patches like normal cache size 
estimation (core.cpuid is broken). Why not to fix core.cpuid?  
Its source code is not manageable comparing with mir-cpuid: it is 
faster to write a module from scratch for this case. DUB package 
appears when I want it, LDC release may appear after 6-10 months 
(depends on distribution).
Dec 20 2016
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 20/12/2016 10:32 PM, Ilya Yaroshenko wrote:

 On Tuesday, 20 December 2016 at 09:12:55 UTC, Timothee Cour wrote:

  ilya: regarding -betterC 's stated goal to bypass druntime:

 besides static linking (which can be inefficient), why not use runtime
 shared libraries to distribute mir (eg GLAS) to C clients ?

 IIRC, from a single C++ program, I am able to dlopen & dlsym & run D
 functions defined in shared libraries built with different compilers
 (eg dmd and ldc) even if they use GC (and therefore dmd's and ldc's
 druntime) ; pseudocode:

 // main.cpp
 ```
 void main(){
   dlopen, dlsym, and run "void fun1()" from libfun_dmd.so
   dlopen, dlsym, and run "void fun2()" from libfun_ldc.so
 }
 ```

 That would provide the needed isolation without suffering from
 disadvantages of static libraries (eg bloat); compared to the betterC
 alternative, you'd have access to all of druntime/phobos and not have
 to reinvent the wheel


 Because I do not have control on phobos, i can not distribute my copy of
 phobos with required patches like normal cache size estimation
 (core.cpuid is broken). Why not to fix core.cpuid?  Its source code is
 not manageable comparing with mir-cpuid: it is faster to write a module
 from scratch for this case. DUB package appears when I want it, LDC
 release may appear after 6-10 months (depends on distribution).


Of course you can bundle your own Phobos.
If you put a std.* module in your path your build will be working 
against that instead. If you don't mess around with sc.ini it will 
override the default for that compilation.
Dec 20 2016
parent reply Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Tuesday, 20 December 2016 at 09:38:58 UTC, rikki cattermole 
wrote:

 On 20/12/2016 10:32 PM, Ilya Yaroshenko wrote:

 On Tuesday, 20 December 2016 at 09:12:55 UTC, Timothee Cour 
 wrote:

  ilya: regarding -betterC 's stated goal to bypass druntime:

 besides static linking (which can be inefficient), why not 
 use runtime
 shared libraries to distribute mir (eg GLAS) to C clients ?

 IIRC, from a single C++ program, I am able to dlopen & dlsym 
 & run D
 functions defined in shared libraries built with different 
 compilers
 (eg dmd and ldc) even if they use GC (and therefore dmd's and 
 ldc's
 druntime) ; pseudocode:

 // main.cpp
 ```
 void main(){
   dlopen, dlsym, and run "void fun1()" from libfun_dmd.so
   dlopen, dlsym, and run "void fun2()" from libfun_ldc.so
 }
 ```

 That would provide the needed isolation without suffering from
 disadvantages of static libraries (eg bloat); compared to the 
 betterC
 alternative, you'd have access to all of druntime/phobos and 
 not have
 to reinvent the wheel


 Because I do not have control on phobos, i can not distribute 
 my copy of
 phobos with required patches like normal cache size estimation
 (core.cpuid is broken). Why not to fix core.cpuid?  Its source 
 code is
 not manageable comparing with mir-cpuid: it is faster to write 
 a module
 from scratch for this case. DUB package appears when I want 
 it, LDC
 release may appear after 6-10 months (depends on distribution).


 Of course you can bundle your own Phobos.
 If you put a std.* module in your path your build will be 
 working against that instead. If you don't mess around with 
 sc.ini it will override the default for that compilation.


CPUID should be precompiled. So this does not work. If GLAS do 
not need anything but CPUID, what the reason to depend on 
DRuntime?

Please stop to force me to use DRuntime because you think it is 
better for Mir projects. This looks like DRuntime religion.

A betterC library can be used with DRuntime. So I do not force 
anyone to drop DRuntime. Please do not force me to use it.

GLAS already works without DRuntime and I am happy about that.
Dec 20 2016
parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 20 December 2016 at 09:56:34 UTC, Ilya Yaroshenko 
wrote:

 Of course you can bundle your own Phobos.
 If you put a std.* module in your path your build will be 
 working against that instead. If you don't mess around with 
 sc.ini it will override the default for that compilation.


 CPUID should be precompiled. So this does not work. If GLAS do 
 not need anything but CPUID, what the reason to depend on 
 DRuntime?

 Please stop to force me to use DRuntime because you think it is 
 better for Mir projects. This looks like DRuntime religion.

 A betterC library can be used with DRuntime. So I do not force 
 anyone to drop DRuntime. Please do not force me to use it.

 GLAS already works without DRuntime and I am happy about that.


What does "you can bundle your own phobos" have to do with that? 
No one's forcing you to do anything.

You're suggesting something radical*, other people are suggesting 
that maybe there are good compromises where everybody wins, 
avoiding fragmentation. Then you say that "looks like DRuntime 
religion" and claim you're being forced to modify your code to 
add extra dependencies. It doesn't make sense.

Your technical arguments have good content, in my opinion 
everyone would benefit from you writing them up with sufficient 
context for people who don't know what you know and without 
hyperbole (none of that "D will fail if we don't do X" or "phobos 
is bloated and useless" stuff, it doesn't help communicate your 
points). Developing the case study / thought experiment of 
getting two libraries (blas and fft) in to a traditional linux 
distro would be a great central point.

See https://github.com/dlang/DIPs/pull/51 where Andrei describes 
not only how he thinks things should be, but also explains in 
detail why other approaches won't work. Now consider that what 
you're talking about is a lot more important and 
different/disruptive than a new import syntax, so it deserves at 
least comparably good description.

* D used to have 2 standard libraries. That was not a happy time 
for the community. People are wary partly because of that.

P.S. a good example of the sort of statement that needs more 
explanation, at least for me:

 The betterC
 std.range and std.algorithm analogs would be released with new 
 ndslice
 implementation. Mir's algorithm would be faster then Phobos and 
 will
 generate less template bloat."


How? Why? Is ndslice actually capable of the same level of 
flexibility as std.algorithm and std.range? You're effectively 
saying "all that stuff that Andrei (and others) designed and 
wrote, I can do better" without really showing anyone why they 
should believe that.
Dec 20 2016
next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 20 December 2016 at 12:33:27 UTC, John Colvin wrote:

 * D used to have 2 standard libraries. That was not a happy 
 time for the community. People are wary partly because of that.


To bring some cool to the discussion, let's remember that they 
also weren't compatible.
Of course it's a concern but this case is different and deserve 
full consideration. let's say the thread title was "A 
runtime-less modular library?"
Dec 20 2016
parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Tuesday, 20 December 2016 at 12:45:13 UTC, Guillaume Piolat 
wrote:

 On Tuesday, 20 December 2016 at 12:33:27 UTC, John Colvin wrote:

 * D used to have 2 standard libraries. That was not a happy 
 time for the community. People are wary partly because of that.


 To bring some cool to the discussion, let's remember that they 
 also weren't compatible.
 Of course it's a concern but this case is different and deserve 
 full consideration. let's say the thread title was "A 
 runtime-less modular library?"


Thanks! Good point
Dec 20 2016
prev sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Tuesday, 20 December 2016 at 12:33:27 UTC, John Colvin wrote:

Mir uses deprecated native complex numbers because we can not 
spend days explaining why they are better then std.complex (in 
short: std.complex breaks compiler optimization logic, function 
inlining works, but it breaks optimization logic). I have already 
spend 4 hours for forum discussions and PRs about this issue 
without any result.

We are doing the best things we can do with current D. If 
something will be improved in future we will upgrade codebase.


 P.S. a good example of the sort of statement that needs more 
 explanation, at least for me:

 The betterC
 std.range and std.algorithm analogs would be released with new 
 ndslice
 implementation. Mir's algorithm would be faster then Phobos 
 and will
 generate less template bloat."


 How? Why? Is ndslice actually capable of the same level of 
 flexibility as std.algorithm and std.range? You're effectively 
 saying "all that stuff that Andrei (and others) designed and 
 wrote, I can do better" without really showing anyone why they 
 should believe that.


The explanation of new idioms will appear after its code.

I did not say "better". I said "betterC" plus "faster".

Yes, i can say that for DCV library new idioms are much better 
then Phobos.

But new idioms have another issues, for example D language is not 
capable to infer their safety for many of them. Other issues is 
that there a lot of fast non CTFE-able types and their CTFE-able 
analogs.

One good things for safety and CTFE is allow multiple return 
value. In combination with `auto ref` it is _very_ powerful:

----
auto ref front()
{
   // Returns 2 values, each value is returned by reference if 
possible
   return (a.front, b.front);
}
----

This would be interesting to discuss and to have it in D in CTFE 
and  safe mode. But I am not good in DIPs.

This sorts of features is very desirable.
Dec 20 2016