digitalmars.D - Sharding Phobos an alternative to sharding druntime
- Richard (Rikki) Andrew Cattermole (26/26) Jun 21 We've had a meeting to discuss splitting up druntime a bit,
- Alexandru Ermicioi (19/22) Jun 22 Those shards might also be published on dub. This could in
- Paolo Invernizzi (5/34) Jun 22 Independent ... but with dependency?
- Alexandru Ermicioi (2/5) Jun 22 Perhaps it is implied optional dependencies?
- Richard (Rikki) Andrew Cattermole (7/10) Jun 22 No no.
- Paolo Invernizzi (14/24) Jun 23 I still think it's over-eng.
- Richard (Rikki) Andrew Cattermole (8/33) Jun 23 What new language concept?
- Paolo Invernizzi (12/47) Jun 23 Language intended as language as a whole, ecosystem included. I
- Richard (Rikki) Andrew Cattermole (10/64) Jun 23 If you want to grow the standard library you have no choice but to split...
- Paolo Invernizzi (19/87) Jun 23 Thank you, now the intent is more clear to me.
- monkyyy (4/11) Jun 23 is it hard? I'd try just "tempateify" everything
- Quirin Schroll (5/10) Jul 02 Why “instead”? If I understood correctly, sharding DRuntime would
- Richard (Rikki) Andrew Cattermole (6/17) Jul 02 Sharding druntime is about giving a place to call home for things like
We've had a meeting to discuss splitting up druntime a bit, although we got slightly off track during it. As part of it, it was suggested by Steven that instead of sharding druntime, we should shard phobos, so here is my proposal. I proposed this many many months ago, right back at the start of PhobosV3 work that we "shard" Phobos, of course back then I didn't call it as such. The reason we did not adopt this earlier is because of some insistence from Walter for PhobosV3 must be import only, which I unsurprisingly have been against this entire time; although he has now accepted that this will not necessarily be possible. The idea is simple: - You have a root package, this has been decided to be ``phobos``. - A sub package below that ``phobos.XYZ`` would be a "shard". - A shard may be mapped to a binary, or it could be import only. - Each shard would be considered independent of the others, however it may declare a dependency on another. - No module can exist outside of a shard. As a proposal it is simple to map a module to its binary. It scales as PhobosV3 grows, with each independent shard allowed to do its own thing that it requires with clear dependencies. We do not need to worry about things like event loops design work right now, but it does allow us to go: its a dependency of sockets therefore we can't do it right now. This too was a concern of Walter's that Adam Wilson was taking on too much.
Jun 21
Alexandru Ermicioi <alexandru.ermicioi gmail.com> On Friday, 21 June 2024 at 16:39:43 UTC, Richard (Rikki) Andrew Cattermole wrote:I proposed this many many months ago, right back at the start of PhobosV3 work that we "shard" Phobos, of course back then I didn't call it as such.Those shards might also be published on dub. This could in principle allow patches and improvements to shards without waiting for D release. Each shard could have it's own versioning, and releases, so people needing a change won't need to wait much. There might also be a meta package that imports other shards, and is locked with D release versions. This meta package would sort of work like bom projects in maven, i.e. it's dependencies would specify versions of shards that are guaranteed to work with specific D version. Ofc, a reference phobos version should be shipped with D installation. The build system for D installers could then just pull a version of phobos meta package and include it as part of D installers. Note: it would be nice if in dub you could specify the language version the library was written in. This defined on shards would also allow easier phobos meta package build.
Jun 22
On Friday, 21 June 2024 at 16:39:43 UTC, Richard (Rikki) Andrew Cattermole wrote:We've had a meeting to discuss splitting up druntime a bit, although we got slightly off track during it. As part of it, it was suggested by Steven that instead of sharding druntime, we should shard phobos, so here is my proposal. I proposed this many many months ago, right back at the start of PhobosV3 work that we "shard" Phobos, of course back then I didn't call it as such. The reason we did not adopt this earlier is because of some insistence from Walter for PhobosV3 must be import only, which I unsurprisingly have been against this entire time; although he has now accepted that this will not necessarily be possible. The idea is simple: - You have a root package, this has been decided to be ``phobos``. - A sub package below that ``phobos.XYZ`` would be a "shard". - A shard may be mapped to a binary, or it could be import only. - Each shard would be considered independent of the others, however it may declare a dependency on another. - No module can exist outside of a shard. As a proposal it is simple to map a module to its binary. It scales as PhobosV3 grows, with each independent shard allowed to do its own thing that it requires with clear dependencies. We do not need to worry about things like event loops design work right now, but it does allow us to go: its a dependency of sockets therefore we can't do it right now. This too was a concern of Walter's that Adam Wilson was taking on too much.Independent ... but with dependency? The impression is that everything in D-lanD is moving towards an over-eng complexity ...
Jun 22
On Saturday, 22 June 2024 at 08:37:40 UTC, Paolo Invernizzi wrote:Independent ... but with dependency? The impression is that everything in D-lanD is moving towards an over-eng complexity ...Perhaps it is implied optional dependencies?
Jun 22
On 22/06/2024 8:37 PM, Paolo Invernizzi wrote:Independent ... but with dependency? The impression is that everything in D-lanD is moving towards an over-eng complexity ...No no. Its literally just the event loop shard depends upon base shard. Explicit dependencies, no accidental importing something that you shouldn't be able to without clear stated intention. This is a key goal of PhobosV3 work, was to break the interdependencies of modules.
Jun 22
On Saturday, 22 June 2024 at 10:28:10 UTC, Richard (Rikki) Andrew Cattermole wrote:On 22/06/2024 8:37 PM, Paolo Invernizzi wrote:I still think it's over-eng. Indirect dependencies of module are direct dependencies of the imported module, that means that the opinion is that the imported module is using another module without a valid good reason. So, the only thing to do is "just" (well, "just" ...) improve the quality of imported module itself, cleaning it. Why is it necessary to have another abstraction, shard, in a already complicated language like D? Another thing is the policy to "forbid the usage of a module" in a project. For example, we forbid to import directly "std.stdio", but the task is handled by our custom builder, that just enforce that.Independent ... but with dependency? The impression is that everything in D-lanD is moving towards an over-eng complexity ...No no. Its literally just the event loop shard depends upon base shard. Explicit dependencies, no accidental importing something that you shouldn't be able to without clear stated intention. This is a key goal of PhobosV3 work, was to break the interdependencies of modules.
Jun 23
On 24/06/2024 3:59 AM, Paolo Invernizzi wrote:On Saturday, 22 June 2024 at 10:28:10 UTC, Richard (Rikki) Andrew Cattermole wrote:What new language concept? Its purely build system, what files get passed to the compiler and made available via ``-I``. Everything is in place except the build system (and enabling of PAYG for linking). It is no different than a dub sub package and explicitly setting the dependencies there.On 22/06/2024 8:37 PM, Paolo Invernizzi wrote:I still think it's over-eng. Indirect dependencies of module are direct dependencies of the imported module, that means that the opinion is that the imported module is using another module without a valid good reason. So, the only thing to do is "just" (well, "just" ...) improve the quality of imported module itself, cleaning it. Why is it necessary to have another abstraction, shard, in a already complicated language like D?Independent ... but with dependency? The impression is that everything in D-lanD is moving towards an over-eng complexity ...No no. Its literally just the event loop shard depends upon base shard. Explicit dependencies, no accidental importing something that you shouldn't be able to without clear stated intention. This is a key goal of PhobosV3 work, was to break the interdependencies of modules.
Jun 23
On Sunday, 23 June 2024 at 16:09:00 UTC, Richard (Rikki) Andrew Cattermole wrote:On 24/06/2024 3:59 AM, Paolo Invernizzi wrote:Language intended as language as a whole, ecosystem included. I still don't grasp where's the improvement over the status quo of a monolithic Phobos and far west dub modules. Phobos modules interdependencies should be reduced? It's a decade that this point is on the table, I was here when Andrei rants about Phobos quality were flying around. Nothing new introduced into the "language as a whole" supersede the simple point that hard analysis work needs to be done on the code itself. I still do not grasp the point. /POn Saturday, 22 June 2024 at 10:28:10 UTC, Richard (Rikki) Andrew Cattermole wrote:What new language concept? Its purely build system, what files get passed to the compiler and made available via ``-I``. Everything is in place except the build system (and enabling of PAYG for linking). It is no different than a dub sub package and explicitly setting the dependencies there.On 22/06/2024 8:37 PM, Paolo Invernizzi wrote:I still think it's over-eng. Indirect dependencies of module are direct dependencies of the imported module, that means that the opinion is that the imported module is using another module without a valid good reason. So, the only thing to do is "just" (well, "just" ...) improve the quality of imported module itself, cleaning it. Why is it necessary to have another abstraction, shard, in a already complicated language like D?Independent ... but with dependency? The impression is that everything in D-lanD is moving towards an over-eng complexity ...No no. Its literally just the event loop shard depends upon base shard. Explicit dependencies, no accidental importing something that you shouldn't be able to without clear stated intention. This is a key goal of PhobosV3 work, was to break the interdependencies of modules.
Jun 23
On 24/06/2024 4:24 AM, Paolo Invernizzi wrote:On Sunday, 23 June 2024 at 16:09:00 UTC, Richard (Rikki) Andrew Cattermole wrote:If you want to grow the standard library you have no choice but to split into different binaries. That 64k symbol limit is quite the limiter here. Having stuff like eventloops or even just curl which are heavily tied to platform specifics does prevent porting even the subset that doesn't depend on anything on the platform. In practice nobody ports phobos but they do copy stuff out of it. At some point a split will occur, a well defined split is better than "we can't release dmd anymore as it won't link".On 24/06/2024 3:59 AM, Paolo Invernizzi wrote:Language intended as language as a whole, ecosystem included. I still don't grasp where's the improvement over the status quo of a monolithic Phobos and far west dub modules. Phobos modules interdependencies should be reduced? It's a decade that this point is on the table, I was here when Andrei rants about Phobos quality were flying around. Nothing new introduced into the "language as a whole" supersede the simple point that hard analysis work needs to be done on the code itself. I still do not grasp the point. /POn Saturday, 22 June 2024 at 10:28:10 UTC, Richard (Rikki) Andrew Cattermole wrote:What new language concept? Its purely build system, what files get passed to the compiler and made available via ``-I``. Everything is in place except the build system (and enabling of PAYG for linking). It is no different than a dub sub package and explicitly setting the dependencies there.On 22/06/2024 8:37 PM, Paolo Invernizzi wrote:I still think it's over-eng. Indirect dependencies of module are direct dependencies of the imported module, that means that the opinion is that the imported module is using another module without a valid good reason. So, the only thing to do is "just" (well, "just" ...) improve the quality of imported module itself, cleaning it. Why is it necessary to have another abstraction, shard, in a already complicated language like D?Independent ... but with dependency? The impression is that everything in D-lanD is moving towards an over-eng complexity ...No no. Its literally just the event loop shard depends upon base shard. Explicit dependencies, no accidental importing something that you shouldn't be able to without clear stated intention. This is a key goal of PhobosV3 work, was to break the interdependencies of modules.
Jun 23
On Sunday, 23 June 2024 at 16:41:37 UTC, Richard (Rikki) Andrew Cattermole wrote:On 24/06/2024 4:24 AM, Paolo Invernizzi wrote:Thank you, now the intent is more clear to me. As a preamble, curl into Phobos is an abomination, dub packages are the right place for stuff like that: if more space for symbols is needed, why not move them out, like std.streams and so? Basic building blocks have the tendency to be templates: more sumtype like inclusion, less std.logger inclusion. More import usage, less library usage. Phobos should be a place where interfaces are defined, for example how many implementations of "socket" are around, sync and async? Vibe? Phobos? Martin std.io? Etienne? arsd? Everyone is free to provide it's own implementation of event loop on as dub package, but why not a common "common" "basic" hygienic definition that allows interoperability? Anyway, I'm digressing too much ... but at the end, please, do not over-complicate the D ecosystem more that the current status quo. We are already at the boundaries of stepping out in using D (well, not that this is so important to anyone but us ...)On Sunday, 23 June 2024 at 16:09:00 UTC, Richard (Rikki) Andrew Cattermole wrote:If you want to grow the standard library you have no choice but to split into different binaries. That 64k symbol limit is quite the limiter here. Having stuff like eventloops or even just curl which are heavily tied to platform specifics does prevent porting even the subset that doesn't depend on anything on the platform. In practice nobody ports phobos but they do copy stuff out of it. At some point a split will occur, a well defined split is better than "we can't release dmd anymore as it won't link".On 24/06/2024 3:59 AM, Paolo Invernizzi wrote:Language intended as language as a whole, ecosystem included. I still don't grasp where's the improvement over the status quo of a monolithic Phobos and far west dub modules. Phobos modules interdependencies should be reduced? It's a decade that this point is on the table, I was here when Andrei rants about Phobos quality were flying around. Nothing new introduced into the "language as a whole" supersede the simple point that hard analysis work needs to be done on the code itself. I still do not grasp the point. /POn Saturday, 22 June 2024 at 10:28:10 UTC, Richard (Rikki) Andrew Cattermole wrote:What new language concept? Its purely build system, what files get passed to the compiler and made available via ``-I``. Everything is in place except the build system (and enabling of PAYG for linking). It is no different than a dub sub package and explicitly setting the dependencies there.On 22/06/2024 8:37 PM, Paolo Invernizzi wrote:I still think it's over-eng. Indirect dependencies of module are direct dependencies of the imported module, that means that the opinion is that the imported module is using another module without a valid good reason. So, the only thing to do is "just" (well, "just" ...) improve the quality of imported module itself, cleaning it. Why is it necessary to have another abstraction, shard, in a already complicated language like D?Independent ... but with dependency? The impression is that everything in D-lanD is moving towards an over-eng complexity ...No no. Its literally just the event loop shard depends upon base shard. Explicit dependencies, no accidental importing something that you shouldn't be able to without clear stated intention. This is a key goal of PhobosV3 work, was to break the interdependencies of modules.
Jun 23
On Sunday, 23 June 2024 at 16:24:23 UTC, Paolo Invernizzi wrote:Phobos modules interdependencies should be reduced? It's a decade that this point is on the table, I was here when Andrei rants about Phobos quality were flying around. Nothing new introduced into the "language as a whole" supersede the simple point that hard analysis work needs to be done on the code itself.is it hard? I'd try just "tempateify" everything I think you could write a script that trasformed `alias foo=`=> `alias foo()=` and `[type] foo(*){`=> `[type] foo()(*){`
Jun 23
On Friday, 21 June 2024 at 16:39:43 UTC, Richard (Rikki) Andrew Cattermole wrote:We've had a meeting to discuss splitting up druntime a bit, although we got slightly off track during it. As part of it, it was suggested by Steven that instead of sharding druntime, we should shard phobos, so here is my proposal.Why “instead”? If I understood correctly, sharding DRuntime would be good, so why not both? Sure, it’s more work, but it should also provide value, right?
Jul 02
On 03/07/2024 4:49 AM, Quirin Schroll wrote:On Friday, 21 June 2024 at 16:39:43 UTC, Richard (Rikki) Andrew Cattermole wrote:Sharding druntime is about giving a place to call home for things like an event loop or curl as a work around to having phobos be import only. It is not really for splitting the compiler hooks out, while it does come into it, its not the same bit of design work. Hence why I used "instead" as it accurately reflects how we got here.We've had a meeting to discuss splitting up druntime a bit, although we got slightly off track during it. As part of it, it was suggested by Steven that instead of sharding druntime, we should shard phobos, so here is my proposal.Why “instead”? If I understood correctly, sharding DRuntime would be good, so why not both? Sure, it’s more work, but it should also provide value, right?
Jul 02