digitalmars.D - Transitioning std lib modules/packages
- Andrei Alexandrescu (26/26) Nov 25 2016 The recent discussion about std.random suggested that we need a
- Ilya Yaroshenko (13/28) Nov 25 2016 The new Random module was builded in Better C concept in mind.
- Andrei Alexandrescu (18/27) Nov 25 2016 I don't know of precedent in other languages whereby the standard
- Jonathan M Davis via Digitalmars-d (37/62) Nov 25 2016 Two other alternatives:
- rikki cattermole (3/28) Nov 25 2016 std.math.random
- Andrei Alexandrescu (3/4) Nov 26 2016 The idea here is not to find new tricks for each of the modules we want
- Andrea Fontana (30/37) Nov 28 2016 My favorite solution is the std.v2.random. Hoping std.v2.stdio
- Andrei Alexandrescu (3/4) Nov 28 2016 That doesn't work - how do you later make breaking changes to
- Andrea Fontana (5/10) Nov 28 2016 It's just an api level. When you need a breaking change all
- Andrea Fontana (8/19) Nov 28 2016 So every (breaking) phobos release import the last version of
- Andrei Alexandrescu (4/13) Nov 28 2016 I see - although that's an awesome idea for github, it seems excessive
- Andrea Fontana (12/30) Nov 28 2016 It's just a simple tag. And I hope that breaking changes are not
- rikki cattermole (3/29) Nov 28 2016 Instead of "API_LEVEL_DMD_*" this already works:
- Ilya Yaroshenko (3/26) Nov 28 2016 Compiler version should be split from a library versions. --Ilya
- Andrea Fontana (8/12) Nov 28 2016 But I can't set from compiler command line, I think. Something
- Ilya Yaroshenko (6/20) Nov 28 2016 I propose Phobos to be an a packed by default collection of
- Andrei Alexandrescu (12/17) Nov 28 2016 This is interesting stuff but I don't know of any precedent, so we'd be
- Ilya Yaroshenko (6/8) Nov 28 2016 We have a small community without commercial support but 3
- Jonathan M Davis via Digitalmars-d (29/44) Nov 28 2016 I confess that I fail to see the benefit. It's normal for the compiler a...
- Jonathan M Davis via Digitalmars-d (25/26) Nov 28 2016 While that might theoretically be nice, I don't think that it's particul...
- Ilya Yaroshenko (8/26) Nov 28 2016 Agreed. In other hand, we may want to evaluate all or a big part
The recent discussion about std.random suggested that we need a transitioning scheme for certain modules and packages in std that provide different designs without breaking the existing ones. C++ has things like std::tr1 and #if __cplusplus >= 201103L. We also need to have separate names for modules that are redone with incompatible APIs, such as std.random or std.json. For a while now I was thinking along the lines of std.v2.random, std.v2.json etc. but that is misleading - it suggests the entire std has a v2 version from which we pick some specific modules. In reality, it's only specific modules that have a v2 version. So I have a low-tech idea and a high-tech one: * Low-tech: just use std.random_v2, std.json_v2, etc. Leave std.random, std.json etc be and put them in maintenance mode. Possibly deprecate them later if the v2 versions work great. * High-tech: use std.random.v2, std.json.v2, etc. This entails more administrative work (convert modules to packages etc) but may have some advantages due to the version being a separate symbol instead of embedded in the name. So I'm leaning toward the low-tech approach unless evidence comes up that the other is better. The clear message here is that we do need to make good strides toward replacing artifacts that need replacing without being crippled by backward compatibility. So if Ilya wants to merge his random work (heh) into std, he would start with std.experimental.random_v2, and then upon approval move that to std.random_v2. Andrei
Nov 25 2016
On Friday, 25 November 2016 at 16:10:51 UTC, Andrei Alexandrescu wrote:The recent discussion about std.random suggested that we need a transitioning scheme for certain modules and packages in std that provide different designs without breaking the existing ones. C++ has things like std::tr1 and #if __cplusplus >= 201103L. We also need to have separate names for modules that are redone with incompatible APIs, such as std.random or std.json. For a while now I was thinking along the lines of std.v2.random, std.v2.json etc. but that is misleading - it suggests the entire std has a v2 version from which we pick some specific modules. In reality, it's only specific modules that have a v2 version. So I have a low-tech idea and a high-tech one: So if Ilya wants to merge his random work (heh) into std, he would start with std.experimental.random_v2, and then upon approval move that to std.random_v2.The new Random module was builded in Better C concept in mind. Comparing with ndslice, random module has precompiled code. And we need to solve not only the naming problem. We need integration with dub where we can select and download/override/recompile appropriate desired version of a Phobos part like random. This is very important for potential contributors and companies. I think that Better C Phobos packages may have their own prefix, like `ext.` (external extension from community). Or, better, `mir.`, hehe Thanks, Ilya
Nov 25 2016
On 11/25/2016 11:39 AM, Ilya Yaroshenko wrote:The new Random module was builded in Better C concept in mind. Comparing with ndslice, random module has precompiled code. And we need to solve not only the naming problem. We need integration with dub where we can select and download/override/recompile appropriate desired version of a Phobos part like random. This is very important for potential contributors and companies.I don't know of precedent in other languages whereby the standard library is built by mixing and matching third-party components. At some point we need to commit to supporting a core standard library. Having folks replace parts of it may be allowed but to support it formally seems a bit much. At any rate it would be good to treat this as an orthogonal matter. Don't let that prevent you from doing good work in std.I think that Better C Phobos packages may have their own prefix, like `ext.` (external extension from community).There are any number of cross-cutting concerns that one might want to look at a library. Which parts are safe? Which parts don't throw? Which parts need the GC? Which parts rely on the runtime? And so on. Ascribing hierarchical namespaces to some or all of these seems counterproductive. Just document is as "this module does not rely on the runtime" and move on.Or, better, `mir.`, heheIf receiving appropriate recognition/branding for your work on the standard library is important to you, I'll be glad to discuss that privately. Thanks, Andrei
Nov 25 2016
On Friday, November 25, 2016 11:10:51 Andrei Alexandrescu via Digitalmars-d wrote:The recent discussion about std.random suggested that we need a transitioning scheme for certain modules and packages in std that provide different designs without breaking the existing ones. C++ has things like std::tr1 and #if __cplusplus >= 201103L. We also need to have separate names for modules that are redone with incompatible APIs, such as std.random or std.json. For a while now I was thinking along the lines of std.v2.random, std.v2.json etc. but that is misleading - it suggests the entire std has a v2 version from which we pick some specific modules. In reality, it's only specific modules that have a v2 version. So I have a low-tech idea and a high-tech one: * Low-tech: just use std.random_v2, std.json_v2, etc. Leave std.random, std.json etc be and put them in maintenance mode. Possibly deprecate them later if the v2 versions work great. * High-tech: use std.random.v2, std.json.v2, etc. This entails more administrative work (convert modules to packages etc) but may have some advantages due to the version being a separate symbol instead of embedded in the name. So I'm leaning toward the low-tech approach unless evidence comes up that the other is better. The clear message here is that we do need to make good strides toward replacing artifacts that need replacing without being crippled by backward compatibility. So if Ilya wants to merge his random work (heh) into std, he would start with std.experimental.random_v2, and then upon approval move that to std.random_v2.Two other alternatives: 1. Move std/json.d into std/json/package.d, and then add the new stuff as modules in std/json/ but don't have std/json/package.d publicly import anything from std.json.*. It would have the advantage of not breaking any code while allowing us to still use std.json longterm. The downside is that you couldn't import std.json and get the new stuff until the old stuff had gone through the whole deprecation process, and it could be confusing in that normally, when something is both a package and a module, importing the module imports the whole package, which it wouldn't do in this case for a couple of years. 2. Make the new json be something like std.data.json. We'd already talked about doing that before. Then the new xml stuff could go in std.data.xml. It avoids the whole v2 thing by further reorganizing Phobos in a hierarchical manner. In the case of std.json, I definitely favor going with the std.data.* suggestion, since I think that it's cleanest, and it avoids the whole v2 problem altogether. Unfortunately, that still leaves the std.random problem, since I'm not aware of an equivalent to std.data.* for std.random nor is there any organizational reason AFAIK to put it somewhere else in the hierarchy. So, for std.random, I don't know. The "high-tech" approach has problems similar to my first alternative with regards to confusion of importing std.random vs std.random.v2, but it's not as bad with the high-tech approach, because it's clearly in a sub-package/module rather than effectively being directly in the same package and getting different versions based on whether you import the package or the individual modules. That being the case, std.random_v2 is probably better, but it just seems ugly IMHO. Then again _any_ solution with v2 is ugly. Long term, we could alias it to std.random, but we'd have to leave the v2 there or be needlessly breaking folks code just for a name change. So, the "low-tech" approach is probably best when it doesn't make sense to move it somewhere else in the package hierarchy, but I don't particularly like it, and in the cases where it makes sense to move the module to make it more hierarchical (as seems to be the case with json and xml), then I favor that approach. - Jonathan M Davis
Nov 25 2016
On 26/11/2016 6:16 AM, Jonathan M Davis via Digitalmars-d wrote:2. Make the new json be something like std.data.json. We'd already talked about doing that before. Then the new xml stuff could go in std.data.xml. It avoids the whole v2 thing by further reorganizing Phobos in a hierarchical manner. In the case of std.json, I definitely favor going with the std.data.* suggestion, since I think that it's cleanest, and it avoids the whole v2 problem altogether. Unfortunately, that still leaves the std.random problem, since I'm not aware of an equivalent to std.data.* for std.random nor is there any organizational reason AFAIK to put it somewhere else in the hierarchy. So, for std.random, I don't know. The "high-tech" approach has problems similar to my first alternative with regards to confusion of importing std.random vs std.random.v2, but it's not as bad with the high-tech approach, because it's clearly in a sub-package/module rather than effectively being directly in the same package and getting different versions based on whether you import the package or the individual modules. That being the case, std.random_v2 is probably better, but it just seems ugly IMHO. Then again _any_ solution with v2 is ugly. Long term, we could alias it to std.random, but we'd have to leave the v2 there or be needlessly breaking folks code just for a name change. So, the "low-tech" approach is probably best when it doesn't make sense to move it somewhere else in the package hierarchy, but I don't particularly like it, and in the cases where it makes sense to move the module to make it more hierarchical (as seems to be the case with json and xml), then I favor that approach. - Jonathan M Davisstd.math.random Well it does deal with numbers pretty much solely.
Nov 25 2016
On 11/25/2016 09:15 PM, rikki cattermole wrote:std.math.randomThe idea here is not to find new tricks for each of the modules we want to evolve, but instead devise a systematic migration path. -- Andrei
Nov 26 2016
On Friday, 25 November 2016 at 16:10:51 UTC, Andrei Alexandrescu wrote:* Low-tech: just use std.random_v2, std.json_v2, etc. Leave std.random, std.json etc be and put them in maintenance mode. Possibly deprecate them later if the v2 versions work great. * High-tech: use std.random.v2, std.json.v2, etc. This entails more administrative work (convert modules to packages etc) but may have some advantages due to the version being a separate symbol instead of embedded in the name.My favorite solution is the std.v2.random. Hoping std.v2.stdio works and it's an alias to std.stdio. In this way it's easier to read documentation (I just need to go to doc/v2). If you write a vXX for each packet we'll have: std.v5.random std.stdio std.v2.json And it's quite confusing. IMHO: std.v5.random std.v5.stdio == std.stdio == std.v2.stdio == std.v3.stdio == std.v4.stdio std.v5.json == std.v3.json == std.v4.json is easier to work with (we simply choose an api level/snapshot of phobos). Moreover I think that a new version of a module can have dependencies from another new module. I mean: std.v5.random could depend on std.v3.math. I'm using std.math and I'm not aware of this dependency and probably it is going to break something (or at least to generate more code than necessary). Instead if i use std.v5.random and std.v5.math they are all on the same level and all dependencies should work fine. Maybe we can define a const (just like c++) PHOBOS_API_LEVEL = XX we can override with a dmd flag (but set to max level inside phobos source) so we can avoid ugly xxx.v3.yyy names if not needed. Andrea
Nov 28 2016
On 11/28/2016 04:13 AM, Andrea Fontana wrote:Hoping std.v2.stdio works and it's an alias to std.stdio.That doesn't work - how do you later make breaking changes to std.v2.stdio? We'd be missing the very point of doing this. -- Andrei
Nov 28 2016
On Monday, 28 November 2016 at 13:34:42 UTC, Andrei Alexandrescu wrote:On 11/28/2016 04:13 AM, Andrea Fontana wrote:It's just an api level. When you need a breaking change all modules will be updated to v3 as alias except the edited ones. V3 it just like a commit-id on github.Hoping std.v2.stdio works and it's an alias to std.stdio.That doesn't work - how do you later make breaking changes to std.v2.stdio? We'd be missing the very point of doing this. -- Andrei
Nov 28 2016
On Monday, 28 November 2016 at 13:42:30 UTC, Andrea Fontana wrote:On Monday, 28 November 2016 at 13:34:42 UTC, Andrei Alexandrescu wrote:So every (breaking) phobos release import the last version of each module. If i have an old code to support i just need to compile with: dmd .... -version=PHOBOS_API_LEVEL_2 and import std.stdio; // will import std.v2.stdio instead of predefine std.v3.stdioOn 11/28/2016 04:13 AM, Andrea Fontana wrote:It's just an api level. When you need a breaking change all modules will be updated to v3 as alias except the edited ones. V3 it just like a commit-id on github.Hoping std.v2.stdio works and it's an alias to std.stdio.That doesn't work - how do you later make breaking changes to std.v2.stdio? We'd be missing the very point of doing this. -- Andrei
Nov 28 2016
On 11/28/2016 08:42 AM, Andrea Fontana wrote:On Monday, 28 November 2016 at 13:34:42 UTC, Andrei Alexandrescu wrote:I see - although that's an awesome idea for github, it seems excessive to bump the version of the entire stdlib when we want to transition one module or package of it. -- AndreiOn 11/28/2016 04:13 AM, Andrea Fontana wrote:It's just an api level. When you need a breaking change all modules will be updated to v3 as alias except the edited ones. V3 it just like a commit-id on github.Hoping std.v2.stdio works and it's an alias to std.stdio.That doesn't work - how do you later make breaking changes to std.v2.stdio? We'd be missing the very point of doing this. -- Andrei
Nov 28 2016
On Monday, 28 November 2016 at 13:51:13 UTC, Andrei Alexandrescu wrote:On 11/28/2016 08:42 AM, Andrea Fontana wrote:It's just a simple tag. And I hope that breaking changes are not that frequent! I think that on worst case there will be a new api level every dmd release. It just happens, but in this way it is managed in a better way. If needed we can use API_LEVEL_DMD_2071_3 as identifier. But IMO PHOBOS_API_LEVEL_2 makes more sense. It would be really easy to mantain a back-compatibility. Just select the right api level, and you can still use new dmd version! Something similar really works fine with android api_level.On Monday, 28 November 2016 at 13:34:42 UTC, Andrei Alexandrescu wrote:I see - although that's an awesome idea for github, it seems excessive to bump the version of the entire stdlib when we want to transition one module or package of it. -- AndreiOn 11/28/2016 04:13 AM, Andrea Fontana wrote:It's just an api level. When you need a breaking change all modules will be updated to v3 as alias except the edited ones. V3 it just like a commit-id on github.Hoping std.v2.stdio works and it's an alias to std.stdio.That doesn't work - how do you later make breaking changes to std.v2.stdio? We'd be missing the very point of doing this. -- Andrei
Nov 28 2016
On 29/11/2016 3:01 AM, Andrea Fontana wrote:On Monday, 28 November 2016 at 13:51:13 UTC, Andrei Alexandrescu wrote:Instead of "API_LEVEL_DMD_*" this already works: static if (__VERSION__ >= 2072)On 11/28/2016 08:42 AM, Andrea Fontana wrote:It's just a simple tag. And I hope that breaking changes are not that frequent! I think that on worst case there will be a new api level every dmd release. It just happens, but in this way it is managed in a better way. If needed we can use API_LEVEL_DMD_2071_3 as identifier. But IMO PHOBOS_API_LEVEL_2 makes more sense. It would be really easy to mantain a back-compatibility. Just select the right api level, and you can still use new dmd version! Something similar really works fine with android api_level.On Monday, 28 November 2016 at 13:34:42 UTC, Andrei Alexandrescu wrote:I see - although that's an awesome idea for github, it seems excessive to bump the version of the entire stdlib when we want to transition one module or package of it. -- AndreiOn 11/28/2016 04:13 AM, Andrea Fontana wrote:It's just an api level. When you need a breaking change all modules will be updated to v3 as alias except the edited ones. V3 it just like a commit-id on github.Hoping std.v2.stdio works and it's an alias to std.stdio.That doesn't work - how do you later make breaking changes to std.v2.stdio? We'd be missing the very point of doing this. -- Andrei
Nov 28 2016
On Monday, 28 November 2016 at 14:03:31 UTC, rikki cattermole wrote:On 29/11/2016 3:01 AM, Andrea Fontana wrote:Compiler version should be split from a library versions. --IlyaOn Monday, 28 November 2016 at 13:51:13 UTC, Andrei Alexandrescu wrote:Instead of "API_LEVEL_DMD_*" this already works: static if (__VERSION__ >= 2072)[...]It's just a simple tag. And I hope that breaking changes are not that frequent! I think that on worst case there will be a new api level every dmd release. It just happens, but in this way it is managed in a better way. If needed we can use API_LEVEL_DMD_2071_3 as identifier. But IMO PHOBOS_API_LEVEL_2 makes more sense. It would be really easy to mantain a back-compatibility. Just select the right api level, and you can still use new dmd version! Something similar really works fine with android api_level.
Nov 28 2016
On Monday, 28 November 2016 at 14:09:10 UTC, Ilya Yaroshenko wrote:But I can't set from compiler command line, I think. Something like: dmd my_old_source.d -version=PHOBOS_API_LEVEL_2 And I think that inside phobos source a lot of breaking changes simply override old code and checking __VERSION__ is not useful at all. Am I wrong?Instead of "API_LEVEL_DMD_*" this already works: static if (__VERSION__ >= 2072)Compiler version should be split from a library versions. --Ilya
Nov 28 2016
On Monday, 28 November 2016 at 14:15:21 UTC, Andrea Fontana wrote:On Monday, 28 November 2016 at 14:09:10 UTC, Ilya Yaroshenko wrote:I propose Phobos to be an a packed by default collection of libraries with clear dependencies and community support. A package version should be overridable in dub configuration. This is what we need to involve companies to invest their efforts in dlang infrastructure. --IlyaBut I can't set from compiler command line, I think. Something like: dmd my_old_source.d -version=PHOBOS_API_LEVEL_2 And I think that inside phobos source a lot of breaking changes simply override old code and checking __VERSION__ is not useful at all. Am I wrong?Instead of "API_LEVEL_DMD_*" this already works: static if (__VERSION__ >= 2072)Compiler version should be split from a library versions. --Ilya
Nov 28 2016
On 11/28/16 9:09 AM, Ilya Yaroshenko wrote:Compiler version should be split from a library versions. --IlyaOn 11/28/16 9:26 AM, Ilya Yaroshenko wrote:I propose Phobos to be an a packed by default collection of libraries with clear dependencies and community support. A package version should be overridable in dub configuration. This is what we need to involve companies to invest their efforts in dlang infrastructure. --IlyaThis is interesting stuff but I don't know of any precedent, so we'd be taking risks trying this. All language platforms I know of ship with one compiler tied to one fixed standard library implementation. (Then, third party libraries come on top of that.) The obvious issue with not doing that is dealing with maintaining a fuzzy combinatorial number of setups instead of one. The risks may, of course, be justified if this turns out to be a nice innovation but definitely the tradeoffs must be analyzed properly. Martin, what do you think of this? Andrei
Nov 28 2016
On Monday, 28 November 2016 at 14:59:28 UTC, Andrei Alexandrescu wrote:All language platforms I know of ship with one compiler tied to one fixed standard library implementation.We have a small community without commercial support but 3 compilers, large standard library and large ambitions to extend it more. We can / should split dependencies and make things clear. Travis, Circle CI, and Appveyor are very helpful. --Ilya
Nov 28 2016
On Monday, November 28, 2016 09:59:28 Andrei Alexandrescu via Digitalmars-d wrote:On 11/28/16 9:09 AM, Ilya Yaroshenko wrote:I confess that I fail to see the benefit. It's normal for the compiler and standard library to be tied together. This sounds like an attempt to get rid of the standard library entirely in favor of _everything_ being 3rd party code, which seems like the complete opposite direction that the programming community at large has been going. We should obviously support 3rd party software, but why would turning the standard library into 3rd party software make sense? If someone wants to create an alternative implementation to something in the standard library, that's perfectly normal. Maybe they have a use case that would be better served by a more specialized API or implementation. There's no need to get the standard library involved with that, and if you can't rely on knowing that std.algorithm or std.datetime or std.random or std.whatever is the one from the actual standard library when you read the code, that seems like a recipe for disaster. It's not standard anymore at all. I say leave 3rd party code in 3rd party land, and standard stuff should be left in druntime/Phobos. And if someone wants to go through the review process and get 3rd party stuff into Phobos and make it standard, then great. But dub already makes it possible for folks to create their own libraries and distribute them to others to use in their projects if they don't want to go through the process of trying to get them included in Phobos or don't think that they belong in the standard library. Phobos and its review process doesn't need to be involved with whatever folks want to do with 3rd party libraries on dub, and I don't see any reason to try and treat the standard library - or any part of the standard library - like a third party component. - Jonathan M DavisCompiler version should be split from a library versions. --IlyaOn 11/28/16 9:26 AM, Ilya Yaroshenko wrote:I propose Phobos to be an a packed by default collection of libraries with clear dependencies and community support. A package version should be overridable in dub configuration. This is what we need to involve companies to invest their efforts in dlang infrastructure. --IlyaThis is interesting stuff but I don't know of any precedent, so we'd be taking risks trying this. All language platforms I know of ship with one compiler tied to one fixed standard library implementation. (Then, third party libraries come on top of that.) The obvious issue with not doing that is dealing with maintaining a fuzzy combinatorial number of setups instead of one. The risks may, of course, be justified if this turns out to be a nice innovation but definitely the tradeoffs must be analyzed properly.
Nov 28 2016
On Monday, November 28, 2016 14:09:10 Ilya Yaroshenko via Digitalmars-d wrote:Compiler version should be split from a library versions. --IlyaWhile that might theoretically be nice, I don't think that it's particularly realistic, and it's definitely not how things have functioned for us historically. dmd and druntime are definitely too interconnected to separate those, and Phobos relies on druntime, so to some extent at least, it's tied to the version of druntime. It's also the case that it's often tied to a specific version of the compiler, because there's a feature change, and Phobos has to be updated accordingly (e.g. Walter has been making various improvements to safe of late, and that has required changes in Phobos). The APIs of druntime and Phobos are reasonably stable, but they aren't completely stable, and their ABIs aren't stable. We're not constantly changing them, but there is no guarantee that they stay the same from release to release. We just try and ensure that user code that built and ran with the previous release continues to do so with the current one. There is no attempt to ensure that Phobos from one release builds with another, and it would make updating stuff way more of a pain if we had to worry about that. There is also no attempt to make it so that code _built_ with the previous release will work with the current one. It's expected that if you're updating the compiler, you're going to update druntime and Phobos. And I don't think that that's unreasonable. It's actually pretty normal. I don't know of any language that separates their compiler releases from their standard library releases. They always go together. That's part of what comes with them being standard. - Jonathan M Davis
Nov 28 2016
On Monday, 28 November 2016 at 13:51:13 UTC, Andrei Alexandrescu wrote:On 11/28/2016 08:42 AM, Andrea Fontana wrote:Agreed. In other hand, we may want to evaluate all or a big part of Phobos to the new version instead of project-by-project migration torment. std2.*/std.v2.* or something like that. Also we need split a code, that should be only generic, only precompiled, only inlined, and other code. This will be very helpful for painless evaluation. --IlyaOn Monday, 28 November 2016 at 13:34:42 UTC, Andrei Alexandrescu wrote:I see - although that's an awesome idea for github, it seems excessive to bump the version of the entire stdlib when we want to transition one module or package of it. -- AndreiOn 11/28/2016 04:13 AM, Andrea Fontana wrote:It's just an api level. When you need a breaking change all modules will be updated to v3 as alias except the edited ones. V3 it just like a commit-id on github.Hoping std.v2.stdio works and it's an alias to std.stdio.That doesn't work - how do you later make breaking changes to std.v2.stdio? We'd be missing the very point of doing this. -- Andrei
Nov 28 2016