www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Interesting article and discussion about Python's standard library

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://news.ycombinator.com/item?id=19948642
May 18
next sibling parent reply ikod <geller.garry gmail.com> writes:
On Sunday, 19 May 2019 at 03:33:15 UTC, Andrei Alexandrescu wrote:
 https://news.ycombinator.com/item?id=19948642
Brown asserted that there were many such dependencies from the 
standard library to PyPI: typing works best with mypy, the ssl 
module requires a monkeypatch to connect to non-ASCII domain 
names, datetime needs pytz, and six is non-optional for writing 
code for Python 2 and 3.
Actually I see no problem here. I'm using python as everyday tool, and yes, there is lot of packages in Pypi repo that are more user-friendly or more complete than standard library modules, but not every pypi module works on every platform where python library works and not always compatible with every python version (and this sometimes is real PITA). Her point is to move some parts of standard library to PyPI, but I'm not sure if this is good. The real point should be to get better support for modules regardless of their location - std library or PyPI.
May 18
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 19 May 2019 at 04:10:22 UTC, ikod wrote:
 Actually I see no problem here. I'm using python as everyday 
 tool, and yes, there is lot of packages in Pypi repo that are 
 more user-friendly or more complete than standard library 
 modules, but not every pypi module works on every platform 
 where python library works and not always compatible with every 
 python version (and this sometimes is real PITA).
Yup, seems like people somehow forgets what a scripting language traditionally has been used for as they now also increasingly are used for applications. The libs that have been outdated the most is the HTTP/Web ones, and that is absolutely crucial as a builtin for a useful scripting language. Anyway, with the standard lib I can just install the default Python build and set up a script to set up daily transfer of stuff from some old windows server to the cloud. All I need is MSIE+notepad. Cost saving efficiency, even despite having to do some workarounds. Get it done same day on a an outdated nothing-installed server with no hickups. Solving boring tasks like these on time is what has made Perl, Python, bash/unix popular. Actually, I've found it bothersome and time consuming that the std lib lacked key crypto algorithms. I've also had to improvise my own encoding for HTTP POST on site. So, no, the Python 2 library was not too big at all, it was quite deficient in terms of features. Anyway, Python is a scripting language often used for hashing out solutions on-site at a break-neck pace (near real time). Python should ship configured as a swiss armyknife and then some. As long as weaknesses are documented in the Python docs, then it is ok that some of it is imperfect. I'd rather have a leaky firehose when the house is burning than none at all! D and Go are not scripting languages! Scripting languages need builtins that enable fast deployment and quick fixes.
May 19
parent reply Francis Locke <f.locke flmail.su> writes:
On Sunday, 19 May 2019 at 13:39:04 UTC, Ola Fosheim Grøstad wrote:
 On Sunday, 19 May 2019 at 04:10:22 UTC, ikod wrote:
 D and Go are not scripting languages! Scripting languages need 
 builtins that enable fast deployment and quick fixes.
I don't know about Go, but I use D for "scripting," a lot. (Go rdmd!) Both D and Python can be described as general-purpose languages. (Hell, D's landing page advertises it as such.) And a good general-purpose language, for me, is one that gives a nice, stable stdlib that allows you to build your initial solution without having to worry about dependencies. In fact, I wish D's stdlib was bigger! Ms. Brown has ostensibly lost sight of this.
May 19
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 19 May 2019 at 18:54:03 UTC, Francis Locke wrote:
 I don't know about Go, but I use D for "scripting," a lot. (Go 
 rdmd!)
Yes, I remember I was amazed that a compiled language could be used like that the first time it was introduced. :-) I view D more as a language that originated with aspirations towards systems-programming then ended up with more emphasis towards application programming and bends the limits towards scripting (or prototyping). Whereas I view Python primarily as a scripting language ("real time" patch-things-together coding) that has bent the limits towards application development.
 And a good general-purpose language, for me, is one that gives 
 a nice, stable stdlib that allows you to build your initial 
 solution without having to worry about dependencies. In fact, I 
 wish D's stdlib was bigger! Ms. Brown has ostensibly lost sight 
 of this.
Got ya. :-) I guess there are three contradicting views: 1. For a low level system programming language then you want a small runtime and small library that is easy to port and that makes it possible to reuse other libraries without too much bloat (so all third party libraries only depend on a limited standard library). 2. For scripting and high level programming you want the exact opposite. You want a large coherent standard library that provides a shared set of concepts and names for constructs that becomes a shared language between programmers (e.g. ranges in D stdlib). 3. For system level application development you want a standard library that complement the native operating system you target in a very clean and coherent fashion. And provides access to the specific features of the OS (like subscribing to file directory changes, GPS location and what not).
May 19
prev sibling next sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Sat, 2019-05-18 at 23:33 -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
 https://news.ycombinator.com/item?id=3D19948642
I'd say Amber Brown is right, the Python standard library needs stripping b= ack quite a lot and stuff moved to PyPI. Or pip-able using Git,Mercurial, Breez= y, etc. directly so as to access packages not in PyPI. Go and Rust have both gone with the small standard library and strong suppo= rt for bringing in dependencies. Go only uses DVCS mechanisms, shunning a cent= ral repository of packages. For me this is wrong. Rust has gone with support fo= r a central repository and DVCS. I find this the right route. D has a central library but no support for DVCS for getting dependencies in Dub. This should be fixed.=20 I failed to do anything about it at the D hack day, but what I think is nee= ded is a library separate from Dub (or perhaps split off from Dub that Dub then uses) for accessing the central D library. This will then allow Meson, SCon= s, etc. to better support working with the central D repository and used DVCS = to access packages. This should then make adding DVCS support relatively straightforward. With this in place Phobos can then be stripped back to only that that is absolutely needed. Rust is a really good model here. Python, Go, and D should follow suit. =20 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Road m: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk
May 19
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 19 May 2019 at 11:48:44 UTC, Russel Winder wrote:
 Rust is a really good model here. Python, Go, and D should 
 follow suit.
I don't think Python should not follow suit. It is in a different league. I can trust the standard library to not contain malicious code. Buggy code is far less problematic than the possibility of facing trojans. If I use code from a third party I will have to read and vet every single line of code to ensure that there is no backdoor. So that limit me to very short code segments. I've done it for crypto code. I'd rather not do it. I'd rather use imperfect standard library code. You want code that has had many eyes on it and where the flaws are well documented when you install scripts behind firewalls of commercial entities.
May 19
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-19 13:48, Russel Winder wrote:

 I failed to do anything about it at the D hack day, but what I think is needed
 is a library separate from Dub (or perhaps split off from Dub that Dub then
 uses) for accessing the central D library. This will then allow Meson, SCons,
 etc. to better support working with the central D repository and used DVCS to
 access packages. This should then make adding DVCS support relatively
 straightforward.
How about a new command for Dub that allows it to work like a package manager without the building stuff? -- /Jacob Carlborg
May 19
parent sarn <sarn theartofmachinery.com> writes:
On Sunday, 19 May 2019 at 19:08:27 UTC, Jacob Carlborg wrote:
 How about a new command for Dub that allows it to work like a 
 package manager without the building stuff?
In *nix land, there's the foo-config idiom, i.e., a command that gives you the flags needed to compile something with foo. For example, sdl-config: $ sdl-config --libs -lSDL -lpthread $ sdl-config --cflags -I/usr/include/SDL -D_GNU_SOURCE=1 -D_REENTRANT Typical usage: $ gcc $(sdl-config --cflags) --other --flags -c thing_using_sdl.c It's useful because it works with any build system. In the past I've used dummy projects containing nothing but a dub.json, and parsed the output of dub describe. But I've only made an 80% solution, not something good enough to release as an official tool.
May 19
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/18/2019 8:33 PM, Andrei Alexandrescu wrote:
 https://news.ycombinator.com/item?id=19948642
Unfortunately, there's no clear answer to this problem. One issue not mentioned - while it shouldn't be, Phobos is an essential part of the compiler test suite. 1. the compiler test suite is inadequate 2. breaking Phobos is a good predictor of what will break in user code
May 19
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 19 May 2019 at 03:33:15 UTC, Andrei Alexandrescu wrote:
 https://news.ycombinator.com/item?id=19948642
Thanks for sharing. I think the hacker news comments were nearly as interesting as the article itself. It's a hard problem figuring out what should go in the standard library or not and there's probably n one right answer.
May 21
parent reply David Gileadi <gileadisNOSPM gmail.com> writes:
On 5/21/19 2:14 AM, Atila Neves wrote:
 On Sunday, 19 May 2019 at 03:33:15 UTC, Andrei Alexandrescu wrote:
 https://news.ycombinator.com/item?id=19948642
Thanks for sharing. I think the hacker news comments were nearly as interesting as the article itself. It's a hard problem figuring out what should go in the standard library or not and there's probably n one right answer.
I didn't know this about Rust, so I found the following comment interesting:
 For example, the libstd doesn't even have support for random number
generation. There's a rand crate, which is now on 6th major breaking version.
That's perfectly fine, because multiple versions can coexist in one program,
and every user can upgrade (or not) at their own pace. And the crate was able
to refine its interface six times, instead of being stuck with the first try
forever.
Multiple coexisting versions is an interesting way of solving dependency hell.
May 21
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/21/19 11:15 AM, David Gileadi wrote:
 On 5/21/19 2:14 AM, Atila Neves wrote:
 For example, the libstd doesn't even have support for random number 
 generation. There's a rand crate, which is now on 6th major breaking 
 version. That's perfectly fine, because multiple versions can coexist 
 in one program, and every user can upgrade (or not) at their own pace. 
 And the crate was able to refine its interface six times, instead of 
 being stuck with the first try forever.
Multiple coexisting versions is an interesting way of solving dependency hell.
In terms of multiple simultaneously *installed* versions, I consider it an absolutely essential feature of any *good* package manager (This is a big part of why I hate most system-level Linux package managers, including the one on my own machine). But the idea of multiple versions *within one program* is indeed interesting. Doesn't strike me as something you'd want to frequently take advantage of, but having it there as a viable workaround for when problems do arise could certainly be quite nice.
May 21
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, May 21, 2019 at 12:29:55PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
 On 5/21/19 11:15 AM, David Gileadi wrote:
 On 5/21/19 2:14 AM, Atila Neves wrote:
 For example, the libstd doesn't even have support for random
 number generation. There's a rand crate, which is now on 6th major
 breaking version. That's perfectly fine, because multiple versions
 can coexist in one program, and every user can upgrade (or not) at
 their own pace. And the crate was able to refine its interface six
 times, instead of being stuck with the first try forever.
Multiple coexisting versions is an interesting way of solving dependency hell.
In terms of multiple simultaneously *installed* versions, I consider it an absolutely essential feature of any *good* package manager (This is a big part of why I hate most system-level Linux package managers, including the one on my own machine). But the idea of multiple versions *within one program* is indeed interesting. Doesn't strike me as something you'd want to frequently take advantage of, but having it there as a viable workaround for when problems do arise could certainly be quite nice.
This is a very interesting idea indeed. Goes along well with Andrei's recent idea of addition vs. replacement. And I think it's probably not hard to adopt in D (should we deem it a good idea -- that's still arguable): all you need is for module declarations to come with an attached version number, then include that in the mangling. This automatically gives a unique identifier to symbols in the module, so you could import multiple versions of the same module and have the symbols resolve correctly. Just a minor change in module declarations and in module name mangling. Of course, how to handle imports that may have multiple satisfying versions is another, stickier, question. T -- Meat: euphemism for dead animal. -- Flora
May 21
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 21 May 2019 09:39:29 -0700 schrieb H. S. Teoh:

 On Tue, May 21, 2019 at 12:29:55PM -0400, Nick Sabalausky (Abscissa) via
 In terms of multiple simultaneously *installed* versions, I consider it
 an absolutely essential feature of any *good* package manager (This is
 a big part of why I hate most system-level Linux package managers,
 including the one on my own machine). But the idea of multiple versions
 *within one program* is indeed interesting.
 
 Doesn't strike me as something you'd want to frequently take advantage
 of, but having it there as a viable workaround for when problems do
 arise could certainly be quite nice.
This is a very interesting idea indeed. Goes along well with Andrei's recent idea of addition vs. replacement. And I think it's probably not hard to adopt in D (should we deem it a good idea -- that's still arguable):
Having multiple versions of libraries only works well though if there are no dependency cycles between third party modules. Random has a very 'closed' API, so for that it works well. As another extreme example, consider we have six different versions for std.range. Then we have std.file which provides a v1 range, std.format which only formats v3 ranges, vibe.d which still uses v2 and your applications which want's to combine all these and pass ranges from std.file to vibe.d and std.format. -- Johannes
May 21
parent reply sarn <sarn theartofmachinery.com> writes:
On Tuesday, 21 May 2019 at 18:06:44 UTC, Johannes Pfau wrote:
 Having multiple versions of libraries only works well though if 
 there are no dependency cycles between third party modules. 
 Random has a very 'closed' API, so for that it works well.
In the JavaScript world, npm doesn't have that problem because it gives each dependency its own dependency tree. (If D tried that, the ABI compatibility issues would be painful, and the embedded developers would weep at the bloat.)
May 21
next sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/21/19 6:30 PM, sarn wrote:
 On Tuesday, 21 May 2019 at 18:06:44 UTC, Johannes Pfau wrote:
 Having multiple versions of libraries only works well though if there 
 are no dependency cycles between third party modules. Random has a 
 very 'closed' API, so for that it works well.
In the JavaScript world, npm doesn't have that problem because it gives each dependency its own dependency tree. (If D tried that, the ABI compatibility issues would be painful, and the embedded developers would weep at the bloat.)
Wow, that actually explains quite a lot about the modern web.
May 21
prev sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Tuesday, 21 May 2019 at 22:30:17 UTC, sarn wrote:
 On Tuesday, 21 May 2019 at 18:06:44 UTC, Johannes Pfau wrote:
 Having multiple versions of libraries only works well though 
 if there are no dependency cycles between third party modules. 
 Random has a very 'closed' API, so for that it works well.
In the JavaScript world, npm doesn't have that problem because it gives each dependency its own dependency tree. (If D tried that, the ABI compatibility issues would be painful, and the embedded developers would weep at the bloat.)
And that's why a simple vue.js project on my machine gives a node_modules directory that's crowded as a termite mound ... Please no ... /P
May 22
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 22 May 2019 at 08:00:15 UTC, Paolo Invernizzi wrote:
 And that's why a simple vue.js project on my machine gives a 
 node_modules directory that's crowded as a termite mound ...

 Please no ...
Plain npm usage does not seem to pull in two versions of the same library. What is common is to have one module for a single function. That's what the directory bloat comes from. So node_modules look more crowded than it actually is. A lot of those functions belong in a standard library though. For instance Angular pulls in: nice-try: call a function and discard exceptions once: throw an error if the function is called twice onetime: a function returns a cached value if called many times extend: merge two objects extend-shallow: merge in properties from other object And so on…
May 22
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 22 May 2019 at 09:16:28 UTC, Ola Fosheim Grøstad 
wrote:
 Plain npm  usage does not seem to pull in two versions of the 
 same library.
npm allows you to specify a semver range, so if the range does not overlap then it will pull in a "duplicate". Although for libraries that are exposed externally (like react), you can specify that it is a peer dependency that should only exists as one semver version. In which case it should complain. At least, that is my shallow understanding of the system. Allowing version 1 and 2 of a library to coexist in the same build is no different than depending on two independent libraries that provides the same functionality under different names. Makes sense to allow a build to succeed if two modules depends on two different versions of some simple set of functions. Especially in a culture where there tends to be one function per module.
May 22
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 21 May 2019 at 16:39:29 UTC, H. S. Teoh wrote:
 On Tue, May 21, 2019 at 12:29:55PM -0400, Nick Sabalausky 
 (Abscissa) via Digitalmars-d wrote:
 [...]
This is a very interesting idea indeed. Goes along well with Andrei's recent idea of addition vs. replacement. And I think it's probably not hard to adopt in D (should we deem it a good idea -- that's still arguable): all you need is for module declarations to come with an attached version number, then include that in the mangling. This automatically gives a unique identifier to symbols in the module, so you could import multiple versions of the same module and have the symbols resolve correctly. Just a minor change in module declarations and in module name mangling. Of course, how to handle imports that may have multiple satisfying versions is another, stickier, question. T
Also, type versioning. Types from different versions would have to be completely separate, which would break function signatures, overloads, introspection etc.
May 22
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 22, 2019 at 10:55:09AM +0000, John Colvin via Digitalmars-d wrote:
 On Tuesday, 21 May 2019 at 16:39:29 UTC, H. S. Teoh wrote:
 On Tue, May 21, 2019 at 12:29:55PM -0400, Nick Sabalausky (Abscissa) via
 Digitalmars-d wrote:
 [...]
This is a very interesting idea indeed. Goes along well with Andrei's recent idea of addition vs. replacement. And I think it's probably not hard to adopt in D (should we deem it a good idea -- that's still arguable): all you need is for module declarations to come with an attached version number, then include that in the mangling. This automatically gives a unique identifier to symbols in the module, so you could import multiple versions of the same module and have the symbols resolve correctly. Just a minor change in module declarations and in module name mangling. Of course, how to handle imports that may have multiple satisfying versions is another, stickier, question.
[...]
 Also, type versioning. Types from different versions would have to be
 completely separate, which would break function signatures, overloads,
 introspection etc.
It doesn't break, because symbols in D are qualified by module name. So if the module name contains version information, types from different versions will be differently-named. For example, moduleA.ver1_0.myStruct will be distinct from moduleA.ver2_0.myStruct, so you couldn't mix the two together unintentionally without a compile error. T -- If it breaks, you get to keep both pieces. -- Software disclaimer notice
May 22