www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Interesting article and discussion about Python's standard library

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://news.ycombinator.com/item?id=19948642
May 18 2019
next sibling parent reply ikod <geller.garry gmail.com> writes:
On Sunday, 19 May 2019 at 03:33:15 UTC, Andrei Alexandrescu wrote:

 https://news.ycombinator.com/item?id=19948642



Brown asserted that there were many such dependencies from the 
standard library to PyPI: typing works best with mypy, the ssl 
module requires a monkeypatch to connect to non-ASCII domain 
names, datetime needs pytz, and six is non-optional for writing 
code for Python 2 and 3.


Actually I see no problem here. I'm using python as everyday 
tool, and yes, there is lot of packages in Pypi repo that are 
more user-friendly or more complete than standard library 
modules, but not every pypi module works on every platform where 
python library works and not always compatible with every python 
version (and this sometimes is real PITA).

Her point is to move some parts of standard library to PyPI, but 
I'm not sure if this is good.  The real point should be to get 
better support for modules regardless of their location - std 
library or PyPI.
May 18 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 19 May 2019 at 04:10:22 UTC, ikod wrote:

 Actually I see no problem here. I'm using python as everyday 
 tool, and yes, there is lot of packages in Pypi repo that are 
 more user-friendly or more complete than standard library 
 modules, but not every pypi module works on every platform 
 where python library works and not always compatible with every 
 python version (and this sometimes is real PITA).


Yup, seems like people somehow forgets what a scripting language 
traditionally has been used for as they now also increasingly are 
used for applications.

The libs that have been outdated the most is the HTTP/Web ones, 
and that is absolutely crucial as a builtin for a useful 
scripting language.

Anyway, with the standard lib I can just install the default 
Python build and set up a script to set up daily transfer of 
stuff from some old windows server to the cloud. All I need is 
MSIE+notepad. Cost saving efficiency, even despite having to do 
some workarounds. Get it done same day on a an outdated 
nothing-installed server with no hickups.

Solving boring tasks like these on time is what has made Perl, 
Python, bash/unix popular.

Actually, I've found it bothersome and time consuming that the 
std lib lacked key crypto algorithms. I've also had to improvise 
my own encoding for HTTP POST on site. So, no, the Python 2 
library was not too big at all, it was quite deficient in terms 
of features.

Anyway, Python is a scripting language often used for hashing out 
solutions on-site at a break-neck pace (near real time). Python 
should ship configured as a swiss armyknife and then some.

As long as weaknesses are documented in the Python docs, then it 
is ok that some of it is imperfect. I'd rather have a leaky 
firehose when the house is burning than none at all!

D and Go are not scripting languages! Scripting languages need 
builtins that enable fast deployment and quick fixes.
May 19 2019
parent reply Francis Locke <f.locke flmail.su> writes:
On Sunday, 19 May 2019 at 13:39:04 UTC, Ola Fosheim Grøstad wrote:

 On Sunday, 19 May 2019 at 04:10:22 UTC, ikod wrote:
 D and Go are not scripting languages! Scripting languages need 
 builtins that enable fast deployment and quick fixes.


I don't know about Go, but I use D for "scripting," a lot. (Go 
rdmd!) Both D and Python can be described as general-purpose 
languages. (Hell, D's landing page advertises it as such.) And a 
good general-purpose language, for me, is one that gives a nice, 
stable stdlib that allows you to build your initial solution 
without having to worry about dependencies. In fact, I wish D's 
stdlib was bigger! Ms. Brown has ostensibly lost sight of this.
May 19 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 19 May 2019 at 18:54:03 UTC, Francis Locke wrote:

 I don't know about Go, but I use D for "scripting," a lot. (Go 
 rdmd!)


Yes, I remember I was amazed that a compiled language could be 
used like that the first time it was introduced. :-)

I view D more as a language that originated with aspirations 
towards systems-programming then ended up with more emphasis 
towards application programming and bends the limits towards 
scripting (or prototyping).

Whereas I view Python primarily as a scripting language ("real 
time" patch-things-together coding) that has bent the limits 
towards application development.


 And a good general-purpose language, for me, is one that gives 
 a nice, stable stdlib that allows you to build your initial 
 solution without having to worry about dependencies. In fact, I 
 wish D's stdlib was bigger! Ms. Brown has ostensibly lost sight 
 of this.


Got ya. :-) I guess there are three contradicting views:

1. For a low level system programming language then you want a 
small runtime and small library that is easy to port and that 
makes it possible to reuse other libraries without too much bloat 
(so all third party libraries only depend on a limited standard 
library).

2. For scripting and high level programming you want the exact 
opposite. You want a large coherent standard library that 
provides a shared set of concepts and names for constructs that 
becomes a shared language between programmers (e.g. ranges in D 
stdlib).

3. For system level application development you want a standard 
library that complement the native operating system you target in 
a very clean and coherent fashion. And provides access to the 
specific features of the OS (like subscribing to file directory 
changes, GPS location and what not).
May 19 2019
prev sibling next sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Sat, 2019-05-18 at 23:33 -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 https://news.ycombinator.com/item?id=3D19948642


I'd say Amber Brown is right, the Python standard library needs stripping b=
ack
quite a lot and stuff moved to PyPI. Or pip-able using Git,Mercurial, Breez=
y,
etc. directly so as to access packages not in PyPI.

Go and Rust have both gone with the small standard library and strong suppo=
rt
for bringing in dependencies. Go only uses DVCS mechanisms, shunning a cent=
ral
repository of packages. For me this is wrong. Rust has gone with support fo=
r a
central repository and DVCS. I find this the right route.

D has a central library but no support for DVCS for getting dependencies in
Dub. This should be fixed.=20

I failed to do anything about it at the D hack day, but what I think is nee=
ded
is a library separate from Dub (or perhaps split off from Dub that Dub then
uses) for accessing the central D library. This will then allow Meson, SCon=
s,
etc. to better support working with the central D repository and used DVCS =
to
access packages. This should then make adding DVCS support relatively
straightforward.

With this in place Phobos can then be stripped back to only that that is
absolutely needed.

Rust is a really good model here. Python, Go, and D should follow suit.
 =20
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
May 19 2019
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Sunday, 19 May 2019 at 11:48:44 UTC, Russel Winder wrote:

 Rust is a really good model here. Python, Go, and D should 
 follow suit.


I don't think Python should not follow suit. It is in a different 
league.

I can trust the standard library to not contain malicious code. 
Buggy code is far less problematic than the possibility of facing 
trojans.

If I use code from a third party I will have to read and vet 
every single line of code to ensure that there is no backdoor.  
So that limit me to very short code segments. I've done it for 
crypto code. I'd rather not do it.

I'd rather use imperfect standard library code. You want code 
that has had many eyes on it and where the flaws are well 
documented when you install scripts behind firewalls of 
commercial entities.
May 19 2019
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-19 13:48, Russel Winder wrote:


 I failed to do anything about it at the D hack day, but what I think is needed
 is a library separate from Dub (or perhaps split off from Dub that Dub then
 uses) for accessing the central D library. This will then allow Meson, SCons,
 etc. to better support working with the central D repository and used DVCS to
 access packages. This should then make adding DVCS support relatively
 straightforward.


How about a new command for Dub that allows it to work like a package 
manager without the building stuff?

-- 
/Jacob Carlborg
May 19 2019
parent sarn <sarn theartofmachinery.com> writes:
On Sunday, 19 May 2019 at 19:08:27 UTC, Jacob Carlborg wrote:

 How about a new command for Dub that allows it to work like a 
 package manager without the building stuff?


In *nix land, there's the foo-config idiom, i.e., a command that 
gives you the flags needed to compile something with foo.

For example, sdl-config:

$ sdl-config --libs
-lSDL -lpthread
$ sdl-config --cflags
-I/usr/include/SDL -D_GNU_SOURCE=1 -D_REENTRANT

Typical usage:
$ gcc $(sdl-config --cflags) --other --flags -c thing_using_sdl.c

It's useful because it works with any build system.

In the past I've used dummy projects containing nothing but a 
dub.json, and parsed the output of dub describe.  But I've only 
made an 80% solution, not something good enough to release as an 
official tool.
May 19 2019
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/18/2019 8:33 PM, Andrei Alexandrescu wrote:

 https://news.ycombinator.com/item?id=19948642


Unfortunately, there's no clear answer to this problem.

One issue not mentioned - while it shouldn't be, Phobos is an essential part of 
the compiler test suite.

1. the compiler test suite is inadequate
2. breaking Phobos is a good predictor of what will break in user code
May 19 2019
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 19 May 2019 at 03:33:15 UTC, Andrei Alexandrescu wrote:

 https://news.ycombinator.com/item?id=19948642


Thanks for sharing. I think the hacker news comments were nearly 
as interesting as the article itself. It's a hard problem 
figuring out what should go in the standard library or not and 
there's probably n one right answer.
May 21 2019
parent reply David Gileadi <gileadisNOSPM gmail.com> writes:
On 5/21/19 2:14 AM, Atila Neves wrote:

 On Sunday, 19 May 2019 at 03:33:15 UTC, Andrei Alexandrescu wrote:

 https://news.ycombinator.com/item?id=19948642


 
 Thanks for sharing. I think the hacker news comments were nearly as 
 interesting as the article itself. It's a hard problem figuring out what 
 should go in the standard library or not and there's probably n one 
 right answer.


I didn't know this about Rust, so I found the following comment interesting:


 For example, the libstd doesn't even have support for random number
generation. There's a rand crate, which is now on 6th major breaking version.
That's perfectly fine, because multiple versions can coexist in one program,
and every user can upgrade (or not) at their own pace. And the crate was able
to refine its interface six times, instead of being stuck with the first try
forever.


Multiple coexisting versions is an interesting way of solving dependency 
hell.
May 21 2019
parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/21/19 11:15 AM, David Gileadi wrote:

 On 5/21/19 2:14 AM, Atila Neves wrote:

 For example, the libstd doesn't even have support for random number 
 generation. There's a rand crate, which is now on 6th major breaking 
 version. That's perfectly fine, because multiple versions can coexist 
 in one program, and every user can upgrade (or not) at their own pace. 
 And the crate was able to refine its interface six times, instead of 
 being stuck with the first try forever.


 
 Multiple coexisting versions is an interesting way of solving dependency 
 hell.


In terms of multiple simultaneously *installed* versions, I consider it 
an absolutely essential feature of any *good* package manager (This is a 
big part of why I hate most system-level Linux package managers, 
including the one on my own machine). But the idea of multiple versions 
*within one program* is indeed interesting.

Doesn't strike me as something you'd want to frequently take advantage 
of, but having it there as a viable workaround for when problems do 
arise could certainly be quite nice.
May 21 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, May 21, 2019 at 12:29:55PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:

 On 5/21/19 11:15 AM, David Gileadi wrote:

 On 5/21/19 2:14 AM, Atila Neves wrote:

 For example, the libstd doesn't even have support for random
 number generation. There's a rand crate, which is now on 6th major
 breaking version. That's perfectly fine, because multiple versions
 can coexist in one program, and every user can upgrade (or not) at
 their own pace. And the crate was able to refine its interface six
 times, instead of being stuck with the first try forever.


 
 Multiple coexisting versions is an interesting way of solving
 dependency hell.


 
 In terms of multiple simultaneously *installed* versions, I consider
 it an absolutely essential feature of any *good* package manager (This
 is a big part of why I hate most system-level Linux package managers,
 including the one on my own machine). But the idea of multiple
 versions *within one program* is indeed interesting.
 
 Doesn't strike me as something you'd want to frequently take advantage
 of, but having it there as a viable workaround for when problems do
 arise could certainly be quite nice.


This is a very interesting idea indeed.  Goes along well with Andrei's
recent idea of addition vs. replacement.

And I think it's probably not hard to adopt in D (should we deem it a
good idea -- that's still arguable): all you need is for module
declarations to come with an attached version number, then include that
in the mangling. This automatically gives a unique identifier to symbols
in the module, so you could import multiple versions of the same module
and have the symbols resolve correctly. Just a minor change in module
declarations and in module name mangling.

Of course, how to handle imports that may have multiple satisfying
versions is another, stickier, question.


T

-- 
Meat: euphemism for dead animal. -- Flora
May 21 2019
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 21 May 2019 09:39:29 -0700 schrieb H. S. Teoh:


 On Tue, May 21, 2019 at 12:29:55PM -0400, Nick Sabalausky (Abscissa) via

 In terms of multiple simultaneously *installed* versions, I consider it
 an absolutely essential feature of any *good* package manager (This is
 a big part of why I hate most system-level Linux package managers,
 including the one on my own machine). But the idea of multiple versions
 *within one program* is indeed interesting.
 
 Doesn't strike me as something you'd want to frequently take advantage
 of, but having it there as a viable workaround for when problems do
 arise could certainly be quite nice.


 
 This is a very interesting idea indeed.  Goes along well with Andrei's
 recent idea of addition vs. replacement.
 
 And I think it's probably not hard to adopt in D (should we deem it a
 good idea -- that's still arguable):


Having multiple versions of libraries only works well though if there are 
no dependency cycles between third party modules. Random has a very 
'closed' API, so for that it works well.

As another extreme example, consider we have six different versions for 
std.range. Then we have std.file which provides a v1 range, std.format 
which only formats v3 ranges, vibe.d which still uses v2 and your 
applications which want's to combine all these and pass ranges from 
std.file to vibe.d and std.format.

-- 
Johannes
May 21 2019
parent reply sarn <sarn theartofmachinery.com> writes:
On Tuesday, 21 May 2019 at 18:06:44 UTC, Johannes Pfau wrote:

 Having multiple versions of libraries only works well though if 
 there are no dependency cycles between third party modules. 
 Random has a very 'closed' API, so for that it works well.


In the JavaScript world, npm doesn't have that problem because it 
gives each dependency its own dependency tree. (If D tried that, 
the ABI compatibility issues would be painful, and the embedded 
developers would weep at the bloat.)
May 21 2019
next sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/21/19 6:30 PM, sarn wrote:

 On Tuesday, 21 May 2019 at 18:06:44 UTC, Johannes Pfau wrote:

 Having multiple versions of libraries only works well though if there 
 are no dependency cycles between third party modules. Random has a 
 very 'closed' API, so for that it works well.


 
 In the JavaScript world, npm doesn't have that problem because it gives 
 each dependency its own dependency tree. (If D tried that, the ABI 
 compatibility issues would be painful, and the embedded developers would 
 weep at the bloat.)


Wow, that actually explains quite a lot about the modern web.
May 21 2019
prev sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Tuesday, 21 May 2019 at 22:30:17 UTC, sarn wrote:

 On Tuesday, 21 May 2019 at 18:06:44 UTC, Johannes Pfau wrote:

 Having multiple versions of libraries only works well though 
 if there are no dependency cycles between third party modules. 
 Random has a very 'closed' API, so for that it works well.


 In the JavaScript world, npm doesn't have that problem because 
 it gives each dependency its own dependency tree. (If D tried 
 that, the ABI compatibility issues would be painful, and the 
 embedded developers would weep at the bloat.)


And that's why a simple vue.js project on my machine gives a 
node_modules directory that's crowded as a termite mound ...

Please no ...

/P
May 22 2019
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 22 May 2019 at 08:00:15 UTC, Paolo Invernizzi wrote:

 And that's why a simple vue.js project on my machine gives a 
 node_modules directory that's crowded as a termite mound ...

 Please no ...


Plain npm  usage does not seem to pull in two versions of the 
same library.

What is common is to have one module for a single function. 
That's what the directory bloat comes from. So node_modules look 
more crowded than it actually is. A lot of those functions belong 
in a standard library though.

For instance Angular pulls in:

nice-try: call a function and discard exceptions
once: throw an error if the function is called twice
onetime: a function returns a cached value if called many times
extend: merge two objects
extend-shallow: merge in properties from other object

And so on…
May 22 2019
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 22 May 2019 at 09:16:28 UTC, Ola Fosheim Grøstad 
wrote:

 Plain npm  usage does not seem to pull in two versions of the 
 same library.


npm allows you to specify a semver range, so if the range does 
not overlap then it will pull in a "duplicate". Although for 
libraries that are exposed externally (like react), you can 
specify that it is a peer dependency that should only exists as 
one semver version. In which case it should complain.

At least, that is my shallow understanding of the system.

Allowing version 1 and 2 of a library to coexist in the same 
build is no different than depending on two independent libraries 
that provides the same functionality under different names.

Makes sense to allow a build to succeed if two modules depends on 
two different versions of some simple set of functions. 
Especially in a culture where there tends to be one function per 
module.
May 22 2019
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 21 May 2019 at 16:39:29 UTC, H. S. Teoh wrote:

 On Tue, May 21, 2019 at 12:29:55PM -0400, Nick Sabalausky 
 (Abscissa) via Digitalmars-d wrote:

 [...]


 This is a very interesting idea indeed.  Goes along well with 
 Andrei's recent idea of addition vs. replacement.

 And I think it's probably not hard to adopt in D (should we 
 deem it a good idea -- that's still arguable): all you need is 
 for module declarations to come with an attached version 
 number, then include that in the mangling. This automatically 
 gives a unique identifier to symbols in the module, so you 
 could import multiple versions of the same module and have the 
 symbols resolve correctly. Just a minor change in module 
 declarations and in module name mangling.

 Of course, how to handle imports that may have multiple 
 satisfying versions is another, stickier, question.


 T


Also, type versioning. Types from different versions would have 
to be completely separate, which would break function signatures, 
overloads, introspection etc.
May 22 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 22, 2019 at 10:55:09AM +0000, John Colvin via Digitalmars-d wrote:

 On Tuesday, 21 May 2019 at 16:39:29 UTC, H. S. Teoh wrote:

 On Tue, May 21, 2019 at 12:29:55PM -0400, Nick Sabalausky (Abscissa) via
 Digitalmars-d wrote:

 [...]


 
 This is a very interesting idea indeed.  Goes along well with
 Andrei's recent idea of addition vs. replacement.
 
 And I think it's probably not hard to adopt in D (should we deem it
 a good idea -- that's still arguable): all you need is for module
 declarations to come with an attached version number, then include
 that in the mangling. This automatically gives a unique identifier
 to symbols in the module, so you could import multiple versions of
 the same module and have the symbols resolve correctly. Just a minor
 change in module declarations and in module name mangling.
 
 Of course, how to handle imports that may have multiple satisfying
 versions is another, stickier, question.




[...]

 Also, type versioning. Types from different versions would have to be
 completely separate, which would break function signatures, overloads,
 introspection etc.


It doesn't break, because symbols in D are qualified by module name. So
if the module name contains version information, types from different
versions will be differently-named. For example, moduleA.ver1_0.myStruct
will be distinct from moduleA.ver2_0.myStruct, so you couldn't mix the
two together unintentionally without a compile error.


T

-- 
If it breaks, you get to keep both pieces. -- Software disclaimer notice
May 22 2019