www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - post on using go 1.5 and GC latency

reply "Laeeth Isharc" <laeeth nospamlaeeth.com> writes:
https://medium.com/ robin.verlangen/billions-of-request-per-day-meet-go-1-5-362bfefa0911

We then started analyzing the behavior of our Go application. On 
average the application spent ~ 2ms per request, which was great! 
It gave us 98 milliseconds to spare for network overhead, SSL 
handshake, DNS lookups and everything else that makes the 
internet work.

Unfortunately the standard deviation of the latency was high, 
about 100 milliseconds. Meeting our SLA became a major gamble. 
With the “runtime” package of Go we started profiling the entire 
application and found out that garbage collection was the cause, 
resulting in 95-percentile latencies of 279 milliseconds…

We decided to rewrite big chunks of the application to generate 
minimal or no garbage at all. This effectively helped reduce the 
interval at which garbage collection froze the rest of 
application to do its cleanup magic. But we were still having 
issues, so we decided to add more nodes to stay within our SLA. 
With over 80K requests per second at peak times, even minimal 
garbage can become a serious issue.

...

Yesterday evening (19 August), the moment had finally arrived. A 
stable version 1.5 of Go was released, claiming:

The “stop the world” phase of the collector will almost always be 
under 10 milliseconds and usually much less.
Just a few hours after the release we rebuilt the application 
with the new version of Go 1.5 and ran our unit and functional 
tests; they all passed. It seemed too good to be true, so we put 
some effort in manually verifying the functionality. After a few 
hours we decided it was safe to release it to a single production 
node.
Aug 21 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
Yes, Go has sacrificed some compute performance in favour of 
latency and convenience. They have also released GC improvement 
plans for 1.6:

https://docs.google.com/document/d/1kBx98ulj5V5M9Zdeamy7v6ofZXX3yPziAf0V27A64Mo/edit

It is rather obvious that  a building a good concurrent GC is a 
time consuming effort.
Aug 21 2015
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Fri, 2015-08-21 at 10:47 +0000, via Digitalmars-d-learn wrote:

 Yes, Go has sacrificed some compute performance in favour of=20
 latency and convenience. They have also released GC improvement=20
 plans for 1.6:
=20
 https://docs.google.com/document/d/1kBx98ulj5V5M9Zdeamy7v6ofZXX3yPziA
 f0V27A64Mo/edit
=20
 It is rather obvious that  a building a good concurrent GC is a=20
 time consuming effort.


But one that Google are entirely happy to fully fund.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Aug 21 2015
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 22 August 2015 at 06:48:48 UTC, Russel Winder wrote:

 But one that Google are entirely happy to fully fund.


Yes, they have made Go fully supported on Google Cloud now, so I 
think it is safe to say that Google management is backing Go 
fully.

I'm kinda hoping for Go++...
Aug 21 2015
next sibling parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Sat, 2015-08-22 at 06:54 +0000, via Digitalmars-d-learn wrote:

 On Saturday, 22 August 2015 at 06:48:48 UTC, Russel Winder wrote:

 But one that Google are entirely happy to fully fund.


=20
 Yes, they have made Go fully supported on Google Cloud now, so I=20
 think it is safe to say that Google management is backing Go=20
 fully.
=20
 I'm kinda hoping for Go++...


I think Go 2 is a long way off, and even then generics will not be part
of the plan.

Go UK 2015 was held yesterday. It was less a conference and more a
Google "rah rah" event. It was though very clear that Google are
looking for new idioms and practices to come from users other than
Google, rather than what has happened to date, which is the Go central
team dictating everything to everyone else.=20

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Aug 22 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 22 August 2015 at 07:02:40 UTC, Russel Winder wrote:

 I think Go 2 is a long way off, and even then generics will not 
 be part of the plan.


I agree that Go from Google will stay close to the ideals of the 
creators. I think it would be difficult get beyond that for 
social reasons.

But I think the mechanics Go provides are generic enough that 
someone could build a transpiler providing more high level 
convenience. I am thinking along the lines of a convenient 
language that can compile to both Go and Javascript... I'm 
tempted to have a go at it. ;)


 Go UK 2015 was held yesterday. It was less a conference and 
 more a Google "rah rah" event. It was though very clear that 
 Google are looking for new idioms and practices to come from 
 users other than Google, rather than what has happened to date, 
 which is the Go central team dictating everything to everyone 
 else.


Go UK sounds interesting. I wonder if they will have one in Oslo? 
Probably not :-/.
Aug 22 2015
prev sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Saturday, 22 August 2015 at 06:54:43 UTC, Ola Fosheim Grøstad 
wrote:

 On Saturday, 22 August 2015 at 06:48:48 UTC, Russel Winder 
 wrote:

 But one that Google are entirely happy to fully fund.


 Yes, they have made Go fully supported on Google Cloud now, so 
 I think it is safe to say that Google management is backing Go 
 fully.

 I'm kinda hoping for Go++...


The other day I thought it'd be hilarious if I did a Bjarne and 
wrote a preprocessor to generate Go code that would accept a 
superset of Go syntax but added generics, function overloading, 
etc. And, of course, called it Go++.

Alas, 'tis too much work for just the lulz. I'd rather spend the 
time making D better.

Atila
Aug 23 2015
prev sibling parent reply "rsw0x" <anonymous anonymous.com> writes:
On Saturday, 22 August 2015 at 06:48:48 UTC, Russel Winder wrote:

 On Fri, 2015-08-21 at 10:47 +0000, via Digitalmars-d-learn 
 wrote:

 Yes, Go has sacrificed some compute performance in favour of 
 latency and convenience. They have also released GC 
 improvement plans for 1.6:
 
 https://docs.google.com/document/d/1kBx98ulj5V5M9Zdeamy7v6ofZXX3yPziA
f0V27A64Mo/edit
 
 It is rather obvious that  a building a good concurrent GC is 
 a time consuming effort.


 But one that Google are entirely happy to fully fund.


because Go is not a general purpose language.
A concurrent GC for D would kill D. Go programs saw a 25-50% 
performance decrease across the board for the lower latencies.

D could make some very minor changes and be capable of a 
per-thread GC with none of these performance drawbacks, but 
nobody seems very interested in it.
Aug 22 2015
next sibling parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Sat, 2015-08-22 at 07:30 +0000, rsw0x via Digitalmars-d-learn wrote:

 [=E2=80=A6]
=20
 because Go is not a general purpose language.


Not entirely true. Go is a general purpose language, it is a successor
to C as envisioned by Rob Pike, Russ Cox, and others (I am not sure how
much input Brian Kernighan has had). However, because of current
traction in Web servers and general networking, it is clear that that
is where the bulk of the libraries are. Canonical also use it for Qt UI
applications. I am not sure of Google real intent for Go on Android,
but there is one.


 A concurrent GC for D would kill D. Go programs saw a 25-50%=20
 performance decrease across the board for the lower latencies.


They also saw a 100% increase in performance when it was rewritten, and
a 20% fall with this latest rewrite. I anticipate great improvement for
the 1.6 rewrite.  I am surprised they are retaining having only a
single garbage collector: different usages generally require different
garbage collection strategies. Having said that Java is moving from
having four collectors, to having one, it is going to be interesting to
see if G1 meets the needs of all JVM usages.=20


 D could make some very minor changes and be capable of a=20
 per-thread GC with none of these performance drawbacks, but=20
 nobody seems very interested in it.


Until some organization properly funds a suite of garbage collectors
for different performance targets, you have what there is.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Aug 22 2015
next sibling parent reply "rsw0x" <anonymous anonymous.com> writes:
On Saturday, 22 August 2015 at 09:16:32 UTC, Russel Winder wrote:

 On Sat, 2015-08-22 at 07:30 +0000, rsw0x via 
 Digitalmars-d-learn wrote:

 [...]


 Not entirely true. Go is a general purpose language, it is a 
 successor to C as envisioned by Rob Pike, Russ Cox, and others 
 (I am not sure how much input Brian Kernighan has had). 
 However, because of current traction in Web servers and general 
 networking, it is clear that that is where the bulk of the 
 libraries are. Canonical also use it for Qt UI applications. I 
 am not sure of Google real intent for Go on Android, but there 
 is one.


 [...]


 They also saw a 100% increase in performance when it was 
 rewritten, and a 20% fall with this latest rewrite. I 
 anticipate great improvement for the 1.6 rewrite.  I am 
 surprised they are retaining having only a single garbage 
 collector: different usages generally require different garbage 
 collection strategies. Having said that Java is moving from 
 having four collectors, to having one, it is going to be 
 interesting to see if G1 meets the needs of all JVM usages.


 [...]


 Until some organization properly funds a suite of garbage 
 collectors for different performance targets, you have what 
 there is.


The performance decrease has been there since 1.4 and there is no 
way to remove it - write barriers are the cost you pay for 
concurrent collection. Go was already much slower than other 
compiled languages, now it probably struggles to keep up with 
mono.
Aug 22 2015
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Sat, 2015-08-22 at 09:27 +0000, rsw0x via Digitalmars-d-learn wrote:

 [=E2=80=A6]
=20
 The performance decrease has been there since 1.4 and there is no=20
 way to remove it - write barriers are the cost you pay for=20
 concurrent collection. Go was already much slower than other=20
 compiled languages, now it probably struggles to keep up with=20
 mono.


I know Walter hates it when people mention the word but: benchmarks.

As soon as someone say things like "it probably struggles to keep up
with mono" further discussion of the topic is probably not worth
entertaining without getting some agreed codes and running them all on
the same machine.

I agree the standard Go compiler generates not well optimized code, but
gccgo generally does, and generally performs at C-level speeds. Of
course Java often performs far better than that, and often fails to.
You have to be careful with benchmarking and performance things
generally.
=20
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Aug 23 2015
parent reply "rsw0x" <anonymous anonymous.com> writes:
On Sunday, 23 August 2015 at 11:06:20 UTC, Russel Winder wrote:

 On Sat, 2015-08-22 at 09:27 +0000, rsw0x via 
 Digitalmars-d-learn wrote:

 […]
 
 The performance decrease has been there since 1.4 and there is 
 no way to remove it - write barriers are the cost you pay for 
 concurrent collection. Go was already much slower than other 
 compiled languages, now it probably struggles to keep up with 
 mono.


 I know Walter hates it when people mention the word but: 
 benchmarks.

 As soon as someone say things like "it probably struggles to 
 keep up with mono" further discussion of the topic is probably 
 not worth entertaining without getting some agreed codes and 
 running them all on the same machine.

 I agree the standard Go compiler generates not well optimized 
 code, but
 gccgo generally does, and generally performs at C-level speeds. 
 Of
 course Java often performs far better than that, and often 
 fails to.
 You have to be careful with benchmarking and performance things
 generally.


https://groups.google.com/forum/#!msg/golang-dev/pIuOcqAlvKU/C0wooVzXLZwJ
25-50% performance decrease across the board in 1.4 with the 
addition of write barriers, to an already slow language.

random benchmarks of Go performing 3x(+) slower than C/C++/D, 
some of these predate Go 1.4.
https://github.com/kostya/benchmarks
https://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=go&lang2=gcc&data=u64
https://togototo.wordpress.com/2013/07/23/benchmarking-level-generation-go
rust-haskell-and-d/ (gcc-go performed the _worst_)
https://togototo.wordpress.com/2013/08/23/benchmarks-round-two-parallel-go-rust
d-scala-and-nimrod/ (and again)
https://github.com/logicchains/LPATHBench/blob/master/writeup.md 
(once again, Go is nowhere near C/C++/D/Rust. Where is it? 


Go is slow. These aren't cherrypicked, just random samples from a 
quick Googling.
Where is Go performing "C-level speeds" at? D claims this, and D 
shows it does. Go falls into the "fast enough" category, because 
it is _not_ a general purpose programming language. So unless 
multiple randomly sampled benchmarks are all wrong, I'm going to 
stick with 'Go is slow.'
Aug 23 2015
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Sun, 2015-08-23 at 11:26 +0000, rsw0x via Digitalmars-d-learn wrote:

 [=E2=80=A6]
=20
 https://groups.google.com/forum/#!msg/golang
 -dev/pIuOcqAlvKU/C0wooVzXLZwJ
 25-50% performance decrease across the board in 1.4 with the=20
 addition of write barriers, to an already slow language.


Garbage collection is a hard problem for performance oriented code. I
am sure someone will come up with a way of improving the Go one.
"Across the board" though does worry me, my codes never get a sweep.
D's garbage collector could also do with some work.


 random benchmarks of Go performing 3x(+) slower than C/C++/D,=20
 some of these predate Go 1.4.
 https://github.com/kostya/benchmarks
 https://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=3Dall&l
 ang=3Dgo&lang2=3Dgcc&data=3Du64
 https://togototo.wordpress.com/2013/07/23/benchmarking-level
 -generation-go-rust-haskell-and-d/ (gcc-go performed the _worst_)
 https://togototo.wordpress.com/2013/08/23/benchmarks-round-two
 -parallel-go-rust-d-scala-and-nimrod/ (and again)
 https://github.com/logicchains/LPATHBench/blob/master/writeup.md=20
 (once again, Go is nowhere near C/C++/D/Rust. Where is it?=20



Thanks for those pointers, I shall have a look at them. Sadly though
not for a couple of weeks due to various commitments.


 Go is slow. These aren't cherrypicked, just random samples from a=20
 quick Googling.


But why use absolutes. Go may be slow for you in your context, but that
doesn't mean the observation applies everywhere (note the interesting
turn of phrase). For my (admittedly small) codes that do not cause a
garbage collect and are basically just a loop, I find things are fine.

So for a =CF=80 approximation sequential code using 64-bit:

C, gcc -O3: 8.847241
C++, gcc -O3: 8.916043
Fortran, gfortran -O3: 8.893000
D, ldc -O -release: 8.722329
D, dmd -O -release: 8.787744
Rust, cargo --release: 8.715818
Go, gccgo: 8.823525
Go, 6g: 8.824643
 Go is definitely not slow there then. Thus Go is not slow.


 Where is Go performing "C-level speeds" at? D claims this, and D=20
 shows it does. Go falls into the "fast enough" category, because=20
 it is _not_ a general purpose programming language. So unless=20
 multiple randomly sampled benchmarks are all wrong, I'm going to=20
 stick with 'Go is slow.'


You are mixing too many factors here. "General purpose" has nothing to
do with performance, it is to do with can the language describe most if
not all forms of computation. Go is a general purpose programming
language just like C, C++, D, Rust, Haskell, OCaml.

If for you Go is not performant enough, that is fine. But for many
people in various contexts, Go is more the comparable with other
languages. This is not just "Go is fast enough", but "Go is as fast as
any other option".

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Aug 23 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 23 August 2015 at 12:49:35 UTC, Russel Winder wrote:

 You are mixing too many factors here. "General purpose" has 
 nothing to do with performance, it is to do with can the 
 language describe most if not all forms of computation. Go is a 
 general purpose programming language just like C, C++, D, Rust, 
 Haskell, OCaml.


Yes, of course it is, but given it's typical use context I find 
it odd that they didn't go more towards higher level constructs. 
For me Go displaces Python where more speed is required, though I 
wish it was more pythonic… (neither C++, Rust or D are really 
eligible)
Aug 23 2015
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Sun, 2015-08-23 at 19:42 +0000, via Digitalmars-d-learn wrote:

=20


[=E2=80=A6]

 Yes, of course it is, but given it's typical use context I find=20
 it odd that they didn't go more towards higher level constructs.=20
 For me Go displaces Python where more speed is required, though I=20
 wish it was more pythonic=E2=80=A6 (neither C++, Rust or D are really=20
 eligible)


I wondered about the Go/Python issue last year, and even did a London
Python Group session on the topic. Clearly organizations like Canonical
are displacing Python with Go for some of their projects, almost
certainly rightly so: as you say Go has duck typing yet is strongly
statically types, where Python (via PEP 484 and MyPy) is only now
getting type hinting. However I think we will see that Go and Python
are actually getting traction in different spaces and so are not
competing as much as many would have us believe.=20

For Python and native code, D is a great fit, perhaps more so that
Rust, except that Rust is getting more mind share, probably because it
is new. It would be good to get rid of C and C++ as the languages of
Python extensions. Of course systems like Numba change the Python
performance game, which undermines D's potential in the Python-verse,
as it does C and C++. Currently I am investigating Python/Numba/Chapel
as the way of doing performance computing. Anyone who just uses
Python/NumPy/SciPy is probably not doing performance computing, NumPy
is so slow (*).

The issue here for me is that Chapel provides something that C, C++, D,
Rust, Numba, NumPy, cannot =E2=80=93 Partitioned Global Address Space (PGAS=
)
programming. This directly attacks the multicore/multiprocessor/cluster
side of computing, but not the GPGPU side, at least not per se.


(*) Which comment allows this piece to be attacked as I have been
attacking similar comments elsewhere ;-)
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Aug 24 2015
next sibling parent reply "rsw0x" <anonymous anonymous.com> writes:
On Monday, 24 August 2015 at 21:20:39 UTC, Russel Winder wrote:

 For Python and native code, D is a great fit, perhaps more so 
 that Rust, except that Rust is getting more mind share, 
 probably because it is new.


I'm of the opinion that Rust's popularity will quickly die when 
people realize it's a pain to use.
Aug 24 2015
parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Monday, 24 August 2015 at 21:57:41 UTC, rsw0x wrote:

 On Monday, 24 August 2015 at 21:20:39 UTC, Russel Winder wrote:

 For Python and native code, D is a great fit, perhaps more so 
 that Rust, except that Rust is getting more mind share, 
 probably because it is new.


 I'm of the opinion that Rust's popularity will quickly die when 
 people realize it's a pain to use.


Horses for courses ?  Eg for Andy Smith's problem of processing 
trade information of tens of gigs where Python was choking, I 
guess nobody in their right mind would use Rust.  But maybe D 
isn't quite yet what you would choose for a highly complex mass 
consumer product like a browser?

Btw had a nice little test the other day.  I haven't yet done 
much with intraday data - I only just lately took the FX data I 
had sitting around and pulled it into my data server.    Someone 
asked me a question about something in that domain - would have 
taken them a few weeks at least to begin to get an answer.  I 
have never been the fastest programmer, but took a couple hundred 
lines and three hours to write the analysis from scratch, and 
half the time was figuring out how to get it to display nicely in 
Mathgl.  Took less than 10 minutes to run on my home machine 
using debug Dmd and pulling data from my server over the Internet 
with a spinning drive (so probably could get it down to a minute 
or two).  I know it takes a big Wall Street firm an hour to run 
the same task.  That gets in the way of using rapid iteration to 
explore the data.
Aug 24 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 25 August 2015 at 05:09:56 UTC, Laeeth Isharc wrote:

 On Monday, 24 August 2015 at 21:57:41 UTC, rsw0x wrote:

 On Monday, 24 August 2015 at 21:20:39 UTC, Russel Winder wrote:

 For Python and native code, D is a great fit, perhaps more so 
 that Rust, except that Rust is getting more mind share, 
 probably because it is new.


 I'm of the opinion that Rust's popularity will quickly die 
 when people realize it's a pain to use.


 Horses for courses ?  Eg for Andy Smith's problem of processing 
 trade information of tens of gigs where Python was choking, I 
 guess nobody in their right mind would use Rust.


I don't think there is much difference between C, D or Rust in 
terms of computing. The core semantics are similar. With Rust you 
have the additional option of linear type checking.

But Rust programmers of course want to use idiomatic linear 
typing as much as possible and that makes designing graph-like 
structures a challenge.
Aug 25 2015
parent reply "rsw0x" <anonymous anonymous.com> writes:
On Tuesday, 25 August 2015 at 07:18:24 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 25 August 2015 at 05:09:56 UTC, Laeeth Isharc wrote:

 On Monday, 24 August 2015 at 21:57:41 UTC, rsw0x wrote:

 [...]


 Horses for courses ?  Eg for Andy Smith's problem of 
 processing trade information of tens of gigs where Python was 
 choking, I guess nobody in their right mind would use Rust.


 I don't think there is much difference between C, D or Rust in 
 terms of computing. The core semantics are similar. With Rust 
 you have the additional option of linear type checking.

 But Rust programmers of course want to use idiomatic linear 
 typing as much as possible and that makes designing graph-like 
 structures a challenge.


An option implies you can turn it off, has this changed since the 
last time I used Rust?(admittedly, a while back)

Memory safety doesn't seem like it's the top priority for 
scientific computing as much as fast turnarouds and 
performance... in my opinion, anyways.
Aug 25 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 25 August 2015 at 07:21:13 UTC, rsw0x wrote:

 An option implies you can turn it off, has this changed since 
 the last time I used Rust?(admittedly, a while back)


Rust supports other reference types, so you decide by design 
whether you want to use linear typing or not?
Aug 25 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
More info on the Go 1.5 concurrent GC, a classic one:

https://blog.golang.org/go15gc
Sep 01 2015
prev sibling next sibling parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
 . Of course systems like Numba change the Python performance 
 game, which undermines D's potential in the Python-verse, as it 
 does C and C++. Currently I am investigating 
 Python/Numba/Chapel as the way of doing performance computing. 
 Anyone who just uses Python/NumPy/SciPy is probably not doing 
 performance computing, NumPy is so slow (*).


Can you elaborate ?


 The issue here for me is that Chapel provides something that C, 
 C++, D, Rust, Numba, NumPy, cannot – Partitioned Global Address 
 Space (PGAS) programming. This directly attacks the 
 multicore/multiprocessor/cluster side of computing, but not the 
 GPGPU side, at least not per se.


What's the best reference to learn more about PGAS?

Thanks.


Laeeth.
Aug 24 2015
parent "jmh530" <john.michael.hall gmail.com> writes:
On Tuesday, 25 August 2015 at 05:12:55 UTC, Laeeth Isharc wrote:

 What's the best reference to learn more about PGAS?


I've seen a few presentations,
https://www.osc.edu/sites/osc.edu/files/staff_files/dhudak/pgas-tutorial.pdf
http://www.inf.unideb.hu/~fazekasg/english/New_Programming_Paradims/PGAS.pdf
papers,
http://www.cs.rochester.edu/u/cding/amp/papers/full/The%20Asynchronous%20Partitioned%20Global%20Address%20Space%20Model.pdf
and a stackoverflow question
http://stackoverflow.com/questions/31992761/what-are-the-differences-between-tls-and-pgas

Not sure what is the best one.
Sep 01 2015
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 24 August 2015 at 21:20:39 UTC, Russel Winder wrote:

 The issue here for me is that Chapel provides something that C, 
 C++, D, Rust, Numba, NumPy, cannot – Partitioned Global Address 
 Space (PGAS) programming. This directly attacks the 
 multicore/multiprocessor/cluster side of computing, but not the 
 GPGPU side, at least not per se.


Yes, I agree that highly parallel batch programming requires a 
specialized approach.

Though for most applications I think something like Pony will fit 
better. But they have to figure out how to do migration and load 
balancing I suppose.

Javascript is moving towards programming using futures/promises, 
it is already built into Chrome. While that isn't actors per se, 
I think the "computing objects" approach will be more familiar 
for programmers over time and that will open up for actor-based 
languages too. Maybe.
Aug 24 2015
prev sibling parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Saturday, 22 August 2015 at 09:16:32 UTC, Russel Winder wrote:

 On Sat, 2015-08-22 at 07:30 +0000, rsw0x via 
 Digitalmars-d-learn wrote:

 […]
 
 because Go is not a general purpose language.


 Not entirely true. Go is a general purpose language, it is a 
 successor to C as envisioned by Rob Pike, Russ Cox, and others 
 (I am not sure how much input Brian Kernighan has had). 
 However, because of current traction in Web servers and general 
 networking, it is clear that that is where the bulk of the 
 libraries are. Canonical also use it for Qt UI applications. I 
 am not sure of Google real intent for Go on Android, but there 
 is one.


 A concurrent GC for D would kill D. Go programs saw a 25-50% 
 performance decrease across the board for the lower latencies.


 They also saw a 100% increase in performance when it was 
 rewritten, and a 20% fall with this latest rewrite. I 
 anticipate great improvement for the 1.6 rewrite.  I am 
 surprised they are retaining having only a single garbage 
 collector: different usages generally require different garbage 
 collection strategies. Having said that Java is moving from 
 having four collectors, to having one, it is going to be 
 interesting to see if G1 meets the needs of all JVM usages.


 D could make some very minor changes and be capable of a 
 per-thread GC with none of these performance drawbacks, but 
 nobody seems very interested in it.


 Until some organization properly funds a suite of garbage 
 collectors for different performance targets, you have what 
 there is.


I didn't mean to start again the whole GC and Go vs D thing.  
Just that one ought to know the lay of the land as it develops.

Out of curiosity, how much funding is required to develop the 
more straightforward kind of GCs ?  Or to take what's been done 
and  make it possible for others to use?  It needn't be a single 
organisation I would think if there are many that would benefit 
and one doesn't get bogged down in a mentality of people worrying 
about possibly spurious free rider problems.  Since the D 
Foundation seems under way, it seems worth asking the question 
first and thinking about goals without worrying for now about 
what seems realistic.
Aug 22 2015
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/22/2015 10:47 PM, Laeeth Isharc wrote:

 On Saturday, 22 August 2015 at 09:16:32 UTC, Russel Winder wrote:

 On Sat, 2015-08-22 at 07:30 +0000, rsw0x via Digitalmars-d-learn wrote:

 […]

 because Go is not a general purpose language.


 Not entirely true. Go is a general purpose language, it is a successor
 to C as envisioned by Rob Pike, Russ Cox, and others (I am not sure
 how much input Brian Kernighan has had). However, because of current
 traction in Web servers and general networking, it is clear that that
 is where the bulk of the libraries are. Canonical also use it for Qt
 UI applications. I am not sure of Google real intent for Go on
 Android, but there is one.


 A concurrent GC for D would kill D. Go programs saw a 25-50%
 performance decrease across the board for the lower latencies.


 They also saw a 100% increase in performance when it was rewritten,
 and a 20% fall with this latest rewrite. I anticipate great
 improvement for the 1.6 rewrite.  I am surprised they are retaining
 having only a single garbage collector: different usages generally
 require different garbage collection strategies. Having said that Java
 is moving from having four collectors, to having one, it is going to
 be interesting to see if G1 meets the needs of all JVM usages.


 D could make some very minor changes and be capable of a per-thread
 GC with none of these performance drawbacks, but nobody seems very
 interested in it.


 Until some organization properly funds a suite of garbage collectors
 for different performance targets, you have what there is.


 I didn't mean to start again the whole GC and Go vs D thing. Just that
 one ought to know the lay of the land as it develops.

 Out of curiosity, how much funding is required to develop the more
 straightforward kind of GCs ?  Or to take what's been done and  make it
 possible for others to use?  It needn't be a single organisation I would
 think if there are many that would benefit and one doesn't get bogged
 down in a mentality of people worrying about possibly spurious free
 rider problems.  Since the D Foundation seems under way, it seems worth
 asking the question first and thinking about goals without worrying for
 now about what seems realistic.


I believe the hardest part is finding somebody can and willing to work 
on it.
For example I'm willing but I don't know how and there are people 
willing with a job and can do it. But cannot dedicated time because of 
money.

Really it comes down to having a budget and if somebody says hey I'll do 
x, y and z features to pay them for their time as they do it.
Even if they only do one small feature which takes a week.
Aug 22 2015
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 22 August 2015 at 10:47:55 UTC, Laeeth Isharc wrote:

 Out of curiosity, how much funding is required to develop the 
 more straightforward kind of GCs ?


A classical GC like D has is very straightforward. It is been 
used since the 60s, I even have a paper from 1974 or so 
describing the implementation used for Simula which is a precise 
stop-the world GC. Trivial to do.


 Or to take what's been done and  make it possible for others to 
 use?


Therein is the trouble, a more advanced GC is intrinsically 
linked to the language semantics and has to be tuned to the 
hardware. Expect at least 2 years of work for anything 
approaching state-of-the-art.

In the web server space you wait a lot for I/O so raw performance 
is not key for Go's success. Stability, memory usage and low 
latency is more important.
Aug 22 2015
prev sibling parent reply "rsw0x" <anonymous anonymous.com> writes:
On Saturday, 22 August 2015 at 10:47:55 UTC, Laeeth Isharc wrote:

 On Saturday, 22 August 2015 at 09:16:32 UTC, Russel Winder 
 wrote:

 [...]


 I didn't mean to start again the whole GC and Go vs D thing.  
 Just that one ought to know the lay of the land as it develops.

 Out of curiosity, how much funding is required to develop the 
 more straightforward kind of GCs ?  Or to take what's been done 
 and  make it possible for others to use?  It needn't be a 
 single organisation I would think if there are many that would 
 benefit and one doesn't get bogged down in a mentality of 
 people worrying about possibly spurious free rider problems.  
 Since the D Foundation seems under way, it seems worth asking 
 the question first and thinking about goals without worrying 
 for now about what seems realistic.


The problem with D's GC is that there's no scaffolding there for 
it, so you can't really improve it.
At best you could make the collector parallel.

If I had the runtime hooks and language guarantees I needed I'd 
begin work on a per-thread GC immediately.
Aug 22 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 22 August 2015 at 12:48:31 UTC, rsw0x wrote:

 The problem with D's GC is that there's no scaffolding there 
 for it, so you can't really improve it.
 At best you could make the collector parallel.

 If I had the runtime hooks and language guarantees I needed I'd 
 begin work on a per-thread GC immediately.


If you had a fiber local reference type and some guarantees 
related to that, you probably could do a per-fiber GC and collect 
when fibers are waiting.
Aug 23 2015
prev sibling parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Saturday, 22 August 2015 at 07:30:23 UTC, rsw0x wrote:

 On Saturday, 22 August 2015 at 06:48:48 UTC, Russel Winder 
 wrote:

 On Fri, 2015-08-21 at 10:47 +0000, via Digitalmars-d-learn 
 wrote:

 Yes, Go has sacrificed some compute performance in favour of 
 latency and convenience. They have also released GC 
 improvement plans for 1.6:
 
 https://docs.google.com/document/d/1kBx98ulj5V5M9Zdeamy7v6ofZXX3yPziA
f0V27A64Mo/edit
 
 It is rather obvious that  a building a good concurrent GC is 
 a time consuming effort.


 But one that Google are entirely happy to fully fund.


 because Go is not a general purpose language.
 A concurrent GC for D would kill D. Go programs saw a 25-50% 
 performance decrease across the board for the lower latencies.

 D could make some very minor changes and be capable of a 
 per-thread GC with none of these performance drawbacks, but 
 nobody seems very interested in it.


This puts ddmd into context, bearing in mind an automated 
translation without won't I guess be much slower in LDC or GDC, 
and it's already a small difference:

Release notes on go 1.5 via stack overflow.

Builds in Go 1.5 will be slower by a factor of about two. The 
automatic translation of the compiler and linker from C to Go 
resulted in unidiomatic Go code that performs poorly compared to 
well-written Go. Analysis tools and refactoring helped to improve 
the code, but much remains to be done. Further profiling and 
optimization will continue in Go 1.6 and future releases. For 
more details, see these slides and associated video.
Aug 22 2015
parent Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Sat, 2015-08-22 at 11:06 +0000, Laeeth Isharc via Digitalmars-d
-learn wrote:

 [=E2=80=A6]
=20
 Builds in Go 1.5 will be slower by a factor of about two. The=20
 automatic translation of the compiler and linker from C to Go=20
 resulted in unidiomatic Go code that performs poorly compared to=20
 well-written Go. Analysis tools and refactoring helped to improve=20
 the code, but much remains to be done. Further profiling and=20
 optimization will continue in Go 1.6 and future releases. For=20
 more details, see these slides and associated video.
=20


This is about compiler performance, not about generated code
performance.

Anyone interested in performance with Go currently uses gccgo: the
standard Go compiler does not generate particularly well optimized
code. This has been a resourcing choice to date, it is not a failing.
gccgo on the other hand makes use of the whole GCC optimization chain.=20

On the other hand gc is blindingly fast at compilation compared to
gccgo. This seems reminiscent of dmd vs. ldc and gdc!

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Aug 23 2015