www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Recommendation on plotting library

reply Chris Piker <chris hoopjump.com> writes:
Hi D

One of my jobs is to release and maintain public data archives 
from long-running scientific instruments.  In order to help 
people understand how to process the data, sample code is often 
included with the archive. Recently this has been in the form of 
short programs that generate a plot of a representative time 
period.  Python works well for this task as matplotlib support is 
pretty much universal.  Matlab is also a reasonable choice for 
many.

Were I to also provide sample code in D, what library would you 
recommend for scientific plotting, especially for dynamic power 
spectral densities generated from time-series data (basically 
heatmaps)?

Since dub can pull down dependencies easily enough and since it 
supports single file projects it seems that D would be well 
suited to this task, but I've never plotted any data directly 
from a D program.  Many GUI libraries I see for D are wrappers 
around external C or C++ based libraries that would need to be 
installed separately.  I'd like to avoid such complications if 
that's possible.

Thanks for any advice,
Jul 19 2023
next sibling parent reply Andrew <andrewlalisofficial gmail.com> writes:
I was using D for some of the computations in my Bachelor's 
thesis paper, and I also looked into plotting in D. My conclusion 
at the time was there aren't yet any serious plotting libraries 
comparable to what's available in more popular languages, so I 
exported the data in json and rendered plots in Java. If you're 
already using python, it's probably best to keep using that.
Jul 19 2023
parent reply Chris Piker <chris hoopjump.com> writes:
On Thursday, 20 July 2023 at 03:58:05 UTC, Andrew wrote:

 If you're already using python, it's probably best to keep 
 using that.


Oh of course.  Examples *have* to be provided in python, since 
that's the default language of science these days.  But extra 
examples don't hurt, and it would be nice to see D along side 
other options.  I find these days that my D "scripts" are about 
the same size as my python scripts, which is really surprising 
for a compiled language.

I just tried ggplotd and it was easy to make it work on Linux, 
only one external apt command needed, but on Windows, even that 
is a deal breaker.  Package management on Windows seems to be 
wild-west/nonexistent.

Side musing... Since I've seen many people use anaconda on 
Windows, I wonder how hard it would be to make a conda package 
that provided dmd+dub?
Jul 19 2023
next sibling parent anonymouse <anony mouse.com> writes:
On Thursday, 20 July 2023 at 04:41:48 UTC, Chris Piker wrote:

 On Thursday, 20 July 2023 at 03:58:05 UTC, Andrew wrote:
 I just tried ggplotd and it was easy to make it work on Linux, 
 only one external apt command needed, but on Windows, even that 
 is a deal breaker.  Package management on Windows seems to be 
 wild-west/nonexistent.


Have you tried https://github.com/koji-kojiro/matplotlib-d? There 
is a slightly more updated fork out there but I can't remember 
exactly who maintains it.


 Side musing... Since I've seen many people use anaconda on 
 Windows, I wonder how hard it would be to make a conda package 
 that provided dmd+dub?


Have you tried chocolatey.org? It's similar to homebrew for macOS 
but for Windows.
Jul 19 2023
prev sibling parent Ruby The Roobster <michaeleverestc79 gmail.com> writes:
On Thursday, 20 July 2023 at 04:41:48 UTC, Chris Piker wrote:

 [SNIP]

 I just tried ggplotd and it was easy to make it work on Linux, 
 only one external apt command needed, but on Windows, even that 
 is a deal breaker.  Package management on Windows seems to be 
 wild-west/nonexistent.


Try MinGW.  There should be SOME way to use GTK with it, though I 
haven't been able to get it to work.


 [SNIP]
Jul 21 2023
prev sibling next sibling parent drug007 <drug2004 bk.ru> writes:
20.07.2023 05:37, Chris Piker пишет:

 Hi D
 
 One of my jobs is to release and maintain public data archives from 
 long-running scientific instruments.  In order to help people understand 
 how to process the data, sample code is often included with the archive. 
 Recently this has been in the form of short programs that generate a 
 plot of a representative time period.  Python works well for this task 
 as matplotlib support is pretty much universal.  Matlab is also a 
 reasonable choice for many.
 
 Were I to also provide sample code in D, what library would you 
 recommend for scientific plotting, especially for dynamic power spectral 
 densities generated from time-series data (basically heatmaps)?
 
 Since dub can pull down dependencies easily enough and since it supports 
 single file projects it seems that D would be well suited to this task, 
 but I've never plotted any data directly from a D program.  Many GUI 
 libraries I see for D are wrappers around external C or C++ based 
 libraries that would need to be installed separately.  I'd like to avoid 
 such complications if that's possible.
 
 Thanks for any advice,


https://code.dlang.org/packages/ggplotd
Jul 20 2023
prev sibling next sibling parent reply harakim <harakim gmail.com> writes:
On Thursday, 20 July 2023 at 02:37:54 UTC, Chris Piker wrote:

 Hi D

 One of my jobs is to release and maintain public data archives 
 from long-running scientific instruments.  In order to help 
 people understand how to process the data, sample code is often 
 included with the archive. Recently this has been in the form 
 of short programs that generate a plot of a representative time 
 period.  Python works well for this task as matplotlib support 
 is pretty much universal.  Matlab is also a reasonable choice 
 for many.

 Were I to also provide sample code in D, what library would you 
 recommend for scientific plotting, especially for dynamic power 
 spectral densities generated from time-series data (basically 
 heatmaps)?

 Since dub can pull down dependencies easily enough and since it 
 supports single file projects it seems that D would be well 
 suited to this task, but I've never plotted any data directly 
 from a D program.  Many GUI libraries I see for D are wrappers 
 around external C or C++ based libraries that would need to be 
 installed separately.  I'd like to avoid such complications if 
 that's possible.

 Thanks for any advice,


If you happen upon a basic charting library for D during this 
hunt, please let me know! Last year, I rolled my own and it got 
the job done, but I wasn't concerned about how they looked. :D  
As for plotting, I imagine it's c bindings only because there's 
no need to re-invent the wheel.
Jul 20 2023
parent reply Chris Piker <chris hoopjump.com> writes:
On Friday, 21 July 2023 at 02:40:10 UTC, harakim wrote:

 On Thursday, 20 July 2023 at 02:37:54 UTC, Chris Piker wrote:



 If you happen upon a basic charting library for D during this 
 hunt, please let me know! Last year, I rolled my own and it got 
 the job done, but I wasn't concerned about how they looked. :D


Well, just for fun, if you have a plotting library on github or 
some other public place I wouldn't mind taking a look.  Given the 
state of things, it doesn't have to be awesome to be a good seed 
point.


 As for plotting, I imagine it's c bindings only because there's 
 no need to re-invent the wheel.


(Warning, possible ill-informed opinions ahead...)

In a way there is a need to reinvent the wheel.  With python I 
can run `pip install matplotlib` and get whatever binaries I need 
to get the job done.  D runs on dub, which I only see handling 
source code, and as far as I can tell, only D source code at 
that.  So unless it's D code, it can't be packaged and delivered 
easily within the D ecosystem.

If dub supports either pre-built binaries, or C code (such as 
libcairo2), I'd be interested in seeing how that's done.  With 
the wizardry I've see around here, it's probably easy, I just 
don't know about it.

Going waaaay out on a limb for a minute, I think D shines as a 
scripting language replacement.  Most of my programs are single 
file projects these days with dub set as the interpreter.  Also 
Rust seems to be crowding the system level space and so focusing 
on it's "compiled scripts" capability avoids that competition.

(If any of the statements above are faulty, I invite correction.)
Jul 20 2023
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, July 20, 2023 10:57:22 PM MDT Chris Piker via Digitalmars-d-learn 
wrote:

 (Warning, possible ill-informed opinions ahead...)

 In a way there is a need to reinvent the wheel.  With python I
 can run `pip install matplotlib` and get whatever binaries I need
 to get the job done.  D runs on dub, which I only see handling
 source code, and as far as I can tell, only D source code at
 that.  So unless it's D code, it can't be packaged and delivered
 easily within the D ecosystem.

 If dub supports either pre-built binaries, or C code (such as
 libcairo2), I'd be interested in seeing how that's done.  With
 the wizardry I've see around here, it's probably easy, I just
 don't know about it.


Well, dub is certainly designed around building projects that are pure D
without anything fancy going on, but with effort, it's possible to do far
more complicated stuff (though it's certainly far more of a pain than would
be desirable). That being said, if the C libraries are already on your
system, it's trivial to have a dub project just use them via bindings in the
D code. And there are libraries on code.dlang.org which are basically just
bindings for C libraries.

Regardless though, dub really isn't designed with packaging anything in
mind. Rather, it's designed to build your code as well as pull in D
libraries that it usees and build those too. Anyone looking to actually
package stuff would create a package from what was built with dub (e.g. with
deb, rpm, flatpacks, etc.).


 Going waaaay out on a limb for a minute, I think D shines as a
 scripting language replacement.  Most of my programs are single
 file projects these days with dub set as the interpreter.  Also
 Rust seems to be crowding the system level space and so focusing
 on it's "compiled scripts" capability avoids that competition.

 (If any of the statements above are faulty, I invite correction.)


D does work quite well as a scripting language replacement for a lot stuff.
In fact, the compiler and standard library have largely replaced their
Makefiles with D scripts. It's definitely worse if you want to write a
script that needs to pull in dependencies that aren't part of the standard
library, but if Phobos has what you need, it works quite well - and of
course, you can always run other shell commands from within a D program.

- Jonathan M Davis
Jul 20 2023
parent reply Chris Piker <chris hoopjump.com> writes:
On Friday, 21 July 2023 at 06:15:10 UTC, Jonathan M Davis wrote:

 On Thursday, July 20, 2023 10:57:22 PM MDT Chris Piker via 
 Digitalmars-d-learn wrote:



 Regardless though, dub really isn't designed with packaging 
 anything in mind. Rather, it's designed to build your code as 
 well as pull in D libraries that it usees and build those too. 
 Anyone looking to actually package stuff would create a package 
 from what was built with dub (e.g. with deb, rpm, flatpacks, 
 etc.).


So as far as I can tell, python pip originally only dealt with 
python code, but eventually wheels were added for binary support. 
  Just as a wild guess, do you see dub ever evolving in that 
direction?  All the reasons for not supporting pre-compiled 
binaries in pip apply to dub, but yet support was added anyway, 
and it's been wildly successful.

I know it's hard to make predictions (especially about the 
future), but I'd be interesting in your opinion on the matter.
Jul 21 2023
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, July 21, 2023 1:03:47 AM MDT Chris Piker via Digitalmars-d-learn 
wrote:

 On Friday, 21 July 2023 at 06:15:10 UTC, Jonathan M Davis wrote:

 On Thursday, July 20, 2023 10:57:22 PM MDT Chris Piker via
 Digitalmars-d-learn wrote:

 Regardless though, dub really isn't designed with packaging
 anything in mind. Rather, it's designed to build your code as
 well as pull in D libraries that it usees and build those too.
 Anyone looking to actually package stuff would create a package
 from what was built with dub (e.g. with deb, rpm, flatpacks,
 etc.).


 So as far as I can tell, python pip originally only dealt with
 python code, but eventually wheels were added for binary support.
   Just as a wild guess, do you see dub ever evolving in that
 direction?  All the reasons for not supporting pre-compiled
 binaries in pip apply to dub, but yet support was added anyway,
 and it's been wildly successful.

 I know it's hard to make predictions (especially about the
 future), but I'd be interesting in your opinion on the matter.


I'd be very surprised if dub added support for pre-compiled binaries -
particularly since D isn't generally binary compatible across releases - but
I really don't know what the folks working on dub want to do with it.

- Jonathan M Davis
Jul 21 2023
parent reply Greggor <foobar notvalid.nope> writes:
On Friday, 21 July 2023 at 15:12:57 UTC, Jonathan M Davis wrote:

 On Friday, July 21, 2023 1:03:47 AM MDT Chris Piker via 
 Digitalmars-d-learn wrote:

 On Friday, 21 July 2023 at 06:15:10 UTC, Jonathan M Davis 
 wrote:

 On Thursday, July 20, 2023 10:57:22 PM MDT Chris Piker via 
 Digitalmars-d-learn wrote:

 Regardless though, dub really isn't designed with packaging 
 anything in mind. Rather, it's designed to build your code 
 as well as pull in D libraries that it usees and build those 
 too. Anyone looking to actually package stuff would create a 
 package from what was built with dub (e.g. with deb, rpm, 
 flatpacks, etc.).


 So as far as I can tell, python pip originally only dealt with
 python code, but eventually wheels were added for binary 
 support.
   Just as a wild guess, do you see dub ever evolving in that
 direction?  All the reasons for not supporting pre-compiled
 binaries in pip apply to dub, but yet support was added anyway,
 and it's been wildly successful.

 I know it's hard to make predictions (especially about the 
 future), but I'd be interesting in your opinion on the matter.


 I'd be very surprised if dub added support for pre-compiled 
 binaries - particularly since D isn't generally binary 
 compatible across releases - but I really don't know what the 
 folks working on dub want to do with it.

 - Jonathan M Davis


Dependency management sucks for windows and I understand wanting 
the ability to just do dub run and have it “just work tm”.

Up to date versions of Windows 10 should have curl included and 
dub can run commands before building, so you could try 
downloading a prebuilt lib for windows via curl. 
https://everything.curl.dev/get/windows
Jul 21 2023
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, July 21, 2023 11:40:25 AM MDT Greggor via Digitalmars-d-learn 
wrote:

 So as far as I can tell, python pip originally only dealt with
 python code, but eventually wheels were added for binary
 support.

   Just as a wild guess, do you see dub ever evolving in that

 direction?  All the reasons for not supporting pre-compiled
 binaries in pip apply to dub, but yet support was added anyway,
 and it's been wildly successful.

 I know it's hard to make predictions (especially about the
 future), but I'd be interesting in your opinion on the matter.


 I'd be very surprised if dub added support for pre-compiled
 binaries - particularly since D isn't generally binary
 compatible across releases - but I really don't know what the
 folks working on dub want to do with it.

 - Jonathan M Davis


 Dependency management sucks for windows and I understand wanting
 the ability to just do dub run and have it “just work tm”.

 Up to date versions of Windows 10 should have curl included and
 dub can run commands before building, so you could try
 downloading a prebuilt lib for windows via curl.
 https://everything.curl.dev/get/windows


Well, from what I recall (though it's been a while since I messed with
anything like it), it's possible with dub to run more or less arbitrary
stuff (e.g. use cmake from dub). It's a pain, but it can be done. And if
that's the case, then you should be able to design a dub project that pulled
in pretty much whatever you want with curl. And simply being able to build
and run a D program as part of the build would be enough to be able to use
curl in general without Windows having added it, since dmd comes with it
because of std.net.curl. But of course, that's a rather different thing from
dub providing a clean way to handle that sort of thing.

dub can do a lot, but because it's really not designed around doing much
beyond pulling in dependencies from code.dlang.org and then building all of
the code, doing stuff beyond that gets to be problematic even if it can be
done. So, if we wanted something more fully featured, then dub would need a
bit of an overhaul. And maybe dub will get that at some point (I don't
know), but as things stand, if anyone really wants to do complex stuff with
their builds, dub can be pretty awkward to use. And that's why projects such
as https://code.dlang.org/packages/reggae exist, though it's geared more
towards providing a more fully-feature build system than providing a way to
pull in pre-built binaries and the like. What additional features we get in
the future will likely be highly dependent on how motivated the people are
who want dub to be able to do more.

- Jonathan M Davis
Jul 21 2023
parent reply Greggor <Greggor notareal.email> writes:
On Friday, 21 July 2023 at 22:51:16 UTC, Jonathan M Davis wrote:

 On Friday, July 21, 2023 11:40:25 AM MDT Greggor via 
 Digitalmars-d-learn wrote:

 So as far as I can tell, python pip originally only dealt 
 with python code, but eventually wheels were added for 
 binary support.

   Just as a wild guess, do you see dub ever evolving in that

 direction?  All the reasons for not supporting pre-compiled 
 binaries in pip apply to dub, but yet support was added 
 anyway, and it's been wildly successful.

 I know it's hard to make predictions (especially about the 
 future), but I'd be interesting in your opinion on the 
 matter.


 I'd be very surprised if dub added support for pre-compiled 
 binaries - particularly since D isn't generally binary 
 compatible across releases - but I really don't know what 
 the folks working on dub want to do with it.

 - Jonathan M Davis


 Dependency management sucks for windows and I understand 
 wanting the ability to just do dub run and have it “just work 
 tm”.

 Up to date versions of Windows 10 should have curl included 
 and dub can run commands before building, so you could try 
 downloading a prebuilt lib for windows via curl. 
 https://everything.curl.dev/get/windows


 Well, from what I recall (though it's been a while since I 
 messed with anything like it), it's possible with dub to run 
 more or less arbitrary stuff (e.g. use cmake from dub). It's a 
 pain, but it can be done. And if that's the case, then you 
 should be able to design a dub project that pulled in pretty 
 much whatever you want with curl. And simply being able to 
 build and run a D program as part of the build would be enough 
 to be able to use curl in general without Windows having added 
 it, since dmd comes with it because of std.net.curl. But of 
 course, that's a rather different thing from dub providing a 
 clean way to handle that sort of thing.

 dub can do a lot, but because it's really not designed around 
 doing much beyond pulling in dependencies from code.dlang.org 
 and then building all of the code, doing stuff beyond that gets 
 to be problematic even if it can be done. So, if we wanted 
 something more fully featured, then dub would need a bit of an 
 overhaul. And maybe dub will get that at some point (I don't 
 know), but as things stand, if anyone really wants to do 
 complex stuff with their builds, dub can be pretty awkward to 
 use. And that's why projects such as 
 https://code.dlang.org/packages/reggae exist, though it's 
 geared more towards providing a more fully-feature build system 
 than providing a way to pull in pre-built binaries and the 
 like. What additional features we get in the future will likely 
 be highly dependent on how motivated the people are who want 
 dub to be able to do more.

 - Jonathan M Davis


I'll be honest, I actually really don't want this, pip and 
especially prebuilt wheels deps have often made my life 
miserable. On the linux side of things. Pulling in dynamic 
libraries that are not from the distro package manager or freshly 
built locally is simply not a good idea. Things like different 
stdlibC(s) will break even the simplest of things.



Python is actually a good example of what NOT to do, I don't 
program in python so take what I am about to say with a grain of 
salt. But not too long ago I installed onto my machine stable 
diffusion and it was a harrowing process that perfectly 
encapsulates my problems with the python ecosystem. Building the 
deps from source was broken and the prebuilt deps where not 
matching my distro's python version. I tried all sorts of things 
like venv and building as much stuff locally all the other 
"correct" ways and nothing worked.

Eventually I went down a path of super jank and essentially 
frankensteined a working runtime by hand. I built a specific 
version of python from src and installed it to its own folder, I 
used a bunch of LD env vars and used patchelf on deps to 
basically make an isolated install of python and glued together a 
bunch of the native pre build packages, because building them on 
my machine was a pain.

What should of been a simple "pip install X" is now a cargo 
cult-ed install. I dread the day when it will stop working.

Whats even more upsetting is that non of it is some ancient out 
of date packages, it's all popular and maintained stuff. Maybe 
part of the problem is that I when I say "linux", I am not 
referring or implying some version of Ubuntu and that my 
understanding of the word includes distros that use different 
stdlibC(s) and could also not use things like system-D.

Despite the amount of users and active support, I guess a lot of 
headaches related to building are flying under the radar due to 
pre-built packages.



Comparing this with Dub & D is a night and day difference, the 
current ecosystem is very easy to use and has little issues, even 
with our smaller community.

I feel comfortable that anytime I ran into an issue with a 
dependency that making a change is easy. To do so is as simple as 
pulling it down into a folder and then editing a dub.json/.sdl to 
point to the path of the edited version. If it worked before I 
needed to make a change I know it will work after, there is no 
question of "will I be able to build it". Whats also nice about 
this is that fixing an issue locally has a few times turned into 
me making an upstream contribution. (this is really good for a 
smaller community)

In general whenever possible I think its better for everyone that 
stuff is built from source. It ensures that builds can be 
re-produced by anyone and that issues with building are caught 
fast by anyone consuming the library.



While I don't have a ton of experience working on the windows 
side, there is no good package manager & DLL(s) work a bit 
differently and don't have the stdlibc issues or having a bunch 
of different distros to target. It's relatively normal for a 
program to just ship with a couple of DLL(s) along side the main 
executable. Hence my recommendation of the curl hack for windows.

TL;DR I understand the issue trying to solved on the windows side 
and I think the positives are stronger then the negatives in the 
windows case. But I do not want my suggestion to be mistaken for 
general support for pre build binaries being distributed by DUB. 
Out side of this case I am actually very strongly against this 
idea.
Jul 21 2023
parent Chris Piker <chris hoopjump.com> writes:
Thanks for the both of the long replies.  I've ready them twice 
and will do so again.  To focus in on one aspect of D package 
support:

On Saturday, 22 July 2023 at 02:24:08 UTC, Greggor wrote:

 In general whenever possible I think its better for everyone 
 that stuff is built from source. It ensures that builds can be 
 re-produced by anyone and that issues with building are caught 
 fast by anyone consuming the library.


I agree 100%.  I've run into similar python package hell dealing 
with Tensor flow on Rocky Linux 8, since it wasn't Ubuntu.  The 
problem is we're small, and many dependencies are already written 
in C, and dub is not setup to handle C dependencies.  If I could 
turn my legacy C projects (pre-processor macros and all) into dub 
projects I'd do so as fast as possible.

If dub supported two things:

1. Building C projects

2. Installing the resulting binaries in "site-packages" or 
similar.

many, many project kickstart issues would be solved.

Like C++, D is designed as a successor to C.  It would be nice to 
bring C into the party with full dub support.
Jul 21 2023
prev sibling parent Chris Piker <chris hoopjump.com> writes:
On Friday, 21 July 2023 at 17:40:25 UTC, Greggor wrote:


 Up to date versions of Windows 10 should have curl included and 
 dub can run commands before building, so you could try 
 downloading a prebuilt lib for windows via curl. 
 https://everything.curl.dev/get/windows


Hey, nice!  This might be a winner.  Unless I'm missing something 
libcairo-2.dll, libpng13.dll and zlib1.dll could be downloaded, 
and then ggplotd plot can build against those (probably?).

The key thing that dub is missing compared to pip is some concept 
of an "install" directory.  Compared to everything else dub 
handles, not adding this capability mostly comes down to self 
imposed limitations.  I wonder how many in the community would 
like to see those limitations removed?

I presume it's just as easy to add commands that run after the 
build as before.  I could have an alternate dub configuration 
that's short for "setup_build_test_install" that runs everything.

If multiple projects start defining an install target the same 
way, then you have a convention, which can eventually lead to a 
standard.
Jul 21 2023
prev sibling parent nbdusr <. example.com> writes:
On Thursday, 20 July 2023 at 02:37:54 UTC, Chris Piker wrote:

 Hi D

 One of my jobs is to release and maintain public data archives 
 from long-running scientific instruments.  In order to help 
 people understand how to process the data, sample code is often 
 included with the archive. Recently this has been in the form 
 of short programs that generate a plot of a representative time 
 period.  Python works well for this task as matplotlib support 
 is pretty much universal.  Matlab is also a reasonable choice 
 for many.

 [...]


https://forum.dlang.org/post/mgxcrmjcgasbdaipdwft forum.dlang.org
Jul 22 2023