www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Netflix opensources its first D library: Vectorflow

reply Walter Bright <newshound2 digitalmars.com> writes:
https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/
Aug 02 2017
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Wednesday, 2 August 2017 at 21:31:19 UTC, Walter Bright wrote:

 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/


No. 2 liked proggit link of the day, should be no. 1 soon:

https://www.reddit.com/r/programming/top/?time=day

Not doing well on HN though:

https://hn.algolia.com/?query=vectorflow
Aug 02 2017
next sibling parent Samuel Lampa <samuel.lampa gmail.com> writes:
On Wednesday, 2 August 2017 at 22:56:32 UTC, Joakim wrote:

 Not doing well on HN though:

 https://hn.algolia.com/?query=vectorflow


HN is very sensitive to time of day when submitting. Did a new 
try:

https://news.ycombinator.com/item?id=14920608
Aug 03 2017
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Wednesday, 2 August 2017 at 22:56:32 UTC, Joakim wrote:

 On Wednesday, 2 August 2017 at 21:31:19 UTC, Walter Bright 
 wrote:

 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/


 No. 2 liked proggit link of the day, should be no. 1 soon:

 https://www.reddit.com/r/programming/top/?time=day

 Not doing well on HN though:

 https://hn.algolia.com/?query=vectorflow


Top 3 for the week:

https://www.reddit.com/r/programming/top/?sort=top&t=week

People seem really enthused by this library.
Aug 05 2017
prev sibling next sibling parent reply Matt <sdfssdfd gmail.com> writes:
On Wednesday, 2 August 2017 at 21:31:19 UTC, Walter Bright wrote:

 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/


Speakng of D in data science (where I think it can get traction), 
is there a standardized linear algebra library in D? Perhaps some 
hooks to Eigen?

I saw a few upstarts LA libraries, but none that I would consider 
"the one to use in D", like numpy in python or Eigen in C++
Aug 02 2017
parent reply Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Thursday, 3 August 2017 at 03:46:11 UTC, Matt wrote:

 On Wednesday, 2 August 2017 at 21:31:19 UTC, Walter Bright 
 wrote:

 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/


 Speakng of D in data science (where I think it can get 
 traction), is there a standardized linear algebra library in D? 
 Perhaps some hooks to Eigen?

 I saw a few upstarts LA libraries, but none that I would 
 consider "the one to use in D", like numpy in python or Eigen 
 in C++


We're using D in finance.  D libraries are far from the maturity 
of Python, but they are developing quickly.

https://github.com/kaleidicassociates/lubeck Library we have 
open-sourced
https://github.com/libmir/mir-algorithm Mir library we sponsor

Other mir libraries:
https://github.com/libmir/mir-lapack
https://github.com/libmir/mir-blas
https://github.com/libmir/lapack
https://github.com/DlangScience/cblas

Performance matters:
http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html
Aug 02 2017
next sibling parent reply Matt <sdfssdfd gmail.com> writes:
That seems like a good start. I'll probably start contributing 
within a few weeks.

In the long run the goal should probably be to have a "go-to" set 
of LA libraries, with appropriate data structures that have 
friendly bindings to GPU LA computations with DCompute.
Aug 02 2017
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 3 August 2017 at 04:37:30 UTC, Matt wrote:

 That seems like a good start. I'll probably start contributing 
 within a few weeks.


Great, http://gitter.im/libmir is probably the best way to get in 
contact.


 In the long run the goal should probably be to have a "go-to" 
 set of LA libraries, with appropriate data structures that have 
 friendly bindings to GPU LA computations with DCompute.


That is the idea, https://github.com/libmir/mir-glas already 
provides a BLAS and I'll eventually get around to getting 
DCompute integration for it and ndslice.
Aug 02 2017
prev sibling parent reply Matt <sdfssdfd gmail.com> writes:
Also note, one of the main advantages of Eigen is the whole lazy 
evaluation of expressions for compound operations.

I haven't dug in the source, but it's my understanding it's done 
through a lot of compile time C++ template hacking
Aug 02 2017
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Thursday, 3 August 2017 at 04:40:05 UTC, Matt wrote:

 Also note, one of the main advantages of Eigen is the whole 
 lazy evaluation of expressions for compound operations.


Yes, Mir does that too:

http://blog.mir.dlang.io/ndslice/algorithm/optimization/2016/12/12/writing-efficient-numerical-code.html


 I haven't dug in the source, but it's my understanding it's 
 done through a lot of compile time C++ template hacking


Meanwhile, the blog post Laeeth gave you shows Mir doing better 
on matrix multiplication benchmarks than Eigen, significantly 
better when dealing with complex numbers.
Aug 02 2017
parent reply Matt <sdfssdfd gmail.com> writes:
 Meanwhile, the blog post Laeeth gave you shows Mir doing better 
 on matrix multiplication benchmarks than Eigen, significantly 
 better when dealing with complex numbers.


I mean by now we should all be jaded enough not to simply take 
toy benchmarks as gospel for which is actually fastest in a 
non-trivial application.

I don't doubt mir is really fast, though.
Aug 03 2017
parent Joakim <dlang joakim.fea.st> writes:
On Thursday, 3 August 2017 at 14:00:31 UTC, Matt wrote:

 Meanwhile, the blog post Laeeth gave you shows Mir doing 
 better on matrix multiplication benchmarks than Eigen, 
 significantly better when dealing with complex numbers.


 I mean by now we should all be jaded enough not to simply take 
 toy benchmarks as gospel for which is actually fastest in a 
 non-trivial application.


That's why I didn't make such a general claim and noted that the 
linked benchmarks only dealt with matrix multiplication. :P


 I don't doubt mir is really fast, though.


Yes, the benchmarks are indicative, but it's up to you come up 
with a benchmark that characterizes your workload better.
Aug 03 2017
prev sibling next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 3 August 2017 at 04:40:05 UTC, Matt wrote:

 Also note, one of the main advantages of Eigen is the whole 
 lazy evaluation of expressions for compound operations.

 I haven't dug in the source, but it's my understanding it's 
 done through a lot of compile time C++ template hacking


Yep, there's a lot of operator overloading and other C++ black 
magic. DCompute should be able to do some of the out of the box 
with the lambda kernels to get some data locality. I'm also in 
the process of integrating Polly (LLVM's polyhedral optimiser) 
into LDC to get great loop optimisations.
Aug 02 2017
prev sibling parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Thursday, 3 August 2017 at 04:40:05 UTC, Matt wrote:

 Also note, one of the main advantages of Eigen is the whole 
 lazy evaluation of expressions for compound operations.

 I haven't dug in the source, but it's my understanding it's 
 done through a lot of compile time C++ template hacking


Note that D provides

     __traits(isRef, Symbol)

which can be used to convenient implement lazy-evaluated 
expressions with free functions. Note that it cannot currently be 
used to check whether `this` was passed as an l-value or r-value 
which prevents the C++-expression-template-pattern from being 
used in operator overloading.

For details see: https://issues.dlang.org/show_bug.cgi?id=17734
Aug 08 2017
parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Tuesday, 8 August 2017 at 18:40:08 UTC, Nordlöw wrote:

 On Thursday, 3 August 2017 at 04:40:05 UTC, Matt wrote:

 Also note, one of the main advantages of Eigen is the whole 
 lazy evaluation of expressions for compound operations.

 I haven't dug in the source, but it's my understanding it's 
 done through a lot of compile time C++ template hacking


 Note that D provides

     __traits(isRef, Symbol)


To clarify here's an incomplete snippet that should clarify:

auto add(A, B)(A a, B b)
    if (isSomeArithmeticType!A &&
        isSomeArithmeticType!B)
{
    static if (__traits(isRef, a) &&
               __traits(isRef, b)) // both `a` and `b` are l-values
    {
        return a + b; // fully eager evaluation
    }
    else static if (__traits(isRef, a)) // `b` is an r-value
    {
        // `b` can incremented by `a` and returned by move (reused)
    }
    else static if (__traits(isRef, b)) // `a` is an r-value
    {
        // `a` can incremented by `b` and returned by move (reused)
    }
    else // both `a` and `b` are r-values
    {
        return Add(a,b); // delay evaluation
    }
}
Aug 08 2017
parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Tuesday, 8 August 2017 at 18:51:26 UTC, Nordlöw wrote:

 To clarify here's an incomplete snippet that should clarify:

 auto add(A, B)(A a, B b)


should be

auto add(A, B)(auto ref A a, auto ref B b)
Oct 19 2017
prev sibling parent reply Stephan Dilly <stephan extrawurst.org> writes:
On 2017-08-02 21:31:19 +0000, Walter Bright said:


 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d
library_vectorflow/ 
 


D got another mention in Netflix's popular tech blog: 
https://medium.com/netflix-techblog/machine-learning-platform
meetup-ddec090f3c17 
(+meetup slides)

--Stephan
Oct 18 2017
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/18/2017 1:49 PM, Stephan Dilly wrote:

 On 2017-08-02 21:31:19 +0000, Walter Bright said:
 
 
 https://www.reddit.com/r/programming/comments/6r6dwp/netflix_opensources_its_first_d_library_vectorflow/
 
 
 D got another mention in Netflix's popular tech blog: 
 https://medium.com/netflix-techblog/machine-learning-platform
meetup-ddec090f3c17 (+meetup 
 slides)
 
 
 --Stephan
 


The quote:

"VectorFlow was designed for these contexts and this philosophy informed
several 
design decisions. For example the choice of the language, D, was one such 
decision. The modern system language was chosen to address the goal of a single 
language in adhoc as well as production contexts — providing the power of
C++ 
and the simplicity/clarity of Python."
Oct 19 2017