www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Working functionally with third party libraries

reply "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com> writes:
Hi

using mongo with vibe.d is easy. But I would like to skip the 
foreach on MongoCursor?

I mean, I want to use map!, filter! and reduce! on the resulting 
docs. Is there a fast way to convert MongoCursor to an array 
without resolving to ugly for loops with appender! ?
Jul 16 2015
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 07/16/2015 12:35 PM, "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= 
<jarl.andre gmail.com>" wrote:

 Hi

 using mongo with vibe.d is easy. But I would like to skip the foreach on
 MongoCursor?

 I mean, I want to use map!, filter! and reduce! on the resulting docs.
 Is there a fast way to convert MongoCursor to an array without resolving
 to ugly for loops with appender! ?


I've never used MongoCursor but judging from the fact that it has empty, 
front, and popFront(), it is an InputRange:

   http://vibed.org/api/vibe.db.mongo.cursor/MongoCursor

Have you tried using it with map and others? What errors do you get?

Ali
Jul 16 2015
next sibling parent "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com> writes:
On Thursday, 16 July 2015 at 20:00:38 UTC, Ali Çehreli wrote:

 On 07/16/2015 12:35 PM, "Jarl 
 =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com>" 
 wrote:

 Hi

 using mongo with vibe.d is easy. But I would like to skip the 
 foreach on
 MongoCursor?

 I mean, I want to use map!, filter! and reduce! on the 
 resulting docs.
 Is there a fast way to convert MongoCursor to an array without 
 resolving
 to ugly for loops with appender! ?


 I've never used MongoCursor but judging from the fact that it 
 has empty, front, and popFront(), it is an InputRange:

   http://vibed.org/api/vibe.db.mongo.cursor/MongoCursor

 Have you tried using it with map and others? What errors do you 
 get?

 Ali


I wish I could delete this thread :) Problem was with lack of 
auto completion for map!, each! etc in Eclipse. Please ignore :) 
Hush foreach away you go :)
Jul 16 2015
prev sibling parent reply "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com> writes:
On Thursday, 16 July 2015 at 20:00:38 UTC, Ali Çehreli wrote:

 On 07/16/2015 12:35 PM, "Jarl 
 =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com>" 
 wrote:

 Hi

 using mongo with vibe.d is easy. But I would like to skip the 
 foreach on
 MongoCursor?

 I mean, I want to use map!, filter! and reduce! on the 
 resulting docs.
 Is there a fast way to convert MongoCursor to an array without 
 resolving
 to ugly for loops with appender! ?


 I've never used MongoCursor but judging from the fact that it 
 has empty, front, and popFront(), it is an InputRange:

   http://vibed.org/api/vibe.db.mongo.cursor/MongoCursor

 Have you tried using it with map and others? What errors do you 
 get?

 Ali


Ah well I got another error now. Using the following code:

	Resource[] getResource() {
		auto coll = client.getCollection("app.resource");
		return coll.find().map!(doc => deserialize!(BsonSerializer, 
Resource)(doc));
	}

I get this error:

src/app.d(51,21): Error: cannot implicitly convert expression 
(map(coll.find())) of type MapResult!(__lambda1, 
MongoCursor!(Bson, Bson, typeof(null))) to Resource[]
Jul 16 2015
next sibling parent "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com> writes:
On Thursday, 16 July 2015 at 20:17:54 UTC, Jarl André Hübenthal 
wrote:

 		return coll.find().map!(doc => deserialize!(BsonSerializer, 
 Resource)(doc));


Solved by wrapping return statement in array(...)
Jul 16 2015
prev sibling parent reply "ZombineDev" <valid_email he.re> writes:
On Thursday, 16 July 2015 at 20:17:54 UTC, Jarl André Hübenthal 
wrote:

 On Thursday, 16 July 2015 at 20:00:38 UTC, Ali Çehreli wrote:

 On 07/16/2015 12:35 PM, "Jarl 
 =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com>" 
 wrote:

 Hi

 using mongo with vibe.d is easy. But I would like to skip the 
 foreach on
 MongoCursor?

 I mean, I want to use map!, filter! and reduce! on the 
 resulting docs.
 Is there a fast way to convert MongoCursor to an array 
 without resolving
 to ugly for loops with appender! ?


 I've never used MongoCursor but judging from the fact that it 
 has empty, front, and popFront(), it is an InputRange:

   http://vibed.org/api/vibe.db.mongo.cursor/MongoCursor

 Have you tried using it with map and others? What errors do 
 you get?

 Ali


 Ah well I got another error now. Using the following code:

 	Resource[] getResource() {
 		auto coll = client.getCollection("app.resource");
 		return coll.find().map!(doc => deserialize!(BsonSerializer, 
 Resource)(doc));
 	}

 I get this error:

 src/app.d(51,21): Error: cannot implicitly convert expression 
 (map(coll.find())) of type MapResult!(__lambda1, 
 MongoCursor!(Bson, Bson, typeof(null))) to Resource[]


Most of the functions from std.algorithm and std.range return a 
lazy range which you need to iterate over go get its elements.
To turn those ranges to an array you need add a .array at the end 
(http://dlang.org/phobos/std_array#array). Here's a larger 
example: http://d.readthedocs.org/en/latest/introduction.html.

Another option is to return the elements as a range (by making 
return type of your function auto), instead of putting them into 
a newly allocated array (with .array). This way may be a bit more 
work (delaying the call to .array), but it can be quite efficient 
because it removes the need to allocate memory.
Jul 16 2015
parent reply "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com> writes:
On Thursday, 16 July 2015 at 20:41:21 UTC, ZombineDev wrote:

 On Thursday, 16 July 2015 at 20:17:54 UTC, Jarl André Hübenthal 
 wrote:

 On Thursday, 16 July 2015 at 20:00:38 UTC, Ali Çehreli wrote:

 On 07/16/2015 12:35 PM, "Jarl 
 =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com>" 
 wrote:

 Hi

 using mongo with vibe.d is easy. But I would like to skip 
 the foreach on
 MongoCursor?

 I mean, I want to use map!, filter! and reduce! on the 
 resulting docs.
 Is there a fast way to convert MongoCursor to an array 
 without resolving
 to ugly for loops with appender! ?


 I've never used MongoCursor but judging from the fact that it 
 has empty, front, and popFront(), it is an InputRange:

   http://vibed.org/api/vibe.db.mongo.cursor/MongoCursor

 Have you tried using it with map and others? What errors do 
 you get?

 Ali


 Ah well I got another error now. Using the following code:

 	Resource[] getResource() {
 		auto coll = client.getCollection("app.resource");
 		return coll.find().map!(doc => deserialize!(BsonSerializer, 
 Resource)(doc));
 	}

 I get this error:

 src/app.d(51,21): Error: cannot implicitly convert expression 
 (map(coll.find())) of type MapResult!(__lambda1, 
 MongoCursor!(Bson, Bson, typeof(null))) to Resource[]


 Most of the functions from std.algorithm and std.range return a 
 lazy range which you need to iterate over go get its elements.
 To turn those ranges to an array you need add a .array at the 
 end (http://dlang.org/phobos/std_array#array). Here's a larger 
 example: http://d.readthedocs.org/en/latest/introduction.html.

 Another option is to return the elements as a range (by making 
 return type of your function auto), instead of putting them 
 into a newly allocated array (with .array). This way may be a 
 bit more work (delaying the call to .array), but it can be 
 quite efficient because it removes the need to allocate memory.


Thanks. Its a lot more cleaner and syntactically readable having 
.array at the end. But about laziness the same applies to clojure 
and scala. In clojure you must force evaluate the list, in scala 
you must to mostly the same as in D, put a toList or something at 
the end. Or loop it. But its pretty nice to know that there is 
laziness in D, but when I query mongo I expect all docs to be 
retrieved, since there are no paging in the underlying queries? 
Thus, having a lazy functionality on top of non lazy db queries 
seem a bit off dont you think?
Jul 17 2015
next sibling parent reply "sigod" <sigod.mail gmail.com> writes:
On Friday, 17 July 2015 at 09:07:29 UTC, Jarl André Hübenthal 
wrote:

 But its pretty nice to know that there is laziness in D, but 
 when I query mongo I expect all docs to be retrieved, since 
 there are no paging in the underlying queries? Thus, having a 
 lazy functionality on top of non lazy db queries seem a bit off 
 dont you think?


Not true. There's paging in MongoDB. See [Cursors][0] and 
documentation for `find` method. Also, look at [`vibe.d`'s 
implementation][1].

[0]: http://docs.mongodb.org/manual/core/cursors/
[1]: 
https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/db/mongo/cursor.d
Jul 17 2015
parent "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com> writes:
On Friday, 17 July 2015 at 11:47:30 UTC, sigod wrote:

 On Friday, 17 July 2015 at 09:07:29 UTC, Jarl André Hübenthal 
 wrote:

 But its pretty nice to know that there is laziness in D, but 
 when I query mongo I expect all docs to be retrieved, since 
 there are no paging in the underlying queries? Thus, having a 
 lazy functionality on top of non lazy db queries seem a bit 
 off dont you think?


 Not true. There's paging in MongoDB. See [Cursors][0] and 
 documentation for `find` method. Also, look at [`vibe.d`'s 
 implementation][1].

 [0]: http://docs.mongodb.org/manual/core/cursors/
 [1]: 
 https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/db/mongo/cursor.d


+1 good to know.
Jul 17 2015
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
On Friday, 17 July 2015 at 09:07:29 UTC, Jarl André Hübenthal 
wrote:

 Or loop it. But its pretty nice to know that there is laziness 
 in D, but when I query mongo I expect all docs to be retrieved, 
 since there are no paging in the underlying queries? Thus, 
 having a lazy functionality on top of non lazy db queries seem 
 a bit off dont you think?


 From the client point of view db is sort of lazy: data is 
received from server as needed. Why would you want to put all 
data into an array before processing it? Why can't you process it 
from the range directly?
Jul 17 2015
parent reply "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com> writes:
On Friday, 17 July 2015 at 12:59:24 UTC, Kagamin wrote:

 On Friday, 17 July 2015 at 09:07:29 UTC, Jarl André Hübenthal 
 wrote:

 Or loop it. But its pretty nice to know that there is laziness 
 in D, but when I query mongo I expect all docs to be 
 retrieved, since there are no paging in the underlying 
 queries? Thus, having a lazy functionality on top of non lazy 
 db queries seem a bit off dont you think?


 From the client point of view db is sort of lazy: data is 
 received from server as needed. Why would you want to put all 
 data into an array before processing it? Why can't you process 
 it from the range directly?


Its simple. In most cases you do an advanced aggregated search in 
mongo, and what you get is then a mongocursor. Lets say I am 
retrieving all projects for a given customer where the project is 
started.. I really am in no interest of lazily evaluating this 
result, because I want to return this data to the client 
(browser) immediately. And lets say I am in a prototype phase 
where i haven't yet implemented all those nasty mongo queries, I 
want to be able to filter, map and reduce the result and work 
with arrays not some sort of non evaluated lazy MapResult. In 
scala luckily I have implicit converts, so that I can just stop 
thinking about it and have it converted automatically.
Jul 18 2015
parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 18 July 2015 at 08:03:56 UTC, Jarl André Hübenthal 
wrote:

 Its simple. In most cases you do an advanced aggregated search 
 in mongo, and what you get is then a mongocursor. Lets say I am 
 retrieving all projects for a given customer where the project 
 is started.. I really am in no interest of lazily evaluating 
 this result, because I want to return this data to the client 
 (browser) immediately.


How big is the slowdown you notice for lazy processing? Lazy 
processing is believed to be faster because it's less resource 
consuming.


 And lets say I am in a prototype phase where i haven't yet 
 implemented all those nasty mongo queries, I want to be able to 
 filter, map and reduce the result and work with arrays not some 
 sort of non evaluated lazy MapResult.


I believe those algorithms were written to work on lazy ranges. 
What makes you think they can't do that?
Jul 18 2015
parent reply "Jarl =?UTF-8?B?QW5kcsOpIEjDvGJlbnRoYWwi?= <jarl.andre gmail.com> writes:
On Saturday, 18 July 2015 at 09:18:14 UTC, Kagamin wrote:

 On Saturday, 18 July 2015 at 08:03:56 UTC, Jarl André Hübenthal 
 wrote:

 Its simple. In most cases you do an advanced aggregated search 
 in mongo, and what you get is then a mongocursor. Lets say I 
 am retrieving all projects for a given customer where the 
 project is started.. I really am in no interest of lazily 
 evaluating this result, because I want to return this data to 
 the client (browser) immediately.


 How big is the slowdown you notice for lazy processing? Lazy 
 processing is believed to be faster because it's less resource 
 consuming.


 And lets say I am in a prototype phase where i haven't yet 
 implemented all those nasty mongo queries, I want to be able 
 to filter, map and reduce the result and work with arrays not 
 some sort of non evaluated lazy MapResult.


 I believe those algorithms were written to work on lazy ranges. 
 What makes you think they can't do that?


I don't understand where you are going with this. I have solved 
my problem. Laziness is good for lets say take 5 out of infinite 
results. When you ask for a complete list and want the complete 
list, you take all. In clojure you actually say that, "doall". In 
D .array does the same thing. Converts lazy to non lazy.
Jul 18 2015
parent "Kagamin" <spam here.lot> writes:
On Saturday, 18 July 2015 at 09:33:37 UTC, Jarl André Hübenthal 
wrote:

 I don't understand where you are going with this. I have solved 
 my problem. Laziness is good for lets say take 5 out of 
 infinite results.


It's also good for saving resources, you don't spend time 
managing those resources and save that time to complete the 
processing earlier.
Jul 18 2015
prev sibling parent reply "ZombineDev" <valid_email he.re> writes:
On Friday, 17 July 2015 at 09:07:29 UTC, Jarl André Hübenthal 
wrote:

 Thanks. Its a lot more cleaner and syntactically readable 
 having .array at the end. But about laziness the same applies 
 to clojure and scala. In clojure you must force evaluate the 
 list, in scala you must to mostly the same as in D, put a 
 toList or something at the end. Or loop it. But its pretty nice 
 to know that there is laziness in D, but when I query mongo I 
 expect all docs to be retrieved, since there are no paging in 
 the underlying queries? Thus, having a lazy functionality on 
 top of non lazy db queries seem a bit off dont you think?


I'm almost certain that the D database driver returns eagerly all 
the results that you've requested. The lazy stuff should happen 
when you start doing range operations after the results are 
returned from the database. It's not impossible to lazily query 
the database, but I think that the developers have chosen the 
eager approach, since it's more straightforward.

Currently, in D most of the laziness is a convention, rather than 
something directly built into the language. There are many 
features that enable (indirectly) effective and easy to use lazy 
algorithms, but these features are have many other uses 
(templates, auto type deduction, compile-time reflection, etc.).

The only two direct features are:
1) foreach can iterate over ranges (objects of structs or classes 
for which isInputRange is true. Here's an example:

import std.algorithm.iteration : map, filter;

foreach (name; persons.filter!(p => p.age > 18).map!(p => p.name))
     writeln(name);

import std.range.primitives : isInputRange;

static assert (
     isInputRange!(
         typeof(
            persons.filter!(p => p.age > 18).map!(p => p.name)
         )
     ) == true
);

See http://dlang.org/phobos/std_range_primitives.html#isInputRange

2) The lazy keyword - when you annotate function parameters with 
lazy they are evaluated not at the caller site, but only when 
needed like in other more traditional functional languages. For 
example:

void calculate(int[] numbers)
{
     import std.format : format;
     // ...

     logErrorIf(numbers[3] < 5,
         format("Expected value < 5, but got %s !", numbers[3]));
     //  ^~~~~~~~~~~~ this is only evaluated

     // ...
}

void logErrorIf(bool condition, lazy string error_message)
{
     if (condition)
         writeln(message);
     //          ^~~~ here, if the condition is true
}

( In D string is just an alias to immutable(char)[], so the above 
signature is identical to this:
void logErrorIf(bool condition, lazy immutable(char)[] 
error_message) )

You can think of lazy parameters as implicit lambdas that return 
the expression passed as argument only when called.
Here you can learn more about the lazy keyword 
http://dlang.org/lazy-evaluation.html

Even though we have 'lazy' built into the language, most of the 
lazy algorithms do not use it. I just made a quick search through 
druntime and phobos for 'lazy' and 'range' (don't how correct it 
was - I admit I'm a unix noob) and here's what I got:

// (I have DMD v2.067.1 installed)

// lazy at the head of the function parameter list or in the tail
$ find /usr/include/dmd/ -name '*.d' -exec cat {} \; | grep -c 
'(lazy \|, lazy '
74

// just containing lazy
$ find /usr/include/dmd/ -name '*.d' -exec cat {} \; | grep -c 
'lazy'
138

// just containing range
$ find /usr/include/dmd/ -name '*.d' -exec cat {} \; | grep -c 
'range'
3548

I think that this because ranges are a more generic, flexible and 
powerful abstraction, and are more efficient maybe because 
they're easier to optimize to simple loops (eg. I've seen that 
the ldc compiler handles them very well).
'lazy' is still useful but generally I have seen it used for more 
simpler stuff (like the above 'lazy' example), and not for 
propagating state through range pipelines (or more simply - 
function chaining).

So you'll see both functions that are lazy and functions that are 
not throughout Phobos (and most use ranges, as you can see from 
the results).

Generally you can distinguish range functions from others by 
their signatures. Since most ranges in D are templated structs 
and not classes inheriting some interface
(though there some, see 
http://dlang.org/phobos/std_range_interfaces#InputRange),
functions that operate on ranges are templated at least on one 
range type:

// Check if the function 'fun' is really a predicate
enum isUnaryPredicate(alias fun, T) =
     is( typeof( fun(T.init) ) : bool);

import std.range.primitives: isInputRange, ElementType;

// templated on predicate and range type
//                ~~~~v~~~~  ~~v~~
auto filter1(alias predicate, Range)(Range range)
     if (isInputRange!Range &&   //       <- some template
         isUnaryPredicate!(predicate, ElementType!Range))
     //  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  constraints
{
     return ...
}

// Even if you can't look at the function body, you
// can guess that it can't be lazy because, it must
// have the whole result, before it returns it.
T[] filter2(alias predicate, T)(const(T)[] arrayToFilter)
     if (isUnaryPredicate!(predicate, T))
     //  ^~~~~~ a bit less template constraints
{
     return ...
}

Here's a full example:
http://dpaste.dzfl.pl/9023c63f9393
Jul 17 2015
next sibling parent "ZombineDev" <valid_email he.re> writes:
On Friday, 17 July 2015 at 15:41:22 UTC, ZombineDev wrote:

 I'm almost certain that the D database driver returns eagerly 
 all the results that you've requested. The lazy stuff should 
 happen when you start doing range operations after the results 
 are returned from the database. It's not impossible to lazily 
 query the database, but I think that the developers have chosen 
 the eager approach, since it's more straightforward.


Discard this. I didn't read the previous comments.
Jul 17 2015
prev sibling parent reply "sigod" <sigod.mail gmail.com> writes:
On Friday, 17 July 2015 at 15:41:22 UTC, ZombineDev wrote:

 eager approach, since it's more straightforward.


What makes you think it's always more straightforward? Sometimes 
(like in this case with MongoDB) you cannot write eager approach 
without first writing lazy one.
Jul 17 2015
parent "ZombineDev" <valid_email he.re> writes:
On Friday, 17 July 2015 at 17:56:51 UTC, sigod wrote:

 On Friday, 17 July 2015 at 15:41:22 UTC, ZombineDev wrote:

 eager approach, since it's more straightforward.


 What makes you think it's always more straightforward? 
 Sometimes (like in this case with MongoDB) you cannot write 
 eager approach without first writing lazy one.


Well I just wrote without properly looking up what MongoDB does. 
I thought collection.find() returned the first n elements, not a 
iterator (cursor in MongoDB's terms) and that it would be 
additional work split those n elements.
Anyway, thanks for the correction, now I am bit more educated ;)
Jul 17 2015