www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - dlang-requests 1.1.0 released

reply ikod <geller.garry gmail.com> writes:
Hello!

Just a note that dlang-requests ver 1.1.0 released with new 
'ByLine' interfaces added for get/post/put requests.

range algorithms can be applied to server responses, so that 
simple chain

getContentByLine("https://httpbin.org/anything")
     .map!"cast(string)a"
     .filter!(a => a.canFind("data"))

should work.

These calls work lazily so you can apply them to large documents.

dlang-requests - HTTP client library, inspired by python-requests 
with goals:

small memory footprint
performance
simple, high level API
native D implementation

https://github.com/ikod/dlang-requests
https://code.dlang.org/packages/requests

Always waiting for your bugreports and proposals on project page.

Best regards!
Apr 05 2020
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

 Hello!

 Just a note that dlang-requests ver 1.1.0 released with new 
 'ByLine' interfaces added for get/post/put requests.

 range algorithms can be applied to server responses, so that 
 simple chain

 getContentByLine("https://httpbin.org/anything")
     .map!"cast(string)a"
     .filter!(a => a.canFind("data"))

 should work.

 These calls work lazily so you can apply them to large 
 documents.

 dlang-requests - HTTP client library, inspired by 
 python-requests with goals:

 small memory footprint
 performance
 simple, high level API
 native D implementation

 https://github.com/ikod/dlang-requests
 https://code.dlang.org/packages/requests

 Always waiting for your bugreports and proposals on project 
 page.

 Best regards!


Nice work!

One quick suggestion: avoid direct casting from `ubyte[]` to 
`string`:

/+dub.sdl:
dependency "requests" version="~>1.1"
+/

void main()
{
     import requests : getContentByLine;
     import std : assumeUTF, canFind, each, filter, map, write;

     getContentByLine("https://httpbin.org/anything")
         .map!assumeUTF // instead of map!"cast(string)a"
         .filter!(a => a.canFind("data"))
         .each!write;
}

1. From a code-style point of view, assumeUTF is better as it 
shows the intention to the reader - assume that the content is 
valid UTF8 encoded text, without performing validation. And if 
there are UTF8 errors, it is easy to go back and add validation 
there.
2. Avoid casting mutable data to immutable. The data path in your 
library is rather complex (getContentByLine -> _LineReader -> 
LineSplitter -> Buffer -> ...) and so it was hard to understand 
from a quick glance whether or not the buffer array is reused 
(but I would guess that it is). If the buffer array is reused, it 
means that the result of calling _LineReader.front() may be 
modified at a later point in time, which I think is obvious that 
it can lead to some rather nasty bugs in users' code.

I suggest you look into Steven's iopipe[1] library, as I believe 
it can help you clean up and refactor this part of the codebase 
(and can probably yield some performance improvements along the 
way).

[1]: https://github.com/schveiguy/iopipe
Apr 05 2020
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Sunday, 5 April 2020 at 11:53:29 UTC, Petar Kirov [ZombineDev] 
wrote:

 On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

 Hello!

 Just a note that dlang-requests ver 1.1.0 released with new 
 'ByLine' interfaces added for get/post/put requests.

 range algorithms can be applied to server responses, so that 
 simple chain

 getContentByLine("https://httpbin.org/anything")
     .map!"cast(string)a"
     .filter!(a => a.canFind("data"))

 should work.

 These calls work lazily so you can apply them to large 
 documents.

 dlang-requests - HTTP client library, inspired by 
 python-requests with goals:

 small memory footprint
 performance
 simple, high level API
 native D implementation

 https://github.com/ikod/dlang-requests
 https://code.dlang.org/packages/requests

 Always waiting for your bugreports and proposals on project 
 page.

 Best regards!


 Nice work!


I noticed that by default trace logging from requests is turned 
on, which is quite verbose. I checked the docs in the readme file 
and at first, I only saw suggestions on how to customize the 
logging by writing a LoggerInterceptor, which I thought was a bit 
too much for casual use. Only after a bit more digging I found 
that you were using std.experimental.logging (e.g. here [1]). 
Later I found that the SSL example in the readme used 
std.experimental.logging [2], but I think it would be better to 
explicitly mention it earlier on, e.g. like this:

/+dub.sdl:
dependency "requests" version="~>1.1"
+/

void main()
{
     import requests : getContentByLine;
     import std : assumeUTF, canFind, each, filter, map, write, 
writeln;

     /*
      * dlang_requests uses `std.experimental.logger` for interal 
logging.
      *
      * The globalLogLevel is set to `LogLevel.all` by default in 
Phobos, which
      * may be too verbose for most applications.
      *
      * This can be changed like this:
      */
     import std.experimental.logger : globalLogLevel, LogLevel;
     globalLogLevel = LogLevel.info;

     getContentByLine("https://httpbin.org/anything")
         .map!assumeUTF
         .filter!(a => a.canFind("data"))
         .each!write;
}


[1]: 
https://github.com/ikod/dlang-requests/blob/v1.1.0/source/requests/streams.d#L383

[2]: 
https://github.com/ikod/dlang-requests/blob/v1.1.0/README.md#ssl-settings
Apr 05 2020
prev sibling parent ikod <geller.garry gmail.com> writes:
On Sunday, 5 April 2020 at 11:53:29 UTC, Petar Kirov [ZombineDev] 
wrote:

 On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

 Hello!

 Just a note that dlang-requests ver 1.1.0 released with new 
 'ByLine' interfaces added for get/post/put requests.

 range algorithms can be applied to server responses, so that 
 simple chain

 getContentByLine("https://httpbin.org/anything")
     .map!"cast(string)a"
     .filter!(a => a.canFind("data"))

 should work.

 These calls work lazily so you can apply them to large 
 documents.

 dlang-requests - HTTP client library, inspired by 
 python-requests with goals:

 small memory footprint
 performance
 simple, high level API
 native D implementation

 https://github.com/ikod/dlang-requests
 https://code.dlang.org/packages/requests

 Always waiting for your bugreports and proposals on project 
 page.

 Best regards!


 Nice work!

 One quick suggestion: avoid direct casting from `ubyte[]` to 
 `string`:

 /+dub.sdl:
 dependency "requests" version="~>1.1"
 +/

 void main()
 {
     import requests : getContentByLine;
     import std : assumeUTF, canFind, each, filter, map, write;

     getContentByLine("https://httpbin.org/anything")
         .map!assumeUTF // instead of map!"cast(string)a"
         .filter!(a => a.canFind("data"))
         .each!write;
 }

 1. From a code-style point of view, assumeUTF is better as it 
 shows the intention to the reader - assume that the content is 
 valid UTF8 encoded text, without performing validation. And if 
 there are UTF8 errors, it is easy to go back and add validation 
 there.
 2. Avoid casting mutable data to immutable. The data path in 
 your library is rather complex (getContentByLine -> _LineReader 
 -> LineSplitter -> Buffer -> ...) and so it was hard to 
 understand from a quick glance whether or not the buffer array 
 is reused (but I would guess that it is). If the buffer array 
 is reused, it means that the result of calling 
 _LineReader.front() may be modified at a later point in time, 
 which I think is obvious that it can lead to some rather nasty 
 bugs in users' code.


Internally my code do not modify these data, but I understand 
your concern.


 I suggest you look into Steven's iopipe[1] library, as I 
 believe it can help you clean up and refactor this part of the 
 codebase (and can probably yield some performance improvements 
 along the way).

 [1]: https://github.com/schveiguy/iopipe


Thanks! You are totally right regarding casting mutable to 
immutable. Base code was written when I did not understand this 
problem well enough. I know how to fix this but this probably 
require large enough rewrite. I hope I'll do this in some near 
future.

PS. I also have package which convert nogc unique mutable byte 
array to immutable byte array and attach this immutable data "as 
is" to buffer (which is essentally array of immutable(ubyte)[]). 
Then every operation on buffer (trim, split, search, etc) 
performed on immutable with zero copy and with warranties on data 
immutability.
Apr 05 2020
prev sibling next sibling parent reply Anonymouse <zorael gmail.com> writes:
On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

 Hello!

 Just a note that dlang-requests ver 1.1.0 released with new 
 'ByLine' interfaces added for get/post/put requests.


[...]

As always, thanks for your hard work! I don't use much more than 
the normal `get` and `receiveAsRange`, but it's been working well.
Apr 05 2020
parent ikod <geller.garry gmail.com> writes:
On Sunday, 5 April 2020 at 16:25:52 UTC, Anonymouse wrote:

 On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

 Hello!

 Just a note that dlang-requests ver 1.1.0 released with new 
 'ByLine' interfaces added for get/post/put requests.


 [...]

 As always, thanks for your hard work! I don't use much more 
 than the normal `get` and `receiveAsRange`, but it's been 
 working well.


Thanks for your support :) This work is pleasure for me, glad 
that it can be useful.
Apr 05 2020
prev sibling parent reply Pavel Shkadzko <p.shkadzko gmail.com> writes:
On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

 Hello!

 Just a note that dlang-requests ver 1.1.0 released with new 
 'ByLine' interfaces added for get/post/put requests.

 range algorithms can be applied to server responses, so that 
 simple chain

 getContentByLine("https://httpbin.org/anything")
     .map!"cast(string)a"
     .filter!(a => a.canFind("data"))

 should work.

 These calls work lazily so you can apply them to large 
 documents.

 dlang-requests - HTTP client library, inspired by 
 python-requests with goals:

 small memory footprint
 performance
 simple, high level API
 native D implementation

 https://github.com/ikod/dlang-requests
 https://code.dlang.org/packages/requests

 Always waiting for your bugreports and proposals on project 
 page.

 Best regards!


Very nice to see requests for D.

It is written in README that it has a small memory footprint and 
performance but how does it compare against Python version?

I am asking because we have some Python packages using requests 
and I wanted to try to rewrite them in D. To avoid ppl raising 
eyebrows it would be nice to know a bit more details if possible.
May 12 2020
parent ikod <geller.garry gmail.com> writes:
On Tuesday, 12 May 2020 at 09:56:44 UTC, Pavel Shkadzko wrote:

 On Sunday, 5 April 2020 at 08:59:50 UTC, ikod wrote:

 Hello!

 Just a note that dlang-requests ver 1.1.0 released with new 
 'ByLine' interfaces added for get/post/put requests.

 range algorithms can be applied to server responses, so that 
 simple chain

 getContentByLine("https://httpbin.org/anything")
     .map!"cast(string)a"
     .filter!(a => a.canFind("data"))

 should work.

 These calls work lazily so you can apply them to large 
 documents.

 dlang-requests - HTTP client library, inspired by 
 python-requests with goals:

 small memory footprint
 performance
 simple, high level API
 native D implementation

 https://github.com/ikod/dlang-requests
 https://code.dlang.org/packages/requests

 Always waiting for your bugreports and proposals on project 
 page.

 Best regards!


 Very nice to see requests for D.

 It is written in README that it has a small memory footprint 
 and performance but how does it compare against Python version?


I never compared performance with Python requests, but I made 
some comparisons with
curl and wget, but results were never published. I just checked 
that there are no any odd delays.

There were some performance issues with connection pooling and 
caching redirects (like 
https://github.com/ikod/dlang-requests/issues/80), but they were 
fixed quite long ago.


 I am asking because we have some Python packages using requests 
 and I wanted to try to rewrite them in D. To avoid ppl raising 
 eyebrows it would be nice to know a bit more details if 
 possible.


There are some functions of Python requests that were not 
implemented for dlang - for example, conversion to json and maybe 
a few others, but I can easily add them if someone really needs 
it.

At the same time I tried to adapt API to Dlang way - like using 
lazy ranges when sending or receiving data, and like lazy byLine 
interator over response body, so your code may look and perform 
better with dlang-requests than with python.


PS. I mentioned small memory footprint in README because, for 
some reason, I failed to keep memory usage in reasonable bounds 
with std.net.curl at the time I tried to use it.
May 12 2020