digitalmars.D.learn - Asynchronous Programming and Eventhandling in D
- O/N/S (17/17) Jul 05 2016 Hi ("Grüss Gott")
- chmike (9/9) Jul 05 2016 Hello, It looks interesting.
- Eugene Wissner (13/30) Jul 05 2016 Servus,
- chmike (21/58) Jul 06 2016 The problem I see with libev is that it isn't compatible with the
- rikki cattermole (3/58) Jul 06 2016 Hmm interesting way to do errors, might have to steal that for
- Eugene Wissner (42/103) Jul 06 2016 The only reason libev was choosen is that it is the simplest
- chmike (31/72) Jul 06 2016 Your project and work is valuable on many aspects. I didn't mean
- Eugene Wissner (7/84) Jul 06 2016 Yes. it would be a way to go.
- O/N/S (29/58) Jul 07 2016 Hi,
Hi ("Grüss Gott")
I like the asynchronous events in Javascript.
Is something similar possible in D?
Found Dragos Carp's asynchronous library
(https://github.com/dcarp/asynchronous).
Are there any more integrated (in Phobos/in D) ways to work
asynchronously?
An example: One server ask a second server to calculate something
big.
The first server continues with his work, until the answer come
back from the second.
And so on...
Using threads or fibers would be a way, but has not the same
elegancy like the Javascript way. (To avoid discussions: D is
better ;-)
Greetings from Munich,
Ozan
Jul 05 2016
chmike <christophe meessen.net>
Hello, It looks interesting.
This is now my current list of asynchronous libraries written in
D (not a C lib wrapper).
Collie : https://github.com/putao-dev/collie
libasync : https://github.com/etcimon/libasync
eventcore : https://github.com/vibe-d/eventcore
asynchronous: https://github.com/dcarp/asynchronous
A benchmarking test suite is still missing. We need a reference
to compare performance of the libraries.
Jul 05 2016
On Tuesday, 5 July 2016 at 08:24:43 UTC, O/N/S wrote:
Hi ("Grüss Gott")
I like the asynchronous events in Javascript.
Is something similar possible in D?
Found Dragos Carp's asynchronous library
(https://github.com/dcarp/asynchronous).
Are there any more integrated (in Phobos/in D) ways to work
asynchronously?
An example: One server ask a second server to calculate
something big.
The first server continues with his work, until the answer come
back from the second.
And so on...
Using threads or fibers would be a way, but has not the same
elegancy like the Javascript way. (To avoid discussions: D is
better ;-)
Greetings from Munich,
Ozan
Servus,
I'm currently rewriting the base skeleton of libev in D (only for
linux for now) for web development aswell. And the next step
would be data structures, basic server, futures and yo on...
I was working with dcarp's asynchronous and i found it very very
good. It is till now the best I've seen in D for async
programming ( I mean its design and usability).
Can you describe what would you like to see more concretly. I
know js but how is it supposed to work for D? Maybe you can give
some example, kind of pseudo code? It would help me much to build
a concept and maybe we will see someday something usable in this
area :)
Jul 05 2016
On Tuesday, 5 July 2016 at 20:38:53 UTC, Eugene Wissner wrote:On Tuesday, 5 July 2016 at 08:24:43 UTC, O/N/S wrote:The problem I see with libev is that it isn't compatible with the CPIO API of windows. The C++ boost asio library, on the contrary, is compatible with the select/epoll/kqueue model and the Windows CPIO model. This is the reason I started on a D implementation of asio I called dasio [https://github.com/chmike/dasio]. Unfortunately my alimentary work didn't leave me any time to make progress on it. The only thing I manage to implement so far is asio's error code system. I'm glad to see other people see an interest to work on this. We definitely should find a way to combine our efforts. That is already a significant work made with the other libraries. My feeling is that providing support for very efficient IO in Phobos might have a strong impact on D's visibility and adoption for backend applications (e.g. servers). Performance is a very strong argument for adoption is such context. A list of requirements has been already published on the wiki. What I think is now missing is a benchmarking tool so that we can get numbers for each async lib implementation that we can also compare with a raw C implementation using the native functions.Hi ("Grüss Gott") I like the asynchronous events in Javascript. Is something similar possible in D? Found Dragos Carp's asynchronous library (https://github.com/dcarp/asynchronous). Are there any more integrated (in Phobos/in D) ways to work asynchronously? An example: One server ask a second server to calculate something big. The first server continues with his work, until the answer come back from the second. And so on... Using threads or fibers would be a way, but has not the same elegancy like the Javascript way. (To avoid discussions: D is better ;-) Greetings from Munich, OzanServus, I'm currently rewriting the base skeleton of libev in D (only for linux for now) for web development aswell. And the next step would be data structures, basic server, futures and yo on... I was working with dcarp's asynchronous and i found it very very good. It is till now the best I've seen in D for async programming ( I mean its design and usability). Can you describe what would you like to see more concretly. I know js but how is it supposed to work for D? Maybe you can give some example, kind of pseudo code? It would help me much to build a concept and maybe we will see someday something usable in this area :)
Jul 06 2016
On 06/07/2016 8:39 PM, chmike wrote:On Tuesday, 5 July 2016 at 20:38:53 UTC, Eugene Wissner wrote:Hmm interesting way to do errors, might have to steal that for http://spew.cfOn Tuesday, 5 July 2016 at 08:24:43 UTC, O/N/S wrote:The problem I see with libev is that it isn't compatible with the CPIO API of windows. The C++ boost asio library, on the contrary, is compatible with the select/epoll/kqueue model and the Windows CPIO model. This is the reason I started on a D implementation of asio I called dasio [https://github.com/chmike/dasio]. Unfortunately my alimentary work didn't leave me any time to make progress on it. The only thing I manage to implement so far is asio's error code system. I'm glad to see other people see an interest to work on this. We definitely should find a way to combine our efforts. That is already a significant work made with the other libraries. My feeling is that providing support for very efficient IO in Phobos might have a strong impact on D's visibility and adoption for backend applications (e.g. servers). Performance is a very strong argument for adoption is such context. A list of requirements has been already published on the wiki. What I think is now missing is a benchmarking tool so that we can get numbers for each async lib implementation that we can also compare with a raw C implementation using the native functions.Hi ("Grüss Gott") I like the asynchronous events in Javascript. Is something similar possible in D? Found Dragos Carp's asynchronous library (https://github.com/dcarp/asynchronous). Are there any more integrated (in Phobos/in D) ways to work asynchronously? An example: One server ask a second server to calculate something big. The first server continues with his work, until the answer come back from the second. And so on... Using threads or fibers would be a way, but has not the same elegancy like the Javascript way. (To avoid discussions: D is better ;-) Greetings from Munich, OzanServus, I'm currently rewriting the base skeleton of libev in D (only for linux for now) for web development aswell. And the next step would be data structures, basic server, futures and yo on... I was working with dcarp's asynchronous and i found it very very good. It is till now the best I've seen in D for async programming ( I mean its design and usability). Can you describe what would you like to see more concretly. I know js but how is it supposed to work for D? Maybe you can give some example, kind of pseudo code? It would help me much to build a concept and maybe we will see someday something usable in this area :)
Jul 06 2016
On Wednesday, 6 July 2016 at 08:39:03 UTC, chmike wrote:On Tuesday, 5 July 2016 at 20:38:53 UTC, Eugene Wissner wrote:The only reason libev was choosen is that it is the simplest implementation I know about. A few C files. I had an educational purpose: I wanted to see how an event loop works on low level. Asyncio was for me no-go, since I've not written a lot in C++, and can only read the code a bit. So I'm not hanging on libev. The only another implementation would be the python event library. It is also pure C code and it was much more cleaner written than libev on the first sight. Now there are two problems with my work: 1) The first is something we all are tired to talk about: manual memory management. I make a proof of concept and am writing the code that is 100% marked as nogc. It has side effects. For example I allocate exceptions and thay should be freed after catching - it is something, phobos doesn't do. As said it is an experiment; I would like to see how it works. 2) Performance. The main reason I started the writing at all is that the existing implementations seem to have only performance as criterium. Performance is super important, but not on the cost of design, usability and extensibility. For example in vibe.d (and libasync) everything possible is defined as structs, everything that would be interesting to extend is final; and after it you go to phobos and see a "workaround". For example Mallocator is a struct but you have to be sure, it is an allocator. How do you force the right interface? static if (hasMember!(Allocator, "deallocate")) { return impl.deallocate(b); } else { return false; } Somethinkg like this would be ok for C. But for a language with interfaces, it is ugly design independent of how it performs. Everything I say here is IMHO. Except these two points I'm interested in some kind of collective work aswell. It is very difficult as one man job. I didn't know other people are also working on similar implementations. Nice to know. Are you aware of any benchmark tools in other languages that could be used?On Tuesday, 5 July 2016 at 08:24:43 UTC, O/N/S wrote:The problem I see with libev is that it isn't compatible with the CPIO API of windows. The C++ boost asio library, on the contrary, is compatible with the select/epoll/kqueue model and the Windows CPIO model. This is the reason I started on a D implementation of asio I called dasio [https://github.com/chmike/dasio]. Unfortunately my alimentary work didn't leave me any time to make progress on it. The only thing I manage to implement so far is asio's error code system. I'm glad to see other people see an interest to work on this. We definitely should find a way to combine our efforts. That is already a significant work made with the other libraries. My feeling is that providing support for very efficient IO in Phobos might have a strong impact on D's visibility and adoption for backend applications (e.g. servers). Performance is a very strong argument for adoption is such context. A list of requirements has been already published on the wiki. What I think is now missing is a benchmarking tool so that we can get numbers for each async lib implementation that we can also compare with a raw C implementation using the native functions.Hi ("Grüss Gott") I like the asynchronous events in Javascript. Is something similar possible in D? Found Dragos Carp's asynchronous library (https://github.com/dcarp/asynchronous). Are there any more integrated (in Phobos/in D) ways to work asynchronously? An example: One server ask a second server to calculate something big. The first server continues with his work, until the answer come back from the second. And so on... Using threads or fibers would be a way, but has not the same elegancy like the Javascript way. (To avoid discussions: D is better ;-) Greetings from Munich, OzanServus, I'm currently rewriting the base skeleton of libev in D (only for linux for now) for web development aswell. And the next step would be data structures, basic server, futures and yo on... I was working with dcarp's asynchronous and i found it very very good. It is till now the best I've seen in D for async programming ( I mean its design and usability). Can you describe what would you like to see more concretly. I know js but how is it supposed to work for D? Maybe you can give some example, kind of pseudo code? It would help me much to build a concept and maybe we will see someday something usable in this area :)
Jul 06 2016
On Wednesday, 6 July 2016 at 11:33:53 UTC, Eugene Wissner wrote:The only reason libev was choosen is that it is the simplest implementation I know about. A few C files. I had an educational purpose: I wanted to see how an event loop works on low level. Asyncio was for me no-go, since I've not written a lot in C++, and can only read the code a bit. So I'm not hanging on libev. The only another implementation would be the python event library. It is also pure C code and it was much more cleaner written than libev on the first sight.Your project and work is valuable on many aspects. I didn't mean to depreciate it.Now there are two problems with my work: 1) The first is something we all are tired to talk about: manual memory management. I make a proof of concept and am writing the code that is 100% marked as nogc. It has side effects. For example I allocate exceptions and thay should be freed after catching - it is something, phobos doesn't do. As said it is an experiment; I would like to see how it works. 2) Performance. The main reason I started the writing at all is that the existing implementations seem to have only performance as criterium. Performance is super important, but not on the cost of design, usability and extensibility. For example in vibe.d (and libasync) everything possible is defined as structs, everything that would be interesting to extend is final; and after it you go to phobos and see a "workaround". For example Mallocator is a struct but you have to be sure, it is an allocator. How do you force the right interface?That is a valid point. I know it is hard to get the best performance and optimal API design at the same time. The reason the methods are final is to avoid the overhead of virtual method indirection.static if (hasMember!(Allocator, "deallocate")) { return impl.deallocate(b); } else { return false; } Somethinkg like this would be ok for C. But for a language with interfaces, it is ugly design independent of how it performs. Everything I say here is IMHO.This would mean we need a new design pattern that supports both.Except these two points I'm interested in some kind of collective work aswell. It is very difficult as one man job. I didn't know other people are also working on similar implementations. Nice to know. Are you aware of any benchmark tools in other languages that could be used?The benchmark tools available are mainly testing web servers. And the exact operation pattern is not very clear. One of such benchmark tool is wrk [https://github.com/wg/wrk] that measure HTTP request speed. The problem is that it measure the performance of the I/O and HTTP handling. Here are some benchmark results: * https://www.techempower.com/benchmarks/ * https://github.com/nanoant/WebFrameworkBenchmark How can D be worse than Java ? My strategy would be to split the problem. First get the async I/O optimal. Then get HTTP handling optimal, and finally get database interaction optimal (the optimal async I/O should help). An investigation on the methods used by Java/undertow to get these performances could help. I would suggest to implement a benchmark client doing some predefined patterns of I/O operations similar to web interactions or the most common types of interactions. And a server implemented in C using native system I/O operations. This server implementation would then be our reference. What would be measured is how much slower our different D implementations are relative to the C reference implementation. This will allow us to have a benchmarking test that doesn't depend that much of the hardware.
Jul 06 2016
On Wednesday, 6 July 2016 at 14:57:08 UTC, chmike wrote:On Wednesday, 6 July 2016 at 11:33:53 UTC, Eugene Wissner wrote:Yes. it would be a way to go. As I see Kore performs pretty well. One could write bindings first and then begin to rewrite step for step. On every development step you have a working http server and you can use these http-benchmark tools for the benchmarking. I have to look in it in the next weeksThe only reason libev was choosen is that it is the simplest implementation I know about. A few C files. I had an educational purpose: I wanted to see how an event loop works on low level. Asyncio was for me no-go, since I've not written a lot in C++, and can only read the code a bit. So I'm not hanging on libev. The only another implementation would be the python event library. It is also pure C code and it was much more cleaner written than libev on the first sight.Your project and work is valuable on many aspects. I didn't mean to depreciate it.Now there are two problems with my work: 1) The first is something we all are tired to talk about: manual memory management. I make a proof of concept and am writing the code that is 100% marked as nogc. It has side effects. For example I allocate exceptions and thay should be freed after catching - it is something, phobos doesn't do. As said it is an experiment; I would like to see how it works. 2) Performance. The main reason I started the writing at all is that the existing implementations seem to have only performance as criterium. Performance is super important, but not on the cost of design, usability and extensibility. For example in vibe.d (and libasync) everything possible is defined as structs, everything that would be interesting to extend is final; and after it you go to phobos and see a "workaround". For example Mallocator is a struct but you have to be sure, it is an allocator. How do you force the right interface?That is a valid point. I know it is hard to get the best performance and optimal API design at the same time. The reason the methods are final is to avoid the overhead of virtual method indirection.static if (hasMember!(Allocator, "deallocate")) { return impl.deallocate(b); } else { return false; } Somethinkg like this would be ok for C. But for a language with interfaces, it is ugly design independent of how it performs. Everything I say here is IMHO.This would mean we need a new design pattern that supports both.Except these two points I'm interested in some kind of collective work aswell. It is very difficult as one man job. I didn't know other people are also working on similar implementations. Nice to know. Are you aware of any benchmark tools in other languages that could be used?The benchmark tools available are mainly testing web servers. And the exact operation pattern is not very clear. One of such benchmark tool is wrk [https://github.com/wg/wrk] that measure HTTP request speed. The problem is that it measure the performance of the I/O and HTTP handling. Here are some benchmark results: * https://www.techempower.com/benchmarks/ * https://github.com/nanoant/WebFrameworkBenchmark How can D be worse than Java ? My strategy would be to split the problem. First get the async I/O optimal. Then get HTTP handling optimal, and finally get database interaction optimal (the optimal async I/O should help). An investigation on the methods used by Java/undertow to get these performances could help. I would suggest to implement a benchmark client doing some predefined patterns of I/O operations similar to web interactions or the most common types of interactions. And a server implemented in C using native system I/O operations. This server implementation would then be our reference. What would be measured is how much slower our different D implementations are relative to the C reference implementation. This will allow us to have a benchmarking test that doesn't depend that much of the hardware.
Jul 06 2016
On Tuesday, 5 July 2016 at 20:38:53 UTC, Eugene Wissner wrote:On Tuesday, 5 July 2016 at 08:24:43 UTC, O/N/S wrote:Hi, I took a typical example (using jquery lib, other js-libs have similar calls) $.ajax({ type: "POST", url: myurl, dataType: "JSON", data: $myData, async: true, success: function(success) { ... }, error: function(error) { ... } }); If response is successful -> call function(success) If response has errors -> call function(error) In the meanwhile the code can continue without waiting for the response. In D-style it would look like ...do something.. funcAjax(["type":"POST"...], delegateSuccess, delegateError); - or - funcAjax(["type":"POST"...], (success) => {...}, (error) => {...}); ...continue without waiting... Why delegates not functions? Because responses could require changes in data, which are outside (yeah, I know, it's also technically possible with function, but that's not the idea behind) Greetings from Munich, OzanHi ("Grüss Gott") I like the asynchronous events in Javascript. Is something similar possible in D? Found Dragos Carp's asynchronous library (https://github.com/dcarp/asynchronous). Are there any more integrated (in Phobos/in D) ways to work asynchronously? An example: One server ask a second server to calculate something big. The first server continues with his work, until the answer come back from the second. And so on... Using threads or fibers would be a way, but has not the same elegancy like the Javascript way. (To avoid discussions: D is better ;-) Greetings from Munich, OzanCan you describe what would you like to see more concretly. I know js but how is it supposed to work for D? Maybe you can give some example, kind of pseudo code? It would help me much to build a concept and maybe we will see someday something usable in this area :)
Jul 07 2016