www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - gRPC in D good idea for GSOC 2018?

reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
I know a project where D could benefit from gRPC in D, which is not 
among the supported languages:

   https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

Ali
Jan 15 2018
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which is 
 not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali


Also, the deadline is in 1 week.
Jan 16 2018
parent Seb <seb wilzba.ch> writes:
On Tuesday, 16 January 2018 at 17:58:40 UTC, jmh530 wrote:

 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which 
 is not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali


 Also, the deadline is in 1 week.


Yes it's a good idea for GSoC18 and we still accept ideas. Reply 
here if you have more ideas ;-)
Jan 18 2018
prev sibling next sibling parent reply yawniek <yawniek srtnwz.com> writes:
On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which is 
 not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali


Problem here is that D lacks a proper http/2 stack.
unfortunately due to time constraints my libh2o binding project 
got stuck.

Imo the real problem with D is still that phobos does not provide 
the important building blocks for today modern infrastrucutre:

- fast and standardized eventloop compatible with Ds concurrency 
primitives
- network protocol libs/ bindings ( http, http/2, tls, dns )

until there is effort to do this D will continue to be an 
ecosystem of hacks.
I really like vibe.d but it comes with a lot of problems and it 
has the focus of providing a usable web framework and not 
implementing protocols in a fast and reusable way.

so as long as "management" does not decide that internet 
protocols is something important in D nothing useful will come 
out. Just look how beautiful Golangs protocol implementations are 
and the whole ecosystem that focused effort spawned.
Jan 19 2018
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 01/19/2018 02:14 AM, yawniek wrote:


 Just look how
 beautiful Golangs protocol implementations are and the whole ecosystem
 that focused effort spawned.


I agree and do have first-hand experience with that but I'm not 
experienced enough in this field to come up with a modern solution. If 
it were left to me, I would start looking at Go and Python solutions but 
I fear I wouldn't see design decisions that could have been better.

Could you please summarize what needs to be done and in what order. I 
would be happy to take active part in this effort.

Thank you,
Ali
Jan 19 2018
next sibling parent aberba <karabutaworld gmail.com> writes:
On Friday, 19 January 2018 at 18:34:31 UTC, Ali Çehreli wrote:

 On 01/19/2018 02:14 AM, yawniek wrote:


 Just look how
 beautiful Golangs protocol implementations are and the whole


 ecosystem

 that focused effort spawned.




I have said this here before.


 I agree and do have first-hand experience with that but I'm not 
 experienced enough in this field to come up with a modern 
 solution. If it were left to me, I would start looking at Go 
 and Python solutions but I fear I wouldn't see design decisions 
 that could have been better.

 Could you please summarize what needs to be done and in what 
 order. I would be happy to take active part in this effort.

 Thank you,
 Ali
Jan 20 2018
prev sibling parent yawniek <yawniek srtnwz.com> writes:
On Friday, 19 January 2018 at 18:34:31 UTC, Ali Çehreli wrote:

 On 01/19/2018 02:14 AM, yawniek wrote:
 Could you please summarize what needs to be done and in what 
 order. I would be happy to take active part in this effort.


well,  my personal opinion this should be designed/decided on a 
green field and not again make the mistake to try to include too 
much that is just laying around. i think vibe.d's eventcore may 
(or may not) be too opinionated and providing too much:

1a. define an overal strategy how/where protocols should be 
implemented and whether to focus on speed or "elegant" 
implementations.

1b. define how high performance servers should ideally be 
implemented and how that works together with D's concurrency 
models. plus provide the necessary things such as streams/ranges. 
ideally endorse/include/create one primary eventloop 
implementation or interface. possibly rusts tokyo.rs could be an 
inspiration?
https://tokio.rs/docs/getting-started/tls/

2a. create reference implementations for a http parser or e.g. 
port something like picohttpparser 
https://github.com/h2o/picohttpparser but this should adhere to 
the approach defined in 1.


3. implement crypto functions. e.g. port picotls ( 
https://github.com/h2o/picotls ) but also provide/maintain proper 
openssl bindings.

4. implement http/2  grpc

also, think about what happens when the world starts switching to 
quic, the ietf wg makes good progress from what i saw.


a totally different approach/decision would be to say, that D is 
a glue language and one should use C libraries (libuv, openssl, 
picohttpparser). but then i think tooling for integrating those 
should be improved/standardized (e.g. package manager, binding 
automation).
Jan 20 2018
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Friday, 19 January 2018 at 10:14:15 UTC, yawniek wrote:

 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which 
 is not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali


 Problem here is that D lacks a proper http/2 stack.
 unfortunately due to time constraints my libh2o binding project 
 got stuck.


Have you seen Etienne's libhttp2?

http://code.dlang.org/packages/libhttp2
Jan 20 2018
prev sibling next sibling parent reply Andrew Benton <Andrew.Benton675 gmail.com> writes:
On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which is 
 not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali


An http/2 and gRPC solutions is probably necessary with tools 
like linkerd, envoy, and istio if D wants to be competitive in 
service mesh and distributed applications.

http/2 and/or gRPC are both excellent ideas for GSoC 2018.
Jan 21 2018
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:

 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which 
 is not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali


 An http/2 and gRPC solutions is probably necessary with tools 
 like linkerd, envoy, and istio if D wants to be competitive in 
 service mesh and distributed applications.

 http/2 and/or gRPC are both excellent ideas for GSoC 2018.


I would consider them awful in a sense that there is no 
foundation to build them on. At best it will be a self-serving 
artifact poorly fitting with anything else.

There is not even a standard way on handling IO as of yet.
Basically do we want fiber-aware IO or blocking IO or explicit 
async with future/promise?

Do we even have protobuf package?
Jan 21 2018
next sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
 Do we even have protobuf package?


https://github.com/msoucy/dproto
it could receive some attention, there are pending issues

for RPC I've been using msgpackrpc since no gRPC was available. But
would be nice to have gRPC.

NOTE: capnproto is a very interesting newer alternative to protobuf;
https://github.com/capnproto/capnproto-dlang shows:
Missing RPC part of Cap'n Proto.
helping out capnproto project (esp around RPC) could be another idea.

we definitely need a good way to do RPC in D, otherwise hard to
integrate D with other services.


 I would consider them awful in a sense that there is no foundation to build
them on. At best it will be a self-serving artifact poorly fitting with
anything else.


But it would enable using D in places that were not previously
possible (integrating with services); we could imaging providing a
(semi) stable interface for grpc in D code and change implementation
to use better foundations later




On Sun, Jan 21, 2018 at 9:54 PM, Dmitry Olshansky via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:

 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which is not among
 the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali



 An http/2 and gRPC solutions is probably necessary with tools like
 linkerd, envoy, and istio if D wants to be competitive in service mesh and
 distributed applications.

 http/2 and/or gRPC are both excellent ideas for GSoC 2018.



 I would consider them awful in a sense that there is no foundation to build
 them on. At best it will be a self-serving artifact poorly fitting with
 anything else.

 There is not even a standard way on handling IO as of yet.
 Basically do we want fiber-aware IO or blocking IO or explicit async with
 future/promise?

 Do we even have protobuf package?
Jan 21 2018
prev sibling parent Andrew Benton <Andrew.Benton675 gmail.com> writes:
On Monday, 22 January 2018 at 05:54:31 UTC, Dmitry Olshansky 
wrote:

 On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:

 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which 
 is not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali


 An http/2 and gRPC solutions is probably necessary with tools 
 like linkerd, envoy, and istio if D wants to be competitive in 
 service mesh and distributed applications.

 http/2 and/or gRPC are both excellent ideas for GSoC 2018.


 I would consider them awful in a sense that there is no 
 foundation to build them on. At best it will be a self-serving 
 artifact poorly fitting with anything else.


I think it would likely be built on top of vibe or collie instead 
of being standalone, but I see your point.


 There is not even a standard way on handling IO as of yet.
 Basically do we want fiber-aware IO or blocking IO or explicit 
 async with future/promise?


You're right.  It's definitely a problem.  Given ease of use and 
the learning curve, I'd be in favor of fiber-aware IO or 
async/await types.


 Do we even have protobuf package?


We have at least two.  dproto is a package that supports proto2.  
I've used it before and it's really nice.  There's also the 
eponymous protobuf package, which integrates with protoc and can 
also do proto3.  I haven't used it yet.
Jan 22 2018
prev sibling parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which is 
 not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali


I can share a fresh experience from mentoring a student in a 
similar (also RPC) project last summer. We built native D-Bus 
bindings in D based on libasync. The student had had no previous 
experience with D or RPC, and within ~2.5 months of focused work 
she implemented the following:

1. (de)serialization of all D-Bus data types, including the use 
of compile-time reflection to recursively marshall structs, 
arrays, and variants. Except Variant, for which we decided to 
make our own D-Bus-specific tagged union type, all other D-Bus 
types are mapped to built-in D types.
2. A class to connect to the bus via libasync sockets, read the 
incoming messages and dispatch them to the registered handlers, 
and send messages to the bus.
3. Proxy (client) and server class templates that generate all 
the code necessary to make the remote calls look almost like 
local ones (the return value/out arguments are passed to a 
delegate that handles the output instead of being returned 
synchronously).

So, more or less an equivalent of vibe.d's REST interface 
generator, only with fewer customization points.

There were still some opportunities for refactorings and 
optimizations, so I wouldn't consider it production ready. Also, 
some planned features weren't implemented, such as a more 
convenient way for handling signals or allowing transports other 
than unix sockets on libasync. On the other hand, what we have is 
almost 100% covered with unit tests. This not only made adding 
successive layers quite pleasant, as little (if any) debugging of 
previously written stuff was ever necessary, but also helps to 
keep the stuff working as we modify it.

Based on my experience, I'd say that implementing gRPC may be of 
a right size for a GSoC project, as long as you split it into 
smaller components/layers, prioritize them, and focus on having 
at least the basic stuff usable and tested, instead of expecting 
it to cover all usage cases and be heavily optimized.
Jan 22 2018
parent Timothee Cour <thelastmammoth gmail.com> writes:
for grpc, we should add to dproto (which is pretty good protobuf
library for D but lacks grpc) instead of starting from scratch, see
https://github.com/msoucy/dproto/issues/113 [your advice/opinions on
integrating with grpc?]


On Mon, Jan 22, 2018 at 12:24 PM, Adrian Matoga via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:

 I know a project where D could benefit from gRPC in D, which is not among
 the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali



 I can share a fresh experience from mentoring a student in a similar (also
 RPC) project last summer. We built native D-Bus bindings in D based on
 libasync. The student had had no previous experience with D or RPC, and
 within ~2.5 months of focused work she implemented the following:

 1. (de)serialization of all D-Bus data types, including the use of
 compile-time reflection to recursively marshall structs, arrays, and
 variants. Except Variant, for which we decided to make our own
 D-Bus-specific tagged union type, all other D-Bus types are mapped to
 built-in D types.
 2. A class to connect to the bus via libasync sockets, read the incoming
 messages and dispatch them to the registered handlers, and send messages to
 the bus.
 3. Proxy (client) and server class templates that generate all the code
 necessary to make the remote calls look almost like local ones (the return
 value/out arguments are passed to a delegate that handles the output instead
 of being returned synchronously).

 So, more or less an equivalent of vibe.d's REST interface generator, only
 with fewer customization points.

 There were still some opportunities for refactorings and optimizations, so I
 wouldn't consider it production ready. Also, some planned features weren't
 implemented, such as a more convenient way for handling signals or allowing
 transports other than unix sockets on libasync. On the other hand, what we
 have is almost 100% covered with unit tests. This not only made adding
 successive layers quite pleasant, as little (if any) debugging of previously
 written stuff was ever necessary, but also helps to keep the stuff working
 as we modify it.

 Based on my experience, I'd say that implementing gRPC may be of a right
 size for a GSoC project, as long as you split it into smaller
 components/layers, prioritize them, and focus on having at least the basic
 stuff usable and tested, instead of expecting it to cover all usage cases
 and be heavily optimized.
Jan 26 2018