www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Cap'n Proto for D v0.1.2

reply Thomas Brix Larsen <brix brix-verden.dk> writes:
"Cap’n Proto is an insanely fast data interchange format and 
capability-based RPC system. Think JSON, except binary. Or think 
Protocol Buffers, except faster."

This is the initial public release of my optimized port of the 
Java implementation of Cap'n Proto.

State:
* Passes Cap'n Proto testsuite.
* Optimized. Just a little slower than the official C++ 
implementation (see benchmarks on github).
* Missing RPC part of Cap'n Proto.

http://code.dlang.org/packages/capnproto-dlang
https://github.com/ThomasBrixLarsen/capnproto-dlang
Apr 18 2017
next sibling parent reply Swoorup Joshi <swoorupjoshi gmail.com> writes:
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
wrote:

 "Cap’n Proto is an insanely fast data interchange format and 
 capability-based RPC system. Think JSON, except binary. Or 
 think Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the 
 Java implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++ 
 implementation (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang


Java?? Yikes
Apr 18 2017
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 4/18/17 9:14 PM, Swoorup Joshi wrote:

 On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen wrote:

 "Cap’n Proto is an insanely fast data interchange format and
 capability-based RPC system. Think JSON, except binary. Or think
 Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the Java
 implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++ implementation
 (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang


 Java?? Yikes


Risking a flamewar but what's wrong with Java?

---
Dmitry Olshansky
Apr 18 2017
parent Jacob Carlborg <doob me.com> writes:
On 2017-04-18 23:08, Dmitry Olshansky wrote:


 Risking a flamewar but what's wrong with Java?


I don't like any language that force me to write in a particular style 
or paradigm. Because all problems cannot be solved (or not in a good 
way) in the same way. That said, my D code is quite heavily influenced 
by Java.

-- 
/Jacob Carlborg
Apr 19 2017
prev sibling next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
wrote:

 "Cap’n Proto is an insanely fast data interchange format and 
 capability-based RPC system. Think JSON, except binary. Or 
 think Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the 
 Java implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++ 
 implementation (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang


Great Job!

I'm following Cap'n Proto, and it's very very interesting...
I would love also the RPC part, maybe based on Vibe... have you 
any plan to implement that?

I'm really curious to try it!

---
Paolo
Apr 18 2017
parent Thomas Brix Larsen <brix brix-verden.dk> writes:
On Tuesday, 18 April 2017 at 20:46:33 UTC, Paolo Invernizzi wrote:

 On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
 wrote:

 "Cap’n Proto is an insanely fast data interchange format and 
 capability-based RPC system. Think JSON, except binary. Or 
 think Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the 
 Java implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++ 
 implementation (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang


 Great Job!

 I'm following Cap'n Proto, and it's very very interesting...
 I would love also the RPC part, maybe based on Vibe... have you 
 any plan to implement that?

 I'm really curious to try it!

 ---
 Paolo


Well personally I have no need for the RPC part. Though if it 
gets added to the Java implementation of Cap'n Proto I will 
probably port it.

The D implementation works well. I developed it for a hobby 
project. Game server in D. Client in Unity. At work we have been 
using the C++ Cap'n Proto for IPC for a while. Recently we have 
also started using the D implementation for configuration files.
Apr 18 2017
prev sibling next sibling parent reply Jay Norwood <jayn prismnet.com> writes:
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
wrote:

 "Cap’n Proto is an insanely fast data interchange format and 
 capability-based RPC system. Think JSON, except binary. Or 
 think Protocol Buffers, except faster."


The features below, from the capnproto.org description, interest 
me.  However a MappedByteBuffer would be used for the mmap 
feature in java.

https://capnproto.org/

mmap: Read a large Cap’n Proto file by memory-mapping it. The OS 
won’t even read in the parts that you don’t access.

Inter-language communication: Calling C++ code from, say, Java or 
Python tends to be painful or slow. With Cap’n Proto, the two 
languages can easily operate on the same in-memory data structure.

Inter-process communication: Multiple processes running on the 
same machine can share a Cap’n Proto message via shared memory. 
No need to pipe data through the kernel. Calling another process 
can be just as fast and easy as calling another thread.

This info from stackoverflow also seems to imply that 
MappedByteBuffer would be required for some of the capnproto 
features.  So, could you explain a little more about what are the 
capabilities of the current d library implementation, with just 
the ByteBuffer implemented from the java nio code?  Thanks, Jay

http://stackoverflow.com/questions/29361058/read-proto-partly-instead-of-full-parsing-in-java

'If you are willing to consider using a different protocol 
framework, Cap'n Proto is extremely similar in design to 
Protobufs but features in the ability to read only the part of 
the message you care about. Cap'n Proto incurs no overhead for 
the fields you don't examine, other than obviously the bandwidth 
and memory to receive the raw message bytes. If you are reading 
from a file, and you use memory mapping (MappedByteBuffer in 
Java), then only the parts of the message you actually use will 
be read from disk.'
Apr 19 2017
parent reply Thomas Brix Larsen <brix brix-verden.dk> writes:
On Wednesday, 19 April 2017 at 16:34:02 UTC, Jay Norwood wrote:

 [...]

 This info from stackoverflow also seems to imply that 
 MappedByteBuffer would be required for some of the capnproto 
 features.  So, could you explain a little more about what are 
 the capabilities of the current d library implementation, with 
 just the ByteBuffer implemented from the java nio code?  
 Thanks, Jay
 
 [...]


The port of ByteBuffer just wraps a slice when reading in D.

Take a look at FileDescriptor[1]. It is a class I've added to 
support read/write using File from std.stdio. You can create a 
similar streamer using std.mmfile. I believe that this would be 
enough for memory mapped reading.

[1]: 
https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/capnproto/FileDescriptor.d
Apr 19 2017
parent reply Jay Norwood <jayn prismnet.com> writes:
On Wednesday, 19 April 2017 at 16:52:14 UTC, Thomas Brix Larsen 
wrote:

 Take a look at FileDescriptor[1]. It is a class I've added to 
 support read/write using File from std.stdio. You can create a 
 similar streamer using std.mmfile. I believe that this would be 
 enough for memory mapped reading.

 [1]: 
 https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/capnproto/FileDescriptor.d


Ok, thanks. I took a look at several capnproto implementations 
just now, and didn't see any tests for a mmap 'feature'.  The 
roadmap doc below indicates it doesn't exist, and perhaps there 
are some details yet to be resolved to make it 'friendly' for a 
mmap.

https://capnproto.org/roadmap.html

mmap-friendly mutable storage format: Define a standard storage 
format that is friendly to mmap-based use while allowing 
modification. (With the current serialization format, mmap is 
only useful for read-only structures.) Possibly based on the ORM 
interface, updates only possible at the granularity of a whole 
ORM entry.


In java the MappedByteBuffer can be used with a RandomAccessFile 
channel, and then the accesses of the loaded map can be random, 
without requiring sequential accesses of the whole mapped file. 
So java nio already has some higher level classes in place that 
would make it easier to develop a first implementation of the 
mmap features.
Apr 19 2017
parent reply Thomas Brix Larsen <brix brix-verden.dk> writes:
On Wednesday, 19 April 2017 at 18:24:46 UTC, Jay Norwood wrote:

 [...]

 Ok, thanks. I took a look at several capnproto implementations 
 just now, and didn't see any tests for a mmap 'feature'.  The 
 roadmap doc below indicates it doesn't exist, and perhaps there 
 are some details yet to be resolved to make it 'friendly' for a 
 mmap.


But reading using random access *is* a feature of Cap'n Proto. So 
when reading a memory mapped Cap'n Proto file, getters will be 
faster if you use it in a non-sequential way.

The format of Cap'n Proto doesn't currently support *writing* to 
a memory mapped file.
Apr 19 2017
parent Thomas Brix Larsen <brix brix-verden.dk> writes:
On Wednesday, 19 April 2017 at 19:30:03 UTC, Thomas Brix Larsen 
wrote:

 On Wednesday, 19 April 2017 at 18:24:46 UTC, Jay Norwood wrote:

 [...]

 Ok, thanks. I took a look at several capnproto implementations 
 just now, and didn't see any tests for a mmap 'feature'.  The 
 roadmap doc below indicates it doesn't exist, and perhaps 
 there are some details yet to be resolved to make it 
 'friendly' for a mmap.


 But reading using random access *is* a feature of Cap'n Proto. 
 So when reading a memory mapped Cap'n Proto file, getters will 
 be faster if you use it in a non-sequential way.


I have made some minor changes to make adaptors responsible for 
allocating buffers. This made it possible to add a new adaptor 
MemoryMapped and a sample[1] to show it off.

I have also verified that no buffer allocations happen while 
reading using MemoryMapped.

Note that writing using memory mapping is still not possible.

[1]: 
https://github.com/ThomasBrixLarsen/capnproto-dlang/blob/master/source/samples/mmap.d
Apr 20 2017
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d-announce writes:
On Tuesday, April 18, 2017 18:09:54 Thomas Brix Larsen via Digitalmars-d-
announce wrote:

 "Cap’n Proto is an insanely fast data interchange format and
 capability-based RPC system. Think JSON, except binary. Or think
 Protocol Buffers, except faster."

 This is the initial public release of my optimized port of the
 Java implementation of Cap'n Proto.

 State:
 * Passes Cap'n Proto testsuite.
 * Optimized. Just a little slower than the official C++
 implementation (see benchmarks on github).
 * Missing RPC part of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang


LOL. Oh, I remember a coworker bringing up this library. It's the one that
the site claimed was infinitely faster than protocol buffers (IIRC, because
of some work that protocol buffers does that Captain Proto skips entirely).

- Jonathan M Davis
Apr 20 2017
parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 20 April 2017 at 18:25:03 UTC, Jonathan M Davis 
wrote:

 On Tuesday, April 18, 2017 18:09:54 Thomas Brix Larsen via 
 Digitalmars-d- announce wrote:

 LOL. Oh, I remember a coworker bringing up this library. It's 
 the one that the site claimed was infinitely faster than 
 protocol buffers (IIRC, because of some work that protocol 
 buffers does that Captain Proto skips entirely).


Exactly!
Well, "Captain Proto" knows what he does, as he was the man that 
wrote google protobuf.

The same way of working of Cap'n Proto is what Google introduced 
later in FlatBuffer...
but it's missing totally all the good stuff taken from e-lang, 
that in my opinion is a big plus.

Don't take me wrong, I know that a lot is to be implemented, 
today, but I love man with a clean path towards brilliant ideas 
[1]

[1] https://capnproto.org/rpc.html

/Paolo
Apr 20 2017
prev sibling parent Johan Engelen <j j.nl> writes:
On Tuesday, 18 April 2017 at 18:09:54 UTC, Thomas Brix Larsen 
wrote:

 This is the initial public release of my optimized port of the 
 Java implementation of Cap'n Proto.

 http://code.dlang.org/packages/capnproto-dlang
 https://github.com/ThomasBrixLarsen/capnproto-dlang


Hi Thomas,
   Great that you are working on this.

I strongly advise you to advertise your work on this page: 
https://capnproto.org/otherlang.html

Cheers,
   Johan
Jun 16 2017