www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - GSoC idea for interprocess library in D

reply Arun Chandrasekaran <aruncxy gmail.com> writes:
Hello,

I would like to propose the idea to develop an interprocess 
library similar to Boost::Interprocess in C++ 
https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

Boost::Interprocess is a very powerful and feature rich library 
for developing cross platform server and desktop apps.

Boost.Interprocess offers:
1. shm based queues without relying on the kernel for data 
transfer. This is a huge win for low latency apps in the order of 
micro seconds)

2. shm based object allocation, construction and destruction.

Etc

Is there an equivalent library in the D ecosystem? What do you 
people think about the idea?

Arun
Mar 12
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 13 Mar 2019 02:20:39 +0000 schrieb Arun Chandrasekaran:

 
 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data transfer.
 This is a huge win for low latency apps in the order of micro seconds)
 
 2. shm based object allocation, construction and destruction.
Interesting. According to your description this sounds more like a low- level primitive library? Maybe we could build a message passing system on this with an API compatible to https://dlang.org/phobos/ std_concurrency.html ? -- Johannes
Mar 13
parent reply Arun Chandrasekaran <aruncxy gmail.com> writes:
On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau wrote:
 Am Wed, 13 Mar 2019 02:20:39 +0000 schrieb Arun Chandrasekaran:

 
 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data 
 transfer.
 This is a huge win for low latency apps in the order of micro 
 seconds)
 
 2. shm based object allocation, construction and destruction.
Interesting. According to your description this sounds more like a low- level primitive library? Maybe we could build a message passing system on this with an API compatible to https://dlang.org/phobos/std_concurrency.html ?
Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue Boost::Interprocess implementation doesn't use kernel layer to transfer the data (which is critical for HPC use cases). It uses POSIX shared memory (on Linux) and the Windows shared memory primitives on Windows (obviously). ::PROCESS 1:: import std.experimental.all; enum MAX_MESSAGE_LIMIT = 100; void main () { import std.ipc; try { // Erase previous message queue remove("message_queue_name"); // Create a message_queue. auto mq = MessageQueue(Operation.create_only // only create ,"message_queue" //name ,MAX_MESSAGE_LIMIT ,sizeof(int) //max size of each message ); // Send 100 ints for(int i = 0; i < 100; ++i){ mq.send(&i, sizeof(i), 0); } } catch(IPCException &ex) { writeln(ex.msg); } } ::PROCESS 2:: import std.experimental.all; enum MAX_MESSAGE_LIMIT = 100; void main () { import std.ipc; try { // Erase previous message queue remove("message_queue_name"); // Create a message_queue. //Open a message queue. auto mq = MessageQueue(open_only //only create ,"message_queue_name" //name ); unsigned int priority; size_t recvd_size; // Receive 100 numbers for (int i = 0; i < MAX_MESSAGE_LIMIT; ++i) { int number; mq.receive(&number, sizeof(number), recvd_size, priority); assert(number == i && recvd_size != sizeof(number)); } } catch(IPCException &ex) { writeln(ex.msg); } } Similarly this example https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/quick_guide.html#interprocess.quick guide.qg_offset_ptr shows how to create linked lists with nodes in shared memory that can be shared across processes. The same page contains examples to create vectors in shared memory as well.
Mar 13
parent reply Francesco Mecca <me francescomecca.eu> writes:
On Thursday, 14 March 2019 at 06:12:21 UTC, Arun Chandrasekaran 
wrote:
 On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau 
 wrote:
 [...]
Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue [...]
You had a very nice idea, currently one of the best proposed for this GSOC (until now). Also we badly need libraries for the HPC space. The example is clear enough but can you provide us examples on how porting this library to D would be better than having proper bindings?
Mar 17
parent reply Arun Chandrasekaran <aruncxy gmail.com> writes:
On Sunday, 17 March 2019 at 12:16:51 UTC, Francesco Mecca wrote:
 On Thursday, 14 March 2019 at 06:12:21 UTC, Arun Chandrasekaran 
 wrote:
 On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau 
 wrote:
 [...]
Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue [...]
You had a very nice idea, currently one of the best proposed for this GSOC (until now). Also we badly need libraries for the HPC space.
Thanks, hope to get the community support for this to be accepted in GSoC.
 The example is clear enough but can you provide us examples on 
 how porting this library to D would be better than having 
 proper bindings?
AFAIK, D shares the same binary interface with C and not with C++. So I doubt if a binding would benefit D, because the new library under proposal should be able to construct D objects (struct/class) on shared memory. A binding, however, would only instantiate C++ objects on shared memory.
Mar 17
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 18/03/2019 6:19 PM, Arun Chandrasekaran wrote:
 
 AFAIK, D shares the same binary interface with C and not with C++.
Gonna need to clarify that. https://dlang.org/spec/cpp_interface.html E.g. what is not supported but is required?
Mar 17
parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Monday, 18 March 2019 at 05:35:06 UTC, rikki cattermole wrote:
 On 18/03/2019 6:19 PM, Arun Chandrasekaran wrote:
 
 AFAIK, D shares the same binary interface with C and not with 
 C++.
Gonna need to clarify that. https://dlang.org/spec/cpp_interface.html E.g. what is not supported but is required?
Just by skimming through: https://dlang.org/spec/cpp_interface.html#lifetime-management https://dlang.org/spec/cpp_interface.html#special-member-functions https://dlang.org/spec/cpp_interface.html#exception-handling
Mar 18
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Monday, 18 March 2019 at 05:19:21 UTC, Arun Chandrasekaran 
wrote:
 On Sunday, 17 March 2019 at 12:16:51 UTC, Francesco Mecca wrote:
 On Thursday, 14 March 2019 at 06:12:21 UTC, Arun 
 Chandrasekaran wrote:
 On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau 
 wrote:
 [...]
Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue [...]
You had a very nice idea, currently one of the best proposed for this GSOC (until now). Also we badly need libraries for the HPC space.
Thanks, hope to get the community support for this to be accepted in GSoC.
 The example is clear enough but can you provide us examples on 
 how porting this library to D would be better than having 
 proper bindings?
AFAIK, D shares the same binary interface with C and not with C++. So I doubt if a binding would benefit D, because the new library under proposal should be able to construct D objects (struct/class) on shared memory. A binding, however, would only instantiate C++ objects on shared memory.
Arun: did you manage to work a bit more on this? A few questions to get a discussion going: - do you want to be API-compatible with std.concurrency? - what would be your rough roadmap (in terms of features that you would want to get done during the summer)? I guess benchmarking one or two examples in std.concurrency vs. Boost::interprocess for fun could make your point crystal clear.
Mar 22
parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Fri, Mar 22, 2019 at 3:55 AM Seb via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 18 March 2019 at 05:19:21 UTC, Arun Chandrasekaran
 wrote:
 On Sunday, 17 March 2019 at 12:16:51 UTC, Francesco Mecca wrote:
 On Thursday, 14 March 2019 at 06:12:21 UTC, Arun
 Chandrasekaran wrote:
 On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau
 wrote:
 [...]
Right, something like that would be good to have. For demonstration, I've literally translated one of the C++ example from here to D - https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue [...]
You had a very nice idea, currently one of the best proposed for this GSOC (until now). Also we badly need libraries for the HPC space.
Thanks, hope to get the community support for this to be accepted in GSoC.
 The example is clear enough but can you provide us examples on
 how porting this library to D would be better than having
 proper bindings?
AFAIK, D shares the same binary interface with C and not with C++. So I doubt if a binding would benefit D, because the new library under proposal should be able to construct D objects (struct/class) on shared memory. A binding, however, would only instantiate C++ objects on shared memory.
Arun: did you manage to work a bit more on this?
Unfortunately I didn't get time to work on this yet. Work is up to the neck.
 A few questions to get a discussion going:

 - do you want to be API-compatible with std.concurrency?
I'm not a student proposing this library. Sorry if I've given that impression. Boost::interprocess exposes wide variety of APIs and use cases. I looked at what std.concurrency offers and I'm pretty sure the APIs can't be compatible.
 - what would be your rough roadmap (in terms of features that you
 would want to get done during the summer)?
At a high level, it offers the following features: 1. default and custom allocators to operate on shm. 2. named IPC primitives: shm, semaphores, spinlock, mutex, condvar, barriers 3. standard containers like vector, deque, list, string, set that operates on shm 4. message queue on shm 5. managed shared memory 6. IPC smart pointers The LOC of boost::interprocess itself is is around 55K (the whole of Boost is around 1.8 million LOC), but I'm not sure how much effort it would take to do this in D. 12:36:05 AIM-ES ~/code/cpp/boost_1_69_0/boost$ cloc container interprocess 275 text files. 275 unique files. 1 file ignored. github.com/AlDanial/cloc v 1.74 T=0.86 s (319.4 files/s, 103385.2 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- C/C++ Header 274 12863 22574 53241 ------------------------------------------------------------------------------- SUM: 274 12863 22574 53241 ------------------------------------------------------------------------------- 12:36:07 AIM-ES ~/code/cpp/boost_1_69_0/boost$
 I guess benchmarking one or two examples in std.concurrency vs.
 Boost::interprocess for fun could make your point crystal clear.
std.concurrency offers few APIs whereas boost::interprocess is feature rich. I will still try to create an equivalent example sometime next week.
Mar 23
prev sibling next sibling parent Radu <void null.pt> writes:
On Wednesday, 13 March 2019 at 02:20:39 UTC, Arun Chandrasekaran 
wrote:
 Hello,

 I would like to propose the idea to develop an interprocess 
 library similar to Boost::Interprocess in C++ 
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

 Boost::Interprocess is a very powerful and feature rich library 
 for developing cross platform server and desktop apps.

 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data 
 transfer. This is a huge win for low latency apps in the order 
 of micro seconds)

 2. shm based object allocation, construction and destruction.

 Etc

 Is there an equivalent library in the D ecosystem? What do you 
 people think about the idea?

 Arun
I remember working with boost IPC, and one of the nice features it had was that it provided some allocators (1) that could be used to share data structures over the shared memory of the processes. This made sharing high level data-structures easier and high performant. Having this replicated in D would be awesome. [1] https://www.boost.org/doc/libs/1_53_0/doc/html/interprocess/allocators_containers.html
Mar 22
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 13 March 2019 at 02:20:39 UTC, Arun Chandrasekaran 
wrote:
 Hello,

 I would like to propose the idea to develop an interprocess 
 library similar to Boost::Interprocess in C++ 
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

 Boost::Interprocess is a very powerful and feature rich library 
 for developing cross platform server and desktop apps.

 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data 
 transfer. This is a huge win for low latency apps in the order 
 of micro seconds)

 2. shm based object allocation, construction and destruction.

 Etc

 Is there an equivalent library in the D ecosystem? What do you 
 people think about the idea?

 Arun
I like the idea, although from my own experience IPC is quite easy, the real stuff is more the communication process, i.e the protocol, which seems to be message loop for you. I had worked a bit on the easy part [1] and you can count on my help if your submission is accepted, although I say it clearly **this is not a proposal for mentorship**. [1] : https://github.com/Basile-z/iz/blob/master/import/iz/ipc.d
Mar 22
parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Fri, Mar 22, 2019 at 4:45 AM Basile B. via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Wednesday, 13 March 2019 at 02:20:39 UTC, Arun Chandrasekaran
 wrote:
 Hello,

 I would like to propose the idea to develop an interprocess
 library similar to Boost::Interprocess in C++
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

 Boost::Interprocess is a very powerful and feature rich library
 for developing cross platform server and desktop apps.

 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data
 transfer. This is a huge win for low latency apps in the order
 of micro seconds)

 2. shm based object allocation, construction and destruction.

 Etc

 Is there an equivalent library in the D ecosystem? What do you
 people think about the idea?

 Arun
I like the idea, although from my own experience IPC is quite easy, the real stuff is more the communication process, i.e the protocol, which seems to be message loop for you. I had worked a bit on the easy part [1] and you can count on my help if your
The IPC itself is easy as OS does most of the weightlifting. But designing a stable, easy to use library will definitely be challenging & achieving. I remember the days back when we had to struggle with ACE[1] (respect the author, this was the only library that I was aware of back then). boost::interprocess just made everyone's life easy in a lot of ways.
 submission is accepted, although I say it clearly **this is not a
 proposal for mentorship**.

 [1] : https://github.com/Basile-z/iz/blob/master/import/iz/ipc.d
I'm not a student either. [1] http://www.dre.vanderbilt.edu/~schmidt/ACE.html
Mar 23