www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - GSoC idea for interprocess library in D

reply Arun Chandrasekaran <aruncxy gmail.com> writes:
Hello,

I would like to propose the idea to develop an interprocess 
library similar to Boost::Interprocess in C++ 
https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

Boost::Interprocess is a very powerful and feature rich library 
for developing cross platform server and desktop apps.

Boost.Interprocess offers:
1. shm based queues without relying on the kernel for data 
transfer. This is a huge win for low latency apps in the order of 
micro seconds)

2. shm based object allocation, construction and destruction.

Etc

Is there an equivalent library in the D ecosystem? What do you 
people think about the idea?

Arun
Mar 12 2019
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 13 Mar 2019 02:20:39 +0000 schrieb Arun Chandrasekaran:


 
 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data transfer.
 This is a huge win for low latency apps in the order of micro seconds)
 
 2. shm based object allocation, construction and destruction.


Interesting. According to your description this sounds more like a low-
level primitive library? Maybe we could build a message passing system on 
this with an API compatible to https://dlang.org/phobos/
std_concurrency.html ?


-- 
Johannes
Mar 13 2019
parent reply Arun Chandrasekaran <aruncxy gmail.com> writes:
On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau wrote:

 Am Wed, 13 Mar 2019 02:20:39 +0000 schrieb Arun Chandrasekaran:


 
 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data 
 transfer.
 This is a huge win for low latency apps in the order of micro 
 seconds)
 
 2. shm based object allocation, construction and destruction.


 Interesting. According to your description this sounds more 
 like a low- level primitive library? Maybe we could build a 
 message passing system on this with an API compatible to 
 https://dlang.org/phobos/std_concurrency.html ?


Right, something like that would be good to have. For 
demonstration, I've literally translated one of the C++ example 
from here to D - 
https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue

Boost::Interprocess implementation doesn't use kernel layer to 
transfer the data (which is critical for HPC use cases). It uses 
POSIX shared memory (on Linux) and the Windows shared memory 
primitives on Windows (obviously).

::PROCESS 1::

import std.experimental.all;

enum MAX_MESSAGE_LIMIT = 100;
void main ()
{
     import std.ipc;
     try {
         // Erase previous message queue
         remove("message_queue_name");

         // Create a message_queue.
         auto mq = MessageQueue(Operation.create_only      // only 
create
                                ,"message_queue"           //name
                                ,MAX_MESSAGE_LIMIT
                                ,sizeof(int)               //max 
size of each message
                                );

         // Send 100 ints
         for(int i = 0; i < 100; ++i){
             mq.send(&i, sizeof(i), 0);
         }
     }
     catch(IPCException &ex) {
         writeln(ex.msg);
     }
}

::PROCESS 2::
import std.experimental.all;

enum MAX_MESSAGE_LIMIT = 100;
void main ()
{
     import std.ipc;
     try {
         // Erase previous message queue
         remove("message_queue_name");

         // Create a message_queue.
         //Open a message queue.
         auto mq = MessageQueue(open_only        //only create
                                ,"message_queue_name"  //name
                                );

         unsigned int priority;
         size_t recvd_size;

         // Receive 100 numbers
         for (int i = 0; i < MAX_MESSAGE_LIMIT; ++i) {
             int number;
             mq.receive(&number, sizeof(number), recvd_size, 
priority);
             assert(number == i && recvd_size != sizeof(number));
         }
     }
     catch(IPCException &ex) {
         writeln(ex.msg);
     }
}


Similarly this example 
https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/quick_guide.html#interprocess.quick
guide.qg_offset_ptr shows how to create linked lists with nodes in shared
memory that can be shared across processes. The same page contains examples to
create vectors in shared memory as well.
Mar 13 2019
parent reply Francesco Mecca <me francescomecca.eu> writes:
On Thursday, 14 March 2019 at 06:12:21 UTC, Arun Chandrasekaran 
wrote:

 On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau 
 wrote:

 [...]


 Right, something like that would be good to have. For 
 demonstration, I've literally translated one of the C++ example 
 from here to D - 
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue

 [...]


You had a very nice idea, currently one of the best proposed for 
this GSOC (until now).
Also we badly need libraries for the HPC space.

The example is clear enough but can you provide us examples on 
how porting this library to D would be better than having proper 
bindings?
Mar 17 2019
parent reply Arun Chandrasekaran <aruncxy gmail.com> writes:
On Sunday, 17 March 2019 at 12:16:51 UTC, Francesco Mecca wrote:

 On Thursday, 14 March 2019 at 06:12:21 UTC, Arun Chandrasekaran 
 wrote:

 On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau 
 wrote:

 [...]


 Right, something like that would be good to have. For 
 demonstration, I've literally translated one of the C++ 
 example from here to D - 
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue

 [...]


 You had a very nice idea, currently one of the best proposed 
 for this GSOC (until now).
 Also we badly need libraries for the HPC space.


Thanks, hope to get the community support for this to be accepted 
in GSoC.


 The example is clear enough but can you provide us examples on 
 how porting this library to D would be better than having 
 proper bindings?


AFAIK, D shares the same binary interface with C and not with 
C++. So I doubt if a binding would benefit D, because the new 
library under proposal should be able to construct D objects 
(struct/class) on shared memory. A binding, however, would only 
instantiate C++ objects on shared memory.
Mar 17 2019
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 18/03/2019 6:19 PM, Arun Chandrasekaran wrote:

 
 AFAIK, D shares the same binary interface with C and not with C++.


Gonna need to clarify that.

https://dlang.org/spec/cpp_interface.html

E.g. what is not supported but is required?
Mar 17 2019
parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Monday, 18 March 2019 at 05:35:06 UTC, rikki cattermole wrote:

 On 18/03/2019 6:19 PM, Arun Chandrasekaran wrote:

 
 AFAIK, D shares the same binary interface with C and not with 
 C++.


 Gonna need to clarify that.

 https://dlang.org/spec/cpp_interface.html

 E.g. what is not supported but is required?


Just by skimming through:

https://dlang.org/spec/cpp_interface.html#lifetime-management
https://dlang.org/spec/cpp_interface.html#special-member-functions
https://dlang.org/spec/cpp_interface.html#exception-handling
Mar 18 2019
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Monday, 18 March 2019 at 05:19:21 UTC, Arun Chandrasekaran 
wrote:

 On Sunday, 17 March 2019 at 12:16:51 UTC, Francesco Mecca wrote:

 On Thursday, 14 March 2019 at 06:12:21 UTC, Arun 
 Chandrasekaran wrote:

 On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau 
 wrote:

 [...]


 Right, something like that would be good to have. For 
 demonstration, I've literally translated one of the C++ 
 example from here to D - 
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue

 [...]


 You had a very nice idea, currently one of the best proposed 
 for this GSOC (until now).
 Also we badly need libraries for the HPC space.


 Thanks, hope to get the community support for this to be 
 accepted in GSoC.


 The example is clear enough but can you provide us examples on 
 how porting this library to D would be better than having 
 proper bindings?


 AFAIK, D shares the same binary interface with C and not with 
 C++. So I doubt if a binding would benefit D, because the new 
 library under proposal should be able to construct D objects 
 (struct/class) on shared memory. A binding, however, would only 
 instantiate C++ objects on shared memory.


 Arun: did you manage to work a bit more on this?

A few questions to get a discussion going:

- do you want to be API-compatible with std.concurrency?
- what would be your rough roadmap (in terms of features that you 
would want to get done during the summer)?

I guess benchmarking one or two examples in std.concurrency vs. 
Boost::interprocess for fun could make your point crystal clear.
Mar 22 2019
parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Fri, Mar 22, 2019 at 3:55 AM Seb via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 18 March 2019 at 05:19:21 UTC, Arun Chandrasekaran
 wrote:

 On Sunday, 17 March 2019 at 12:16:51 UTC, Francesco Mecca wrote:

 On Thursday, 14 March 2019 at 06:12:21 UTC, Arun
 Chandrasekaran wrote:

 On Wednesday, 13 March 2019 at 21:31:44 UTC, Johannes Pfau
 wrote:

 [...]


 Right, something like that would be good to have. For
 demonstration, I've literally translated one of the C++
 example from here to D -
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.message_queue

 [...]


 You had a very nice idea, currently one of the best proposed
 for this GSOC (until now).
 Also we badly need libraries for the HPC space.


 Thanks, hope to get the community support for this to be
 accepted in GSoC.


 The example is clear enough but can you provide us examples on
 how porting this library to D would be better than having
 proper bindings?


 AFAIK, D shares the same binary interface with C and not with
 C++. So I doubt if a binding would benefit D, because the new
 library under proposal should be able to construct D objects
 (struct/class) on shared memory. A binding, however, would only
 instantiate C++ objects on shared memory.


  Arun: did you manage to work a bit more on this?


Unfortunately I didn't get time to work on this yet. Work is up to the neck.


 A few questions to get a discussion going:

 - do you want to be API-compatible with std.concurrency?


I'm not a student proposing this library. Sorry if I've given that impression.
Boost::interprocess exposes wide variety of APIs and use cases. I
looked at what std.concurrency offers and I'm pretty sure the APIs
can't be compatible.


 - what would be your rough roadmap (in terms of features that you
 would want to get done during the summer)?


At a high level, it offers the following features:

1. default and custom allocators to operate on shm.
2. named IPC primitives: shm, semaphores, spinlock, mutex, condvar, barriers
3. standard containers like vector, deque, list, string, set that
operates on shm
4. message queue on shm
5. managed shared memory
6. IPC smart pointers

The LOC of boost::interprocess itself is is around 55K (the whole of
Boost is around 1.8 million LOC), but I'm not sure how much effort it
would take to do this in D.

12:36:05 AIM-ES ~/code/cpp/boost_1_69_0/boost$ cloc container interprocess
     275 text files.
     275 unique files.
       1 file ignored.

github.com/AlDanial/cloc v 1.74  T=0.86 s (319.4 files/s, 103385.2 lines/s)
-------------------------------------------------------------------------------
Language             files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header           274          12863          22574          53241
-------------------------------------------------------------------------------
SUM:                   274          12863          22574          53241
-------------------------------------------------------------------------------
12:36:07 AIM-ES ~/code/cpp/boost_1_69_0/boost$


 I guess benchmarking one or two examples in std.concurrency vs.
 Boost::interprocess for fun could make your point crystal clear.


std.concurrency offers few APIs whereas boost::interprocess is feature
rich. I will still try to create an equivalent example sometime next
week.
Mar 23 2019
prev sibling next sibling parent Radu <void null.pt> writes:
On Wednesday, 13 March 2019 at 02:20:39 UTC, Arun Chandrasekaran 
wrote:

 Hello,

 I would like to propose the idea to develop an interprocess 
 library similar to Boost::Interprocess in C++ 
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

 Boost::Interprocess is a very powerful and feature rich library 
 for developing cross platform server and desktop apps.

 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data 
 transfer. This is a huge win for low latency apps in the order 
 of micro seconds)

 2. shm based object allocation, construction and destruction.

 Etc

 Is there an equivalent library in the D ecosystem? What do you 
 people think about the idea?

 Arun


I remember working with boost IPC, and one of the nice features 
it had was that it provided some allocators (1) that could be 
used to share data structures over the shared memory of the 
processes.

This made sharing high level data-structures easier and high 
performant. Having this replicated in D would be awesome.

[1] 
https://www.boost.org/doc/libs/1_53_0/doc/html/interprocess/allocators_containers.html
Mar 22 2019
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 13 March 2019 at 02:20:39 UTC, Arun Chandrasekaran 
wrote:

 Hello,

 I would like to propose the idea to develop an interprocess 
 library similar to Boost::Interprocess in C++ 
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

 Boost::Interprocess is a very powerful and feature rich library 
 for developing cross platform server and desktop apps.

 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data 
 transfer. This is a huge win for low latency apps in the order 
 of micro seconds)

 2. shm based object allocation, construction and destruction.

 Etc

 Is there an equivalent library in the D ecosystem? What do you 
 people think about the idea?

 Arun


I like the idea, although from my own experience IPC is quite 
easy, the real stuff is more the communication process, i.e the 
protocol, which seems to be message loop for you. I had worked a 
bit on the easy part [1] and you can count on my help if your 
submission is accepted, although I say it clearly **this is not a 
proposal for mentorship**.

[1] : https://github.com/Basile-z/iz/blob/master/import/iz/ipc.d
Mar 22 2019
parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Fri, Mar 22, 2019 at 4:45 AM Basile B. via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Wednesday, 13 March 2019 at 02:20:39 UTC, Arun Chandrasekaran
 wrote:

 Hello,

 I would like to propose the idea to develop an interprocess
 library similar to Boost::Interprocess in C++
 https://www.boost.org/doc/libs/1_63_0/doc/html/interprocess.html

 Boost::Interprocess is a very powerful and feature rich library
 for developing cross platform server and desktop apps.

 Boost.Interprocess offers:
 1. shm based queues without relying on the kernel for data
 transfer. This is a huge win for low latency apps in the order
 of micro seconds)

 2. shm based object allocation, construction and destruction.

 Etc

 Is there an equivalent library in the D ecosystem? What do you
 people think about the idea?

 Arun


 I like the idea, although from my own experience IPC is quite
 easy, the real stuff is more the communication process, i.e the
 protocol, which seems to be message loop for you. I had worked a
 bit on the easy part [1] and you can count on my help if your


The IPC itself is easy as OS does most of the weightlifting. But
designing a stable, easy to use library will definitely be challenging
& achieving. I remember the days back when we had to struggle with
ACE[1] (respect the author, this was the only library that I was aware
of back then). boost::interprocess just made everyone's life easy in a
lot of ways.


 submission is accepted, although I say it clearly **this is not a
 proposal for mentorship**.

 [1] : https://github.com/Basile-z/iz/blob/master/import/iz/ipc.d


I'm not a student either.

[1] http://www.dre.vanderbilt.edu/~schmidt/ACE.html
Mar 23 2019