www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Is there an efficient byte buffer queue?

reply John Burton <john.burton jbmail.com> writes:
My use case is sending data to a socket.

One part of my program generates blocks of bytes, and the socket 
part tries to send them to the socket and then removes from the 
queue the number that got sent.

I am currently using a byte[] and using concatenation and slicing 
to maintain the queue but this seems like it will do many 
unnecessary copies, allocations etc. than are needed. I would do 
much better to maintain a fixed size buffer and maintain read and 
write positions etc.

I'm happy to write my own fixed sized buffer queue for this, but 
just wanted to ask if there was anything in the standard library 
that could be used to do this so I'm not reinventing it all? I 
don't need thread safety for this case. (I know i could probably 
use vibe-d to implement my whole socket sender but don't want to 
in this case for .... reasons)
Oct 08 2018
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 8 October 2018 at 09:39:55 UTC, John Burton wrote:

 My use case is sending data to a socket.

 One part of my program generates blocks of bytes, and the 
 socket part tries to send them to the socket and then removes 
 from the queue the number that got sent.

 [...]


Try searching for "circular buffer". I'm sure 
http://code.dlang.org/packages/iopipe has them in some form but I 
can't find them with a cursory search.
Oct 08 2018
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/8/18 6:31 AM, Nicholas Wilson wrote:

 On Monday, 8 October 2018 at 09:39:55 UTC, John Burton wrote:

 My use case is sending data to a socket.

 One part of my program generates blocks of bytes, and the socket part 
 tries to send them to the socket and then removes from the queue the 
 number that got sent.

 [...]


 
 Try searching for "circular buffer". I'm sure 
 http://code.dlang.org/packages/iopipe has them in some form but I can't 
 find them with a cursory search.


I called it a ring buffer:

http://schveiguy.github.io/iopipe/iopipe/buffer/RingBuffer.html

A couple notes here. The RingBuffer currently only works on Posix 
systems (I haven't had the motivation to dig into doing it on Windows, 
even though the docs mention VirtualAlloc). Also, the typical way one 
would use the RingBuffer is using rbufd in 
http://schveiguy.github.io/iopipe/iopipe/bufpipe/rbufd.html. But In the 
OP's case, if you aren't using iopipes to build a pipeline, that may 
prove more confusing than is worth (iopipe uses pull mechanisms 
exclusively, with somewhat novel mechanisms to enable buffered output). 
The RingBuffer type itself could be an easy-to-use mechanism for his use 
case.

Let me know if you decide to use it and need help. I'm always looking 
for more use cases for iopipe!

-Steve
Oct 09 2018
prev sibling parent Dukc <ajieskola gmail.com> writes:
On Monday, 8 October 2018 at 10:31:33 UTC, Nicholas Wilson wrote:

 Try searching for "circular buffer". I'm sure 
 http://code.dlang.org/packages/iopipe has them in some form but 
 I can't find them with a cursory search.


https://github.com/dlang-community/containers/blob/master/src/containers/cyclicbuffer.d
Oct 10 2018
prev sibling next sibling parent reply Heromyth <bitworld qq.com> writes:
On Monday, 8 October 2018 at 09:39:55 UTC, John Burton wrote:

 My use case is sending data to a socket.


We have ported some containers from JAVA.

ByteBuffer is a basic container interface and widely used in JAVA.

See also:
https://github.com/huntlabs/hunt/blob/master/source/hunt/container/ByteBuffer.d
https://github.com/huntlabs/hunt/tree/master/examples/ContainerDemo/source
Oct 14 2018
parent John Burton <john.burton jbmail.com> writes:
On Sunday, 14 October 2018 at 13:07:30 UTC, Heromyth wrote:

 On Monday, 8 October 2018 at 09:39:55 UTC, John Burton wrote:

 My use case is sending data to a socket.


 We have ported some containers from JAVA.

 ByteBuffer is a basic container interface and widely used in 
 JAVA.

 See also:
 https://github.com/huntlabs/hunt/blob/master/source/hunt/container/ByteBuffer.d
 https://github.com/huntlabs/hunt/tree/master/examples/ContainerDemo/source


Thanks for this, and everyone elses comments. This looks to be 
close to what I need.
Oct 16 2018
prev sibling parent Guillaume Piolat <spam smam.org> writes:
On Monday, 8 October 2018 at 09:39:55 UTC, John Burton wrote:

 I would do much better to maintain a fixed size buffer and 
 maintain read and write positions etc.


Perhaps 
https://github.com/AuburnSounds/Dplug/blob/master/core/dplug/core/ringbuf.d#L16
Oct 14 2018