www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Bitfields

reply Russel Winder <russel winder.org.uk> writes:
Hi,

Has anyone used D to work with arbitrary length bitfields with multiple
occurences of a sub-bitfield. I am working with DVB Sections and EIT packet=
s
are defined as bitfields with loops in them and the header is 112 bits. The
loops are handleable with subfields obviously, assuming you can work out ho=
w
the bigendian works on the byte sequence.

As far as I can see std.bitmanip only caters for 8, 16, 32, and 64 bit long
bitfields.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
May 21 2019
next sibling parent reply Boris-Barboris <ismailsiege gmail.com> writes:
On Tuesday, 21 May 2019 at 17:16:05 UTC, Russel Winder wrote:

 Hi,

 Has anyone used D to work with arbitrary length bitfields with 
 multiple occurences of a sub-bitfield. I am working with DVB 
 Sections and EIT packets are defined as bitfields with loops in 
 them and the header is 112 bits. The loops are handleable with 
 subfields obviously, assuming you can work out how the 
 bigendian works on the byte sequence.

 As far as I can see std.bitmanip only caters for 8, 16, 32, and 
 64 bit long bitfields.


Never used it myself, but BitArray with careful handling of 
endianess might fit your task.

https://dlang.org/phobos/std_bitmanip.html#.BitArray.this.2
https://dlang.org/phobos/std_bitmanip.html#.peek
May 21 2019
parent Russel Winder <russel winder.org.uk> writes:
On Tue, 2019-05-21 at 18:22 +0000, Boris-Barboris via Digitalmars-d-learn
wrote:
[=E2=80=A6]

=20
 Never used it myself, but BitArray with careful handling of=20
 endianess might fit your task.
=20
 https://dlang.org/phobos/std_bitmanip.html#.BitArray.this.2
 https://dlang.org/phobos/std_bitmanip.html#.peek


I'll have to mull that one over. The incoming data is a sequence of bytes t=
hat
is treated as a bitfield. Although the standard says "big-endian" it is isn=
't
entirely clear how this relates to the bitfields, I guess I need to read th=
e
standard more. :-(

I guess the question is whether BitArray can work with bytes rather than
size_t as elements of the backing array.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
May 22 2019
prev sibling parent reply Era Scarecrow <rtcvb32 yahoo.com> writes:
On Tuesday, 21 May 2019 at 17:16:05 UTC, Russel Winder wrote:

 As far as I can see std.bitmanip only caters for 8, 16, 32, and 
 64 bit long bitfields.


  I worked on/with bitfields in the past, the limit sizes is more 
or less for natural int types that D supports.

  However this limitation is kinda arbitrary, as for simplicity it 
relies on shifting bits, going larger or any byte size is 
possible depending on what needs to be stored, but ti's the speed 
that really takes a penalty when you aren't using native types or 
you have to do a lot of shifting to get the job done.

  What's the layout of what you need? I'll see if i can't make 
something that would work for you.

  Would be better if you can use a object that breaks the parts 
down and you can actually fully access those parts, then just 
re-store it into the limited space you want for storage, which 
then would be faster than bitfields (although not by much)
May 21 2019
parent reply Russel Winder <russel winder.org.uk> writes:
On Tue, 2019-05-21 at 19:14 +0000, Era Scarecrow via Digitalmars-d-learn
wrote:

=20


[=E2=80=A6]

   I worked on/with bitfields in the past, the limit sizes is more=20
 or less for natural int types that D supports.


Rust bitfield crate and it's macros are the same, the underlying type for a
bitfield must be a primitive integer type. Fortunately, Rust has i128 and u=
128
which is enough for my 112 bit EIT header.

Boris Barboris suggested using BitArray and I willinvestigate but the
size_t/byte problem would need to go away.


   However this limitation is kinda arbitrary, as for simplicity it=20
 relies on shifting bits, going larger or any byte size is=20
 possible depending on what needs to be stored, but ti's the speed=20
 that really takes a penalty when you aren't using native types or=20
 you have to do a lot of shifting to get the job done.
=20
   What's the layout of what you need? I'll see if i can't make=20
 something that would work for you.
=20
   Would be better if you can use a object that breaks the parts=20
 down and you can actually fully access those parts, then just=20
 re-store it into the limited space you want for storage, which=20
 then would be faster than bitfields (although not by much)


I found an interesting way forward in the source code of dvbsnoop. It
basically uses the byte sequence as a backing store and then has a function=
 to
do the necessary accesses to treat it as a bit array.

If D's Bit Array can work with bytes instead of size_t then it is exactly w=
hat
dvbsnoop does (in C) but adds writing as well as reading.

The Rust solution using bitfield with a u128 backing it seems to work, but =
it
is all very clumsy even if it is efficacious.=20

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk
May 22 2019
parent Basile B. <b2.temp gmx.com> writes:
On Wednesday, 22 May 2019 at 08:54:45 UTC, Russel Winder wrote:

 On Tue, 2019-05-21 at 19:14 +0000, Era Scarecrow via 
 Digitalmars-d-learn wrote:

 


 […]

   I worked on/with bitfields in the past, the limit sizes is 
 more
 or less for natural int types that D supports.


 Rust bitfield crate and it's macros are the same, the 
 underlying type for a bitfield must be a primitive integer 
 type. Fortunately, Rust has i128 and u128 which is enough for 
 my 112 bit EIT header.

 Boris Barboris suggested using BitArray and I willinvestigate 
 but the size_t/byte problem would need to go away.


   However this limitation is kinda arbitrary, as for 
 simplicity it
 relies on shifting bits, going larger or any byte size is
 possible depending on what needs to be stored, but ti's the 
 speed
 that really takes a penalty when you aren't using native types 
 or
 you have to do a lot of shifting to get the job done.
 
   What's the layout of what you need? I'll see if i can't make
 something that would work for you.
 
   Would be better if you can use a object that breaks the parts
 down and you can actually fully access those parts, then just
 re-store it into the limited space you want for storage, which
 then would be faster than bitfields (although not by much)


 I found an interesting way forward in the source code of 
 dvbsnoop. It basically uses the byte sequence as a backing 
 store and then has a function to do the necessary accesses to 
 treat it as a bit array.

 If D's Bit Array can work with bytes instead of size_t then it 
 is exactly what dvbsnoop does (in C) but adds writing as well 
 as reading.

 The Rust solution using bitfield with a u128 backing it seems 
 to work, but it is all very clumsy even if it is efficacious.


If the type of operations you need on your 128 bit container is 
simple enough (bitwise op) you can implement them, backed by two 
ulong in a custom struct.
I did something similar for the tokens if my language[1]. There 
are probably wrong stuff in there but for my usage (include and 
bt) that works seamlessly.

[1]: 
https://github.com/Basile-z/styx/blob/master/src/styx/data_structures.d#L40
May 22 2019