www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Handling endianness in Tango's FileConduit

reply "Nick Sabalausky" <a a.a> writes:
I've been looking at the Tango docs, but I'm unclear on how IProtocol works, 
specifically in relation to endianness.

I see stuff about protocols and endianness in the docs, so I assume Tango 
provides a way to say "The file I'm reading/writing through this conduit 
uses X type of endieanness, so Tango, make sure that any shorts, ints, etc, 
that I read/write are adjusted to match the system's native endianness."  If 
so, how would I adjust the following to tell Tango "This file uses 
little-endian"?:

auto file = new FileConduit(infilename);
scope(exit) file.close();
auto input = new Reader(file.input);
// Load data through 'input'

BTW, A small side question: Tango's conduits don't have anything like 
"char[] readNullTerminatedString()", do they? It's trivial enough to 
implement, but I thought if it was there, I may as well just use it. (I 
don't think it is there though, right? No big deal, though, just wondering.)
Oct 04 2008
parent reply =?UTF-8?B?QWxleGFuZGVyIFDDoW5law==?= writes:
Nick Sabalausky wrote:

 I've been looking at the Tango docs, but I'm unclear on how IProtocol works, 
 specifically in relation to endianness.
 
 I see stuff about protocols and endianness in the docs, so I assume Tango 
 provides a way to say "The file I'm reading/writing through this conduit 
 uses X type of endieanness, so Tango, make sure that any shorts, ints, etc, 
 that I read/write are adjusted to match the system's native endianness."  If 
 so, how would I adjust the following to tell Tango "This file uses 
 little-endian"?:
 
 auto file = new FileConduit(infilename);
 scope(exit) file.close();
 auto input = new Reader(file.input);
 // Load data through 'input'


tango.io.protocol.EndianProtocol should do that:

auto file = new FileConduit(infilename);
scope(exit) file.close;

auto protocol = new EndianProtocol(file);
auto input = new DerivedReader(protocol);

Note: it seems like this is always assuming the other endianess as your 
current system runs on.


 BTW, A small side question: Tango's conduits don't have anything like 
 "char[] readNullTerminatedString()", do they? It's trivial enough to 
 implement, but I thought if it was there, I may as well just use it. (I 
 don't think it is there though, right? No big deal, though, just wondering.) 


I don’t think so.
Oct 04 2008
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
"Alexander Pánek" wrote

 Nick Sabalausky wrote:

 I've been looking at the Tango docs, but I'm unclear on how IProtocol 
 works, specifically in relation to endianness.

 I see stuff about protocols and endianness in the docs, so I assume Tango 
 provides a way to say "The file I'm reading/writing through this conduit 
 uses X type of endieanness, so Tango, make sure that any shorts, ints, 
 etc, that I read/write are adjusted to match the system's native 
 endianness."  If so, how would I adjust the following to tell Tango "This 
 file uses little-endian"?:

 auto file = new FileConduit(infilename);
 scope(exit) file.close();
 auto input = new Reader(file.input);
 // Load data through 'input'


 tango.io.protocol.EndianProtocol should do that:

 auto file = new FileConduit(infilename);
 scope(exit) file.close;

 auto protocol = new EndianProtocol(file);
 auto input = new DerivedReader(protocol);

 Note: it seems like this is always assuming the other endianess as your 
 current system runs on.


Try PickleProtocol.  This always assumes the data being read/written is big 
endian.  I don't think there's an equivalent one if the file is in little 
endian form, but you could always just copy what PickleProtocol does (make 
an alias depending on native endianness).  In fact, PickleProtocol should 
probably be changed to allow you to specify the endianness you want.

-Steve
Oct 04 2008
parent reply "Nick Sabalausky" <a a.a> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
news:gc8ape$2i2r$1 digitalmars.com...

 "Alexander Pánek" wrote

 Nick Sabalausky wrote:

 I've been looking at the Tango docs, but I'm unclear on how IProtocol 
 works, specifically in relation to endianness.

 I see stuff about protocols and endianness in the docs, so I assume 
 Tango provides a way to say "The file I'm reading/writing through this 
 conduit uses X type of endieanness, so Tango, make sure that any shorts, 
 ints, etc, that I read/write are adjusted to match the system's native 
 endianness."  If so, how would I adjust the following to tell Tango 
 "This file uses little-endian"?:

 auto file = new FileConduit(infilename);
 scope(exit) file.close();
 auto input = new Reader(file.input);
 // Load data through 'input'


 tango.io.protocol.EndianProtocol should do that:

 auto file = new FileConduit(infilename);
 scope(exit) file.close;

 auto protocol = new EndianProtocol(file);
 auto input = new DerivedReader(protocol);

 Note: it seems like this is always assuming the other endianess as your 
 current system runs on.


 Try PickleProtocol.  This always assumes the data being read/written is 
 big endian.  I don't think there's an equivalent one if the file is in 
 little endian form, but you could always just copy what PickleProtocol 
 does (make an alias depending on native endianness).  In fact, 
 PickleProtocol should probably be changed to allow you to specify the 
 endianness you want.

 -Steve


So, if I understand everything right:

- What EndianProtocol does: endianness is always switched.
- What PickleProtocol does: file's endianness is always assumed to be big 
endian and converted appropriately.

(I don't see any description of those in the docs, it should probably 
mention what they do and how they're used.)

If that's all what they do, then seems to me it should work more like this:

1. Rename "EndianProtocol" to "EndianSwitchProtocol" to make it's behavior 
more clear. Possibly make it internal to Tango.
2. Create a new "EndianProtocol" that takes an "Endianness" enum of 
"LittleEndian" or "BigEndian" (and maybe also "SwitchEndian" and 
"NativeEndian") as a ctor parameter, checks the native endianness, and 
appropriately adds or doesn't add the "EndianSwitchProtocol" (or just does 
the byte-order swapping itself instead of using "EndianSwitchProtocol").
3. Ditch the PickleProtocol (unless there's something else it also does that 
I'm not aware of.)

In any case, it looks like step 2 is something I'd currently have to 
implement myself. Does Tango provide a way to detect the system's 
endianness?
Oct 04 2008
next sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
Nick Sabalausky wrote:

 3. Ditch the PickleProtocol (unless there's something else it also does that 
 I'm not aware of.)


http://www.python.org/doc/2.5.2/lib/pickle-protocol.html
Oct 04 2008
prev sibling parent reply =?UTF-8?B?QWxleGFuZGVyIFDDoW5law==?= writes:
Nick Sabalausky wrote:

 1. Rename "EndianProtocol" to "EndianSwitchProtocol" to make it's behavior 
 more clear. Possibly make it internal to Tango.
 2. Create a new "EndianProtocol" that takes an "Endianness" enum of 
 "LittleEndian" or "BigEndian" (and maybe also "SwitchEndian" and 
 "NativeEndian") as a ctor parameter, checks the native endianness, and 
 appropriately adds or doesn't add the "EndianSwitchProtocol" (or just does 
 the byte-order swapping itself instead of using "EndianSwitchProtocol").


Please create a ticket for that!


 In any case, it looks like step 2 is something I'd currently have to 
 implement myself. Does Tango provide a way to detect the system's 
 endianness?


Tango doesn’t, but D does with version(BigEndian) or version(LittleEndian).

Also, if you implement it yourself, you can as well contribute it to 
Tango, so other people don’t have the same problem. :) You can also add 
a patch/code to the ticket. Would be awesome.

Kind regards, Alex
Oct 04 2008
parent "Nick Sabalausky" <a a.a> writes:
 Nick Sabalausky wrote:

 1. Rename "EndianProtocol" to "EndianSwitchProtocol" to make it's 
 behavior more clear. Possibly make it internal to Tango.
 2. Create a new "EndianProtocol" that takes an "Endianness" enum of 
 "LittleEndian" or "BigEndian" (and maybe also "SwitchEndian" and 
 "NativeEndian") as a ctor parameter, checks the native endianness, and 
 appropriately adds or doesn't add the "EndianSwitchProtocol" (or just 
 does the byte-order swapping itself instead of using 
 "EndianSwitchProtocol").


 Please create a ticket for that!


 In any case, it looks like step 2 is something I'd currently have to 
 implement myself. Does Tango provide a way to detect the system's 
 endianness?


 Tango doesn't, but D does with version(BigEndian) or 
 version(LittleEndian).


*Smacks forehead* I knew that, and managed to forget :) That was one of the
early things that first impressd me about D.


 Also, if you implement it yourself, you can as well contribute it to 
 Tango, so other people don't have the same problem. :) You can also add a 
 patch/code to the ticket. Would be awesome.


Good idea. Done:

http://www.dsource.org/projects/tango/ticket/1321


 Kind regards, Alex
Oct 08 2008