www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Orange 1.0.0 beta - serialization library

reply Jacob Carlborg <doob me.com> writes:
I've almost finished the rewrite of my serialization library Orange. I'm 
hoping that someone wants to give it a try and see what issues/bugs are 
found.

Project page: http://dsource.org/projects/orange
Source code: https://github.com/jacob-carlborg/orange

There are two usage examples on the project page. For more examples I 
recommend looking at the unit tests in the "tests" directory.

Description:

Orange is a serialization library for D1 and D2, supporting both Tango 
and Phobos. It can serialize most of the available types in D, including 
third party types and can serialize through base class references. It 
supports fully automatic serialization of all supported types and also 
supports several ways to customize the serialization process. Orange has 
a separate front end (the serializer) and back end (the archive) making 
it possible for the user to create new archive types that can be used 
with the existing serializer.

Features:

* Automatically serializes the base classes
* Supports events (before and after (de)serializing)

* Supports non-serialized fields and classes (you can say that some 
fields in a class should not be serialized)

* Licensed under the Boost license
* Std/runtime library independent

* Extendable - possible to create new archive types and use them with 
the existing serializer

* Serializes through base class references
* Serializes third party types

* Customization of the (de)serialization process, both intrusive and 
non-intrusive

* Properly (de)serializes slices and pointers

Known Issues/Limitations:

* Due to limitations in the XML module provided by Phobos the XMLArchive 
will only work with "char" as the template type with D2

* Due to several bugs/limitations in the compiler/runtime even the D2 
version requires you to register the type when serializing through base 
class references

* No built-in support for versioning
* Floating point numbers are not serialized as hexadecimal (D1)

-- 
/Jacob Carlborg
Aug 20 2011
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
This looks really great!  I'll try it out sometime later.  A few questions:

1.  What other archiver formats besides XML might be useful?  (I 
remember binary can't work because of the way keys work, though 
messagepack is probably a better option for lightweight serialization in 
general.  Orange's niche is portability and serializing as many types as 
possible.)

2.  What are the prospects for submitting this for inclusion in Phobos? 
  Serialization is something basic and universally needed enough that it 
should not require a third-party library.

3.  Given that all of the necessary introspection already works, what 
are the prospects for creating a Clone archiver?  A Clone archiver would 
not actually serialize anything and instead would deep clone whatever 
data structure it receives.  If Orange gets into Phobos, cloning could 
be recognized by std.concurrency as a safe, simple way to pass complex 
object graphs between threads.  I'm picturing an API something like this:

import std.stdio, whateverOrangeEndsUpBeingNamed.

struct Cloned(T) {
     T obj;
}

Cloned!T cloned(T)(T obj) { return typeof(return)(obj); }

void spawmMe() {
     receive((int[][]) { writeln("Received an int[][]."); });
}

void main() {
     auto tid = spawn(&spawnMe);
     auto mat = new int[][](42, 42);

     // FAILS:  Implicit sharing.
     tid.send(mat);

     // Works.  mat is automatically deep cloned and the clone is sent
     // to tid.  send recognized the Cloned struct as special and
     // processes the payload accordingly.
     tid.send(cloned(mat));
}



On 8/20/2011 11:13 AM, Jacob Carlborg wrote:

 I've almost finished the rewrite of my serialization library Orange. I'm
 hoping that someone wants to give it a try and see what issues/bugs are
 found.

 Project page: http://dsource.org/projects/orange
 Source code: https://github.com/jacob-carlborg/orange

 There are two usage examples on the project page. For more examples I
 recommend looking at the unit tests in the "tests" directory.

 Description:

 Orange is a serialization library for D1 and D2, supporting both Tango
 and Phobos. It can serialize most of the available types in D, including
 third party types and can serialize through base class references. It
 supports fully automatic serialization of all supported types and also
 supports several ways to customize the serialization process. Orange has
 a separate front end (the serializer) and back end (the archive) making
 it possible for the user to create new archive types that can be used
 with the existing serializer.

 Features:

 * Automatically serializes the base classes
 * Supports events (before and after (de)serializing)

 * Supports non-serialized fields and classes (you can say that some
 fields in a class should not be serialized)

 * Licensed under the Boost license
 * Std/runtime library independent

 * Extendable - possible to create new archive types and use them with
 the existing serializer

 * Serializes through base class references
 * Serializes third party types

 * Customization of the (de)serialization process, both intrusive and
 non-intrusive

 * Properly (de)serializes slices and pointers

 Known Issues/Limitations:

 * Due to limitations in the XML module provided by Phobos the XMLArchive
 will only work with "char" as the template type with D2

 * Due to several bugs/limitations in the compiler/runtime even the D2
 version requires you to register the type when serializing through base
 class references

 * No built-in support for versioning
 * Floating point numbers are not serialized as hexadecimal (D1)
Aug 20 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Masahiro Nakagawa seems to be working on msgpack:
https://bitbucket.org/repeatedly/msgpack4d/

Is RIFF considered a good format? I've ran into it when porting C
code,  it seems it's also used by Google and probably other companies.
http://en.wikipedia.org/wiki/Resource_Interchange_File_Format
Aug 20 2011
parent reply dsimcha <dsimcha yahoo.com> writes:
On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:

 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/


Yeah, this does look quite useful.  I tried it a few weeks ago and ran 
into a couple showstopper bugs.  I filed them and he fixed them, but I 
never got around to trying it again.

I actually think that since both are very commonly needed things and 
satisfy completely different niches, Orange and msgpack4d might both 
eventually have a rightful place in Phobos.  Orange would be recommended 
when portability and serializing as many types as possible are 
important, and msgpack4d would be recommended when speed and space 
efficiency are the primary concerns.


 Is RIFF considered a good format? I've ran into it when porting C
 code,  it seems it's also used by Google and probably other companies.
 http://en.wikipedia.org/wiki/Resource_Interchange_File_Format


 From reading the Wikipedia article (I knew nothing about RIFF before 
this discussion) it sounds like it's in a valley in between two utility 
peaks.  XML provides maximum portability.  msgpack provides maximum 
efficiency.  RIFF doesn't really provide either all that well.
Aug 20 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-20 19:45, dsimcha wrote:

 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:

 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/


 Yeah, this does look quite useful. I tried it a few weeks ago and ran
 into a couple showstopper bugs. I filed them and he fixed them, but I
 never got around to trying it again.

 I actually think that since both are very commonly needed things and
 satisfy completely different niches, Orange and msgpack4d might both
 eventually have a rightful place in Phobos. Orange would be recommended
 when portability and serializing as many types as possible are
 important, and msgpack4d would be recommended when speed and space
 efficiency are the primary concerns.


If it's possible to extract a value given a key with messagepack I think 
it would be possible to use messagepack as an archive for Orange. Since 
messagepack supports maps every type Orange can serialize should be 
possible serialize with messagepack, although probably not as efficient 
as "regular" messagepack.

-- 
/Jacob Carlborg
Aug 20 2011
parent reply "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Sun, 21 Aug 2011 05:28:33 +0900, Jacob Carlborg <doob me.com> wrote:


 On 2011-08-20 19:45, dsimcha wrote:

 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:

 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/


 Yeah, this does look quite useful. I tried it a few weeks ago and ran
 into a couple showstopper bugs. I filed them and he fixed them, but I
 never got around to trying it again.

 I actually think that since both are very commonly needed things and
 satisfy completely different niches, Orange and msgpack4d might both
 eventually have a rightful place in Phobos. Orange would be recommended
 when portability and serializing as many types as possible are
 important, and msgpack4d would be recommended when speed and space
 efficiency are the primary concerns.


 If it's possible to extract a value given a key with messagepack I think  
 it would be possible to use messagepack as an archive for Orange. Since  
 messagepack supports maps every type Orange can serialize should be  
 possible serialize with messagepack, although probably not as efficient  
 as "regular" messagepack.


I agree.
Unfortunately, Orange with MessagePack decreases MessagePack merits...
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 23:16, Masahiro Nakagawa wrote:

 On Sun, 21 Aug 2011 05:28:33 +0900, Jacob Carlborg <doob me.com> wrote:
 

 On 2011-08-20 19:45, dsimcha wrote:

 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:

 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/


 Yeah, this does look quite useful. I tried it a few weeks ago and ran
 into a couple showstopper bugs. I filed them and he fixed them, but I
 never got around to trying it again.

 I actually think that since both are very commonly needed things and
 satisfy completely different niches, Orange and msgpack4d might both
 eventually have a rightful place in Phobos. Orange would be recommended
 when portability and serializing as many types as possible are
 important, and msgpack4d would be recommended when speed and space
 efficiency are the primary concerns.


 If it's possible to extract a value given a key with messagepack I 
 think it would be possible to use messagepack as an archive for 
 Orange. Since messagepack supports maps every type Orange can 
 serialize should be possible serialize with messagepack, although 
 probably not as efficient as "regular" messagepack.


 
 I agree.
 Unfortunately, Orange with MessagePack decreases MessagePack merits...


I suspected that.

-- 
/Jacob Carlborg
Aug 21 2011
prev sibling next sibling parent simendsjo <simendsjo gmail.com> writes:
On 20.08.2011 19:45, dsimcha wrote:

 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:

 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/


 Yeah, this does look quite useful. I tried it a few weeks ago and ran
 into a couple showstopper bugs. I filed them and he fixed them, but I
 never got around to trying it again.

 I actually think that since both are very commonly needed things and
 satisfy completely different niches, Orange and msgpack4d might both
 eventually have a rightful place in Phobos. Orange would be recommended
 when portability and serializing as many types as possible are
 important, and msgpack4d would be recommended when speed and space
 efficiency are the primary concerns.


 Is RIFF considered a good format? I've ran into it when porting C
 code, it seems it's also used by Google and probably other companies.
 http://en.wikipedia.org/wiki/Resource_Interchange_File_Format


  From reading the Wikipedia article (I knew nothing about RIFF before
 this discussion) it sounds like it's in a valley in between two utility
 peaks. XML provides maximum portability. msgpack provides maximum
 efficiency. RIFF doesn't really provide either all that well.


Agreed. I've worked a lot with RIFF in the past, and it only specifies a 
high level structure by having sections with a known size. It doesn't 
give much over a plain binary file.
Aug 21 2011
prev sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Sun, 21 Aug 2011 02:45:20 +0900, dsimcha <dsimcha yahoo.com> wrote:


 On 8/20/2011 12:50 PM, Andrej Mitrovic wrote:

 Masahiro Nakagawa seems to be working on msgpack:
 https://bitbucket.org/repeatedly/msgpack4d/


 Yeah, this does look quite useful.  I tried it a few weeks ago and ran  
 into a couple showstopper bugs.  I filed them and he fixed them, but I  
 never got around to trying it again.

 I actually think that since both are very commonly needed things and  
 satisfy completely different niches, Orange and msgpack4d might both  
 eventually have a rightful place in Phobos.


I agree. In the past, I mentioned this point ;)


 Orange would be recommended when portability and serializing as many  
 types as possible are important, and msgpack4d would be recommended when  
 speed and space efficiency are the primary concerns.


Yes for MessagePack.
In addition, communicating other language is important like JSON.

P.S.
My D libraries moved to github. msgpack4d is here.

https://github.com/msgpack/msgpack-d

FTTB, bitbucket is available.
Aug 21 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-20 18:16, dsimcha wrote:

 This looks really great! I'll try it out sometime later. A few questions:

 1. What other archiver formats besides XML might be useful? (I remember
 binary can't work because of the way keys work, though messagepack is
 probably a better option for lightweight serialization in general.
 Orange's niche is portability and serializing as many types as possible.)


I assume any archive type that can unpack a value given a key will work. 
I assume JSON and YAML will work as well. I no nothing about binary 
serialization but I think messagepack can work.

Another problem might be pointers and slices. For example, at the first 
step every array is assumed to be pointing to its own memory, i.e. not a 
slice. After everything is serialized the serializer post process the 
serialized data and replaces all arrays that points into another array 
with a slice.


 2. What are the prospects for submitting this for inclusion in Phobos?
 Serialization is something basic and universally needed enough that it
 should not require a third-party library.


 From my part:

* Remove all Tango/D1 specific code
* Add documentation
* There are some few things left in the rewrite process
* I've made some modifications to the XML module in Phobos, these need 
to be applied as well
* Probably something else I can't think of right now

I'm not particular happy about removing the Tango/D1 specific code if 
I'm not sure that the library can be included in Phobos.


 3. Given that all of the necessary introspection already works, what are
 the prospects for creating a Clone archiver? A Clone archiver would not
 actually serialize anything and instead would deep clone whatever data
 structure it receives. If Orange gets into Phobos, cloning could be
 recognized by std.concurrency as a safe, simple way to pass complex
 object graphs between threads. I'm picturing an API something like this:

 import std.stdio, whateverOrangeEndsUpBeingNamed.

 struct Cloned(T) {
 T obj;
 }

 Cloned!T cloned(T)(T obj) { return typeof(return)(obj); }

 void spawmMe() {
 receive((int[][]) { writeln("Received an int[][]."); });
 }

 void main() {
 auto tid = spawn(&spawnMe);
 auto mat = new int[][](42, 42);

 // FAILS: Implicit sharing.
 tid.send(mat);

 // Works. mat is automatically deep cloned and the clone is sent
 // to tid. send recognized the Cloned struct as special and
 // processes the payload accordingly.
 tid.send(cloned(mat));
 }


Currently you could serialize the data an pass it send it over to 
another thread and deserialize on the other side, but I'm guessing this 
would be not very effective. I'm not exactly sure what's needed to be 
done to make a clone archive and what can be shared but as you say, the 
necessary introspection already work.


 On 8/20/2011 11:13 AM, Jacob Carlborg wrote:

 I've almost finished the rewrite of my serialization library Orange. I'm
 hoping that someone wants to give it a try and see what issues/bugs are
 found.

 Project page: http://dsource.org/projects/orange
 Source code: https://github.com/jacob-carlborg/orange

 There are two usage examples on the project page. For more examples I
 recommend looking at the unit tests in the "tests" directory.

 Description:

 Orange is a serialization library for D1 and D2, supporting both Tango
 and Phobos. It can serialize most of the available types in D, including
 third party types and can serialize through base class references. It
 supports fully automatic serialization of all supported types and also
 supports several ways to customize the serialization process. Orange has
 a separate front end (the serializer) and back end (the archive) making
 it possible for the user to create new archive types that can be used
 with the existing serializer.

 Features:

 * Automatically serializes the base classes
 * Supports events (before and after (de)serializing)

 * Supports non-serialized fields and classes (you can say that some
 fields in a class should not be serialized)

 * Licensed under the Boost license
 * Std/runtime library independent

 * Extendable - possible to create new archive types and use them with
 the existing serializer

 * Serializes through base class references
 * Serializes third party types

 * Customization of the (de)serialization process, both intrusive and
 non-intrusive

 * Properly (de)serializes slices and pointers

 Known Issues/Limitations:

 * Due to limitations in the XML module provided by Phobos the XMLArchive
 will only work with "char" as the template type with D2

 * Due to several bugs/limitations in the compiler/runtime even the D2
 version requires you to register the type when serializing through base
 class references

 * No built-in support for versioning
 * Floating point numbers are not serialized as hexadecimal (D1)





-- 
/Jacob Carlborg
Aug 20 2011
parent reply dsimcha <dsimcha yahoo.com> writes:
On 8/20/2011 4:24 PM, Jacob Carlborg wrote:

 2. What are the prospects for submitting this for inclusion in Phobos?
 Serialization is something basic and universally needed enough that it
 should not require a third-party library.


  From my part:

 * Remove all Tango/D1 specific code
 * Add documentation
 * There are some few things left in the rewrite process


Ok, it sounds like this isn't going to be ready that soon.  I'll start 
using it as-is, as a third party library.  Then I can make suggestions, 
file bug reports, etc. and streamline its eventual Phobos review.


 * I've made some modifications to the XML module in Phobos, these need
 to be applied as well


If we're really lucky, by the time your turn in the review queue comes 
along, Phobos will have a new and much better XML module, since Thomas 
Sowinki is apparently working on one.


 * Probably something else I can't think of right now

 I'm not particular happy about removing the Tango/D1 specific code if
 I'm not sure that the library can be included in Phobos.


Hmm. We could review the module before you remove it, with the 
assumption that this code will be removed if accepted.



 Currently you could serialize the data an pass it send it over to
 another thread and deserialize on the other side, but I'm guessing this
 would be not very effective. I'm not exactly sure what's needed to be
 done to make a clone archive and what can be shared but as you say, the
 necessary introspection already work.


Right.  The idea would be to bypass serializing and unserializing and 
leverage the introspection that Orange provides to create an efficient 
clone() function.  This is mostly a long-term prospect, to be considered 
more seriously if/when Orange gets into Phobos.  All we'd need from 
orange is an archiver that efficiently clones without actually 
serializing or deserializing anything.  Given this, the changes to 
std.concurrency would be trivial.
Aug 20 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-20 23:29, dsimcha wrote:

 On 8/20/2011 4:24 PM, Jacob Carlborg wrote:

 2. What are the prospects for submitting this for inclusion in Phobos?
 Serialization is something basic and universally needed enough that it
 should not require a third-party library.


 From my part:

 * Remove all Tango/D1 specific code
 * Add documentation
 * There are some few things left in the rewrite process


 Ok, it sounds like this isn't going to be ready that soon. I'll start
 using it as-is, as a third party library. Then I can make suggestions,
 file bug reports, etc. and streamline its eventual Phobos review.


I will add documentation and finish the rewrite regardless so it's only 
the D1/Tango specific code.

Oh, one more thing. I'm using my own kind of unit test "framework", 
don't know what people think about that. It could easily be removed though.


 * I've made some modifications to the XML module in Phobos, these need
 to be applied as well


 If we're really lucky, by the time your turn in the review queue comes
 along, Phobos will have a new and much better XML module, since Thomas
 Sowinki is apparently working on one.


Yes, hopefully. Also, some of the changes I've made are because of 
differences with the Tango and Phobos XML modules. Since it will only 
support the Phobos module it will be easier and hopefully not require as 
many changes.


 * Probably something else I can't think of right now

 I'm not particular happy about removing the Tango/D1 specific code if
 I'm not sure that the library can be included in Phobos.


 Hmm. We could review the module before you remove it, with the
 assumption that this code will be removed if accepted.


Exactly, sounds like a good idea.


 Currently you could serialize the data an pass it send it over to
 another thread and deserialize on the other side, but I'm guessing this
 would be not very effective. I'm not exactly sure what's needed to be
 done to make a clone archive and what can be shared but as you say, the
 necessary introspection already work.


 Right. The idea would be to bypass serializing and unserializing and
 leverage the introspection that Orange provides to create an efficient
 clone() function. This is mostly a long-term prospect, to be considered
 more seriously if/when Orange gets into Phobos. All we'd need from
 orange is an archiver that efficiently clones without actually
 serializing or deserializing anything. Given this, the changes to
 std.concurrency would be trivial.


Yeah, I wonder if any data needs to be stored in the archive and in that 
case, how it could be stored.

-- 
/Jacob Carlborg
Aug 21 2011
prev sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On Aug 20, 2011, at 9:16 AM, dsimcha wrote:


 This looks really great!  I'll try it out sometime later.  A few =


questions:

=20
 1.  What other archiver formats besides XML might be useful?  (I =


remember binary can't work because of the way keys work, though =
messagepack is probably a better option for lightweight serialization in =
general.  Orange's niche is portability and serializing as many types as =
possible.)

JSON, Protobuf=85


 2.  What are the prospects for submitting this for inclusion in =


Phobos?  Serialization is something basic and universally needed enough =
that it should not require a third-party library.

Some form of serialization definitely needs to be in Phobos/Druntime.  =
The other day I was thinking that the basic framework for struct/class =
serialization might be nice in Druntime as a replacement for x.toString, =
with most of the actual functional bits in Phobos.=
Aug 29 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-29 19:19, Sean Kelly wrote:

 On Aug 20, 2011, at 9:16 AM, dsimcha wrote:


 This looks really great!  I'll try it out sometime later.  A few questions:

 1.  What other archiver formats besides XML might be useful?  (I remember
binary can't work because of the way keys work, though messagepack is probably
a better option for lightweight serialization in general.  Orange's niche is
portability and serializing as many types as possible.)


 JSON, Protobuf…


YAML as well.


 2.  What are the prospects for submitting this for inclusion in Phobos? 
Serialization is something basic and universally needed enough that it should
not require a third-party library.


 Some form of serialization definitely needs to be in Phobos/Druntime.  The
other day I was thinking that the basic framework for struct/class
serialization might be nice in Druntime as a replacement for x.toString, with
most of the actual functional bits in Phobos.


I've been think the same several times before and think it would be 
really useful for debugging.

-- 
/Jacob Carlborg
Aug 29 2011
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:


 I've almost finished the rewrite of my serialization library Orange.


While I've never had the chance to use Orange myself, one of the problems  
I've often heard being associated with using Orange was large binary file  
sizes. What are your thoughts about this issue? If this is a real problem,  
do you think the toolchain could be improved in this regard (perhaps ideas  
such as merging bitwise-identical functions at link time)?

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 21:50, Vladimir Panteleev wrote:

 On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:


 I've almost finished the rewrite of my serialization library Orange.


 While I've never had the chance to use Orange myself, one of the
 problems I've often heard being associated with using Orange was large
 binary file sizes. What are your thoughts about this issue? If this is a
 real problem, do you think the toolchain could be improved in this
 regard (perhaps ideas such as merging bitwise-identical functions at
 link time)?


I never heard anything being associated with using Orange. The library 
takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World app 
using Orange takes up 1.2 mb, I don't know if that's a problem.

I don't have any thoughts, I didn't know it was an issue. Yes I think 
the tool chain can be improved, at least I hope it can.

-- 
/Jacob Carlborg
Aug 22 2011
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob me.com> said:


 On 2011-08-22 21:50, Vladimir Panteleev wrote:

 On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:
 

 I've almost finished the rewrite of my serialization library Orange.


 
 While I've never had the chance to use Orange myself, one of the
 problems I've often heard being associated with using Orange was large
 binary file sizes. What are your thoughts about this issue? If this is a
 real problem, do you think the toolchain could be improved in this
 regard (perhaps ideas such as merging bitwise-identical functions at
 link time)?


 
 I never heard anything being associated with using Orange. The library 
 takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World 
 app using Orange takes up 1.2 mb, I don't know if that's a problem.
 
 I don't have any thoughts, I didn't know it was an issue. Yes I think 
 the tool chain can be improved, at least I hope it can.


If there's any truth in that (which I don't know), it'd probably have 
something to do with the amount of generated code through templates.

When I was still developing the D/Objective-C bridge as a bunch of 
templates and stub objects, the binary size was growing significantly 
for each new Objective-C class I added (due to the insane number of 
stub functions instantiated). And at some point it became totally 
impractical to use, even though it worked.

I doubt this is a problem for Orange because of its more limited scope, 
but it'd nevertheless be wise to measure how much the binary size grows 
when you add more types to serialize and unserialize and more code is 
generated as a result.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Aug 23 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-24 02:53, Michel Fortin wrote:

 On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob me.com> said:


 On 2011-08-22 21:50, Vladimir Panteleev wrote:

 On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:


 I've almost finished the rewrite of my serialization library Orange.


 While I've never had the chance to use Orange myself, one of the
 problems I've often heard being associated with using Orange was large
 binary file sizes. What are your thoughts about this issue? If this is a
 real problem, do you think the toolchain could be improved in this
 regard (perhaps ideas such as merging bitwise-identical functions at
 link time)?


 I never heard anything being associated with using Orange. The library
 takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World
 app using Orange takes up 1.2 mb, I don't know if that's a problem.

 I don't have any thoughts, I didn't know it was an issue. Yes I think
 the tool chain can be improved, at least I hope it can.


 If there's any truth in that (which I don't know), it'd probably have
 something to do with the amount of generated code through templates.

 When I was still developing the D/Objective-C bridge as a bunch of
 templates and stub objects, the binary size was growing significantly
 for each new Objective-C class I added (due to the insane number of stub
 functions instantiated). And at some point it became totally impractical
 to use, even though it worked.


Yeah, that was insane.


 I doubt this is a problem for Orange because of its more limited scope,
 but it'd nevertheless be wise to measure how much the binary size grows
 when you add more types to serialize and unserialize and more code is
 generated as a result.


I guess I should try serializing a big data structure of some kind.

-- 
/Jacob Carlborg
Aug 23 2011
parent "Robert Jacques" <sandford jhu.edu> writes:
On Wed, 24 Aug 2011 02:25:19 -0400, Jacob Carlborg <doob me.com> wrote:


 On 2011-08-24 02:53, Michel Fortin wrote:

 On 2011-08-23 06:51:57 +0000, Jacob Carlborg <doob me.com> said:


 On 2011-08-22 21:50, Vladimir Panteleev wrote:

 On Sat, 20 Aug 2011 18:13:32 +0300, Jacob Carlborg <doob me.com> wrote:


 I've almost finished the rewrite of my serialization library Orange.


 While I've never had the chance to use Orange myself, one of the
 problems I've often heard being associated with using Orange was large
 binary file sizes. What are your thoughts about this issue? If this is a
 real problem, do you think the toolchain could be improved in this
 regard (perhaps ideas such as merging bitwise-identical functions at
 link time)?


 I never heard anything being associated with using Orange. The library
 takes up 77.8 kb, compiled with dmd 1.067 on Mac OS X. A Hello World
 app using Orange takes up 1.2 mb, I don't know if that's a problem.

 I don't have any thoughts, I didn't know it was an issue. Yes I think
 the tool chain can be improved, at least I hope it can.


 If there's any truth in that (which I don't know), it'd probably have
 something to do with the amount of generated code through templates.

 When I was still developing the D/Objective-C bridge as a bunch of
 templates and stub objects, the binary size was growing significantly
 for each new Objective-C class I added (due to the insane number of stub
 functions instantiated). And at some point it became totally impractical
 to use, even though it worked.


 Yeah, that was insane.


 I doubt this is a problem for Orange because of its more limited scope,
 but it'd nevertheless be wise to measure how much the binary size grows
 when you add more types to serialize and unserialize and more code is
 generated as a result.


 I guess I should try serializing a big data structure of some kind.


Actually, it's the number of types rather than the amount of memory which
causes issues. You can try something like this:

     template Qual(T) {
         alias TypeTuple!(T,T[],const(T)[],immutable(T)[],shared(T)[], const T,
immutable T, shared T, const T[], immutable T[], shared T[]) Qual;
     }
     alias TypeTuple!(bool,ubyte,byte,ushort,short,uint,int,ulong,long,float,double,re
l,char,wchar,dchar) ValueTypes;

     alias staticMap!(Qual,ValueTypes) QualifiedTypes;

To generate a large tuple of all the basic D types and then see how much code
is generated.
Aug 24 2011