www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Accurately serializing and deserializing a SysTime in binary format

reply Ecstatic Coder <ecstatic.coder gmail.com> writes:
I'm currently implementing a small open source backup tool (dub), 
and therefore I need to accurately store the file modification 
SysTime in binary format, so that I can later load this SysTime 
from the snapshot file to compare it with the current file 
modification SysTime.

Having unfortunately not understood how to do this from the 
SysTime documentation, in despair, I've tried to directly 
serialize the 16 bytes of the SysTime value. This worked fine 
until I call the ".toISOString()" on the deserialized SysTime, 
which inevitably crashes the executable ;)

Anyway, that's not really want I intended to do, as in practice a 
"ulong" already has enough  resolution for that purpose.

So sorry for my ignorance, but I would definitely need some help 
on how to :
- convert a file modification SysTime to a serializable number, 
for instance the number of hectonanoseconds since 1/1/1970 in UTC;
- convert that number back into a SysTime that I can compare to 
the modification SysTime of the same file.

Eric
Jul 20 2020
next sibling parent Ecstatic Coder <ecstatic.coder gmail.com> writes:
As my question obviously didn't interest any expert, I took 
advantage of my lunch break to do some more research ;)

Maybe I'm wrong, but to my knowledge, there is no function to get 
the number of hectonanoseconds since January 1, 1970.

Fortunately I can get the number of seconds since the same date, 
and the number of remaining hectonanoseconds, and then use them 
in conjunction to create a new "SysTime".

With that I've got everything needed to fix my problem, and as I 
can store those values as two independent "uint", it's easy to 
compress them in the snapshot file, so no regrets :)
Jul 21 2020
prev sibling parent reply drug <drug2004 bk.ru> writes:
On 7/20/20 10:04 PM, Ecstatic Coder wrote:
 I'm currently implementing a small open source backup tool (dub), and 
 therefore I need to accurately store the file modification SysTime in 
 binary format, so that I can later load this SysTime from the snapshot 
 file to compare it with the current file modification SysTime.
 
 Having unfortunately not understood how to do this from the SysTime 
 documentation, in despair, I've tried to directly serialize the 16 bytes 
 of the SysTime value. This worked fine until I call the ".toISOString()" 
 on the deserialized SysTime, which inevitably crashes the executable ;)
That is probably a bug. I serialize SysTime as long by means msgpack for exchanging between C++ client and D server and it works pretty nice.
 
 Anyway, that's not really want I intended to do, as in practice a 
 "ulong" already has enough  resolution for that purpose.
 
 So sorry for my ignorance, but I would definitely need some help on how 
 to :
 - convert a file modification SysTime to a serializable number, for 
 instance the number of hectonanoseconds since 1/1/1970 in UTC;
 - convert that number back into a SysTime that I can compare to the 
 modification SysTime of the same file.
 
 Eric
 
Jul 21 2020
parent reply Ecstatic Coder <ecstatic.coder gmail.com> writes:
On Tuesday, 21 July 2020 at 11:01:20 UTC, drug wrote:
 On 7/20/20 10:04 PM, Ecstatic Coder wrote:
 I'm currently implementing a small open source backup tool 
 (dub), and therefore I need to accurately store the file 
 modification SysTime in binary format, so that I can later 
 load this SysTime from the snapshot file to compare it with 
 the current file modification SysTime.
 
 Having unfortunately not understood how to do this from the 
 SysTime documentation, in despair, I've tried to directly 
 serialize the 16 bytes of the SysTime value. This worked fine 
 until I call the ".toISOString()" on the deserialized SysTime, 
 which inevitably crashes the executable ;)
That is probably a bug. I serialize SysTime as long by means msgpack for exchanging between C++ client and D server and it works pretty nice.
 
 Anyway, that's not really want I intended to do, as in 
 practice a "ulong" already has enough  resolution for that 
 purpose.
 
 So sorry for my ignorance, but I would definitely need some 
 help on how to :
 - convert a file modification SysTime to a serializable 
 number, for instance the number of hectonanoseconds since 
 1/1/1970 in UTC;
 - convert that number back into a SysTime that I can compare 
 to the modification SysTime of the same file.
 
 Eric
Ah thanks for telling me :) The loaded byte array in the union type was indeed the same as the saved one, so I immediately thought it was crashing because of some hidden pointer for timezone or something which was then pointing to garbage at reloading, causing the crash of the ".toISOString" call.
Jul 21 2020
next sibling parent drug <drug2004 bk.ru> writes:
On 7/21/20 2:44 PM, Ecstatic Coder wrote:
 
 Ah thanks for telling me :)
 
 The loaded byte array in the union type was indeed the same as the saved 
 one, so I immediately thought it was crashing because of some hidden 
 pointer for timezone or something which was then pointing to garbage at 
 reloading, causing the crash of the ".toISOString" call.
 
 
Ah, sorry, I serialize exactly long value and use it as SysTime, for example: ```D struct Foo { long value; void toString(Writer)(ref Writer w) const if (isOutputRange!(Writer, char)) { import std.datetime: SysTime; value.SysTime.toUTC.toISOExtString(w); } } ``` So it is not exactly what you said
Jul 21 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 7/21/20 7:44 AM, Ecstatic Coder wrote:
 On Tuesday, 21 July 2020 at 11:01:20 UTC, drug wrote:
 On 7/20/20 10:04 PM, Ecstatic Coder wrote:
 I'm currently implementing a small open source backup tool (dub), and 
 therefore I need to accurately store the file modification SysTime in 
 binary format, so that I can later load this SysTime from the 
 snapshot file to compare it with the current file modification SysTime.

 Having unfortunately not understood how to do this from the SysTime 
 documentation, in despair, I've tried to directly serialize the 16 
 bytes of the SysTime value. This worked fine until I call the 
 ".toISOString()" on the deserialized SysTime, which inevitably 
 crashes the executable ;)
That is probably a bug. I serialize SysTime as long by means msgpack for exchanging between C++ client and D server and it works pretty nice.
Ah thanks for telling me :) The loaded byte array in the union type was indeed the same as the saved one, so I immediately thought it was crashing because of some hidden pointer for timezone or something which was then pointing to garbage at reloading, causing the crash of the ".toISOString" call.
Not a bug. 8 of those 16 bytes is a pointer to the timezone, which is going to be different on different processes. What you should do I think is serialize the stdTime [1], and set the time zone to whatever you want: long serialize(SysTime st) { return st.stdTime; } SysTime deserialize(long st) { return SysTime(st, UTC()); } The stdTime is always stored as UTC to make math a lot easier. The time zone is only used for display. -Steve [1] https://dlang.org/phobos/std_datetime_systime.html#.SysTime.stdTime
Jul 21 2020
parent Ecstatic Coder <ecstatic.coder gmail.com> writes:
On Tuesday, 21 July 2020 at 12:21:16 UTC, Steven Schveighoffer 
wrote:
 On 7/21/20 7:44 AM, Ecstatic Coder wrote:
 On Tuesday, 21 July 2020 at 11:01:20 UTC, drug wrote:
 On 7/20/20 10:04 PM, Ecstatic Coder wrote:
 I'm currently implementing a small open source backup tool 
 (dub), and therefore I need to accurately store the file 
 modification SysTime in binary format, so that I can later 
 load this SysTime from the snapshot file to compare it with 
 the current file modification SysTime.

 Having unfortunately not understood how to do this from the 
 SysTime documentation, in despair, I've tried to directly 
 serialize the 16 bytes of the SysTime value. This worked 
 fine until I call the ".toISOString()" on the deserialized 
 SysTime, which inevitably crashes the executable ;)
That is probably a bug. I serialize SysTime as long by means msgpack for exchanging between C++ client and D server and it works pretty nice.
Ah thanks for telling me :) The loaded byte array in the union type was indeed the same as the saved one, so I immediately thought it was crashing because of some hidden pointer for timezone or something which was then pointing to garbage at reloading, causing the crash of the ".toISOString" call.
Not a bug. 8 of those 16 bytes is a pointer to the timezone, which is going to be different on different processes. What you should do I think is serialize the stdTime [1], and set the time zone to whatever you want: long serialize(SysTime st) { return st.stdTime; } SysTime deserialize(long st) { return SysTime(st, UTC()); } The stdTime is always stored as UTC to make math a lot easier. The time zone is only used for display. -Steve [1] https://dlang.org/phobos/std_datetime_systime.html#.SysTime.stdTime
Smart :) Now I understand my mistake was to try to directly serialize a SysTime as provided by the "getTimes" function, instead of converting it to a StdTime, which is more versatile...
Jul 21 2020