www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - add uda (e.g. csvIgnore) support in std.csv?

reply mw <mingwu gmail.com> writes:
Hi,

I'm trying this example:

https://dlang.org/phobos/std_csv.html

```
struct Layout
{
     string name;
     int value;
     double other;

      csvIgnore
     Layout* parent;  // add extra pointer
}

auto records = text.csvReader!Layout(';');
```

Right now, if we add this extra pointer, the compiler will error 
out:

https://run.dlang.io/is/EvU6Ee

/dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): Error: 
template std.conv.toImpl cannot deduce function from argument 
types !(Layout*)(string), candidates are:
...

I think if we can add add uda (e.g.  csvIgnore) support in 
std.csv, it will make the library more flexible to use.

Thoughts?
Sep 07 2020
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-09-08 06:53, mw wrote:
 Hi,
 
 I'm trying this example:
 
 https://dlang.org/phobos/std_csv.html
 
 ```
 struct Layout
 {
      string name;
      int value;
      double other;
 
       csvIgnore
      Layout* parent;  // add extra pointer
 }
 
 auto records = text.csvReader!Layout(';');
 ```
 
 Right now, if we add this extra pointer, the compiler will error out:
 
 https://run.dlang.io/is/EvU6Ee
 
 /dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): Error: template 
 std.conv.toImpl cannot deduce function from argument types 
 !(Layout*)(string), candidates are:
 ...
 
 I think if we can add add uda (e.g.  csvIgnore) support in std.csv, it 
 will make the library more flexible to use.
How about a more general attribute in `core.attribute`, like ` nonSerialized`? This could be used by third party serialization libraries. Then it could also be used by druntime and Phobos to ignore things that should not be serializable. Like threads, processes and sockets. -- /Jacob Carlborg
Sep 10 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Thursday, 10 September 2020 at 14:57:11 UTC, Jacob Carlborg 
wrote:
 On 2020-09-08 06:53, mw wrote:
 Hi,
 
 I'm trying this example:
 
 https://dlang.org/phobos/std_csv.html
 
 ```
 struct Layout
 {
      string name;
      int value;
      double other;
 
       csvIgnore
      Layout* parent;  // add extra pointer
 }
 
 auto records = text.csvReader!Layout(';');
 ```
 
 Right now, if we add this extra pointer, the compiler will 
 error out:
 
 https://run.dlang.io/is/EvU6Ee
 
 /dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): 
 Error: template std.conv.toImpl cannot deduce function from 
 argument types !(Layout*)(string), candidates are:
 ...
 
 I think if we can add add uda (e.g.  csvIgnore) support in 
 std.csv, it will make the library more flexible to use.
How about a more general attribute in `core.attribute`, like ` nonSerialized`? This could be used by third party serialization libraries. Then it could also be used by druntime and Phobos to ignore things that should not be serializable. Like threads, processes and sockets.
Existing serialization libraries [1][2] already define their own attributes for this sort of thing. I don't think there's anything to be gained by attempting to force standardization here. If Phobos and/or the D ecosystem does someday decide to standardize on a shared ignore attribute, std.csv can be easily upgraded for compatibility by adding `alias csvIgnore = core.attribute.ignore`. [1] https://code.dlang.org/packages/asdf [2] https://vibed.org/api/vibe.data.serialization/
Sep 10 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/10/20 1:57 PM, Paul Backus wrote:
 On Thursday, 10 September 2020 at 14:57:11 UTC, Jacob Carlborg wrote:
 How about a more general attribute in `core.attribute`, like 
 ` nonSerialized`? This could be used by third party serialization 
 libraries. Then it could also be used by druntime and Phobos to ignore 
 things that should not be serializable. Like threads, processes and 
 sockets.
Existing serialization libraries already define their own attributes for this sort of thing. I don't think there's anything to be gained by attempting to force standardization here. If Phobos and/or the D ecosystem does someday decide to standardize on a shared ignore attribute, std.csv can be easily upgraded for compatibility by adding `alias csvIgnore = core.attribute.ignore`.
To add to this, I have code that serializes the same types for both vibe.d JSON for REST interfaces, AND to a database. In one case, I want certain fields to be ignored (e.g. no reason to send internal database ids to the browser), and in another case I want other fields to be ignored. I'd rather have libraries define their own UDAs, and not expect Phobos to design a UDA system that pleases all parties. -Steve
Sep 10 2020
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 10 September 2020 at 19:11:03 UTC, Steven 
Schveighoffer wrote:
 To add to this, I have code that serializes the same types for 
 both vibe.d JSON for REST interfaces, AND to a database. In one 
 case, I want certain fields to be ignored (e.g. no reason to 
 send internal database ids to the browser), and in another case 
 I want other fields to be ignored.

 I'd rather have libraries define their own UDAs, and not expect 
 Phobos to design a UDA system that pleases all parties.

 -Steve
I have sometimes used a dedicated struct just to let the serialiser know how to deal with the fields. It would mostly be a copy of the original struct but with some udas here and there. It allows me to have different shapes even when targeting the same format. Wrt to csv, you could serialise into an intermediate struct first, and from that into your target struct. It wouldn't be hard to convert struct a { string timestamp, name; double value; } into struct ab { Datetime timestamp; string name; double value; }
Oct 09 2020
prev sibling parent reply 9il <ilyayaroshenko gmail.com> writes:
On Thursday, 10 September 2020 at 14:57:11 UTC, Jacob Carlborg 
wrote:
 On 2020-09-08 06:53, mw wrote:
 [...]
How about a more general attribute in `core.attribute`, like ` nonSerialized`? This could be used by third party serialization libraries. Then it could also be used by druntime and Phobos to ignore things that should not be serializable. Like threads, processes and sockets.
Ready to use module for this purpose. http://mir-algorithm.libmir.org/mir_serde.html
Oct 10 2020
parent reply mw <mingwu gmail.com> writes:
On Saturday, 10 October 2020 at 10:13:55 UTC, 9il wrote:
 Ready to use module for this purpose.
 http://mir-algorithm.libmir.org/mir_serde.html
Nice, but this is in a 3rd-party library, and std.csv is the DLang standard library. Maybe can you create a PR just contribute this module to std? (I've read there are some issues regarding licensing to contribute the whole mir to stb lib).
Oct 10 2020
parent reply 9il <ilyayaroshenko gmail.com> writes:
On Saturday, 10 October 2020 at 16:52:32 UTC, mw wrote:
 On Saturday, 10 October 2020 at 10:13:55 UTC, 9il wrote:
 Ready to use module for this purpose.
 http://mir-algorithm.libmir.org/mir_serde.html
Nice, but this is in a 3rd-party library, and std.csv is the DLang standard library. Maybe can you create a PR just contribute this module to std? (I've read there are some issues regarding licensing to contribute the whole mir to stb lib).
We are making Mir the real standard D library.
Oct 11 2020
parent reply mw <mingwu gmail.com> writes:
On Sunday, 11 October 2020 at 14:04:47 UTC, 9il wrote:
 On Saturday, 10 October 2020 at 16:52:32 UTC, mw wrote:
 On Saturday, 10 October 2020 at 10:13:55 UTC, 9il wrote:
 Ready to use module for this purpose.
 http://mir-algorithm.libmir.org/mir_serde.html
Nice, but this is in a 3rd-party library, and std.csv is the DLang standard library. Maybe can you create a PR just contribute this module to std? (I've read there are some issues regarding licensing to contribute the whole mir to stb lib).
We are making Mir the real standard D library.
This is excellent news. BTW, what namespace it will use? std.???? From a purely marketing / promotional perspective, may I suggest numD? People will immediately recognize numPy => numD, and if we can even make the interface the same, people may adopt it more quickly. I know there is numir project, but there are several repos under https://github.com/libmir, as a user I sometimes got confused which one to use, can they *all* be merged into a single library numD?
Oct 11 2020
parent 9il <ilyayaroshenko gmail.com> writes:
On Sunday, 11 October 2020 at 16:15:30 UTC, mw wrote:
 On Sunday, 11 October 2020 at 14:04:47 UTC, 9il wrote:
 On Saturday, 10 October 2020 at 16:52:32 UTC, mw wrote:
 On Saturday, 10 October 2020 at 10:13:55 UTC, 9il wrote:
 Ready to use module for this purpose.
 http://mir-algorithm.libmir.org/mir_serde.html
Nice, but this is in a 3rd-party library, and std.csv is the DLang standard library. Maybe can you create a PR just contribute this module to std? (I've read there are some issues regarding licensing to contribute the whole mir to stb lib).
We are making Mir the real standard D library.
This is excellent news. BTW, what namespace it will use? std.????
mir.* std.* is used by Phobos and it is fine. It isn't the same as it was in D1 with two standard libraries. Mir is compatible with Phobos.
 From a purely marketing / promotional perspective, may I 
 suggest numD?
Mir isn't limited by numeric code. For example, mir.rc (mir-algorithm) provides a production-ready full-featured ref-counted type system.
 People will immediately recognize numPy => numD, and if we can 
 even make the interface the same, people may adopt it more 
 quickly. I know there is numir project, but there are several 
 repos under https://github.com/libmir, as a user I sometimes 
 got confused which one to use, can they *all* be merged into a 
 single library numD?
Different code of different purpose needs different maintaining rules and coding standards.
Oct 11 2020
prev sibling parent reply mw <mingwu gmail.com> writes:
On Tuesday, 8 September 2020 at 04:53:26 UTC, mw wrote:
 Hi,

 I'm trying this example:

 https://dlang.org/phobos/std_csv.html

 ```
 struct Layout
 {
     string name;
     int value;
     double other;

      csvIgnore
     Layout* parent;  // add extra pointer
 }

 auto records = text.csvReader!Layout(';');
 ```

 Right now, if we add this extra pointer, the compiler will 
 error out:

 https://run.dlang.io/is/EvU6Ee

 /dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): Error: 
 template std.conv.toImpl cannot deduce function from argument 
 types !(Layout*)(string), candidates are:
 ...

 I think if we can add add uda (e.g.  csvIgnore) support in 
 std.csv, it will make the library more flexible to use.

 Thoughts?
I do run into this problem today, i.e. my struct need some extra non-simple-scalar fields (e.g. SysTime) other than those defined in the csv file, I'm wondering what kind of work-around people use in such situation? Thanks.
Oct 09 2020
parent mw <mingwu gmail.com> writes:
On Saturday, 10 October 2020 at 04:31:20 UTC, mw wrote:
 On Tuesday, 8 September 2020 at 04:53:26 UTC, mw wrote:
 Hi,

 I'm trying this example:

 https://dlang.org/phobos/std_csv.html

 ```
 struct Layout
 {
     string name;
     int value;
     double other;

      csvIgnore
     Layout* parent;  // add extra pointer
 }

 auto records = text.csvReader!Layout(';');
 ```

 Right now, if we add this extra pointer, the compiler will 
 error out:

 https://run.dlang.io/is/EvU6Ee

 /dlang/dmd/linux/bin64/../../src/phobos/std/conv.d(223): 
 Error: template std.conv.toImpl cannot deduce function from 
 argument types !(Layout*)(string), candidates are:
 ...

 I think if we can add add uda (e.g.  csvIgnore) support in 
 std.csv, it will make the library more flexible to use.

 Thoughts?
I do run into this problem today, i.e. my struct need some extra non-simple-scalar fields (e.g. SysTime) other than those defined in the csv file, I'm wondering what kind of work-around people use in such situation? Thanks.
Run into this problem yet again: ``` class Foo { Foo[] foos; // how to mark this field to be csv ignored? } ``` Any work around? Thanks.
Sep 24 2022