www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [GSoC] Dataframes for D

reply Prateek Nayak <lelouch.cpp gmail.com> writes:
Hello everyone,

I have began work on my Google Summer of Code 2019 project 
DataFrame for D.

-----------------
About the Project
-----------------

DataFrames have become a standard while handling and manipulating 
data. They give a neat representation, access and power to 
modulate the data in way user wants.
This project aims at bringing native DataFrame to D one which 
brings with it:

* A User Friendly API
* Multi - Indexing
* Writing to CSV and parsing from CSV
* Column binary operation in the form: df["Index1"] = 
df["Index2"] + df["Index3"];
* groupBy on an arbitrary number of columns
* Data Aggregation

Disclaimer: The entire structuring was inspired by Pandas, the 
most popular DataFrame library in Python and hence most of the 
usage will look very similar to the ones in Pandas.

Main focus of this project is user-friendliness of the API while 
also maintaining fair amount of speed and power.
The preliminary road map can be viewed here -> 
https://docs.google.com/document/d/1Zrf_tFYLauAd_NM4-UMBGt_z-fORhFMrGvW633x8rZs/edit?usp=sharing

The core developments can be seen here -> 
https://github.com/Kriyszig/magpie


-----------------------------
Brief idea of what is to come
-----------------------------

This month
----------
* Finish up with structure of DataFrame
* Finish Terminal Output (What good is data which cannot be seen)
* Finish writing to CSV
* Parsing DataFrame from CSV (Both single and multi-indexed)
* Accessing Elements
* Accessing Rows and Columns
* Assignment of element, an entire row or column
* Binary operation on rows and columns

Next Month
----------
* groupBy
* join
* Begin writing ops for aggregation


-----------
Speed Bumps
-----------

I am relatively new to D and hail from functional C background. 
Sometimes (most of the times) my code can start to look more C 
than D.
However I am adapting thanks to my mentors Nicholas Wilson and 
Ilya Yaroshenko. They have helped me a ton - whether it be with 
debugging errors or me falling back to my functional C past, they 
have always come for my rescue and I am grateful for their 
support.


-------------------------------------
Addressing Suggestions from Community
-------------------------------------

This suggestion comes from Laeeth Isharc
Source: 
https://github.com/dlang/projects/issues/15#issuecomment-495831750

Though this is not on my current road map, I would love to pursue 
this idea. Adding an easy way to inter operate with other 
libraries would be very beneficial.
Although I haven't formally addressed this in the road map, I 
would love to implement a msgpack based I/O as I continue to 
develop the library. Also JSON I/O was something on my mind to 
implement after the data aggregation part.  (I had prioritised 
JSON as I believed there were much more datasets as JSON compared 
to any other format)
May 29
next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 29 May 2019 at 18:00:02 UTC, Prateek Nayak wrote:
 Hello everyone,

 I have began work on my Google Summer of Code 2019 project 
 DataFrame for D.

 -----------------
 About the Project
 -----------------

 DataFrames have become a standard while handling and 
 manipulating data. They give a neat representation, access and 
 power to modulate the data in way user wants.
 This project aims at bringing native DataFrame to D one which 
 brings with it:

 * A User Friendly API
 * Multi - Indexing
 * Writing to CSV and parsing from CSV
 * Column binary operation in the form: df["Index1"] = 
 df["Index2"] + df["Index3"];
 * groupBy on an arbitrary number of columns
 * Data Aggregation

 Disclaimer: The entire structuring was inspired by Pandas, the 
 most popular DataFrame library in Python and hence most of the 
 usage will look very similar to the ones in Pandas.

 Main focus of this project is user-friendliness of the API 
 while also maintaining fair amount of speed and power.
 The preliminary road map can be viewed here -> 
 https://docs.google.com/document/d/1Zrf_tFYLauAd_NM4-UMBGt_z-fORhFMrGvW633x8rZs/edit?usp=sharing

 The core developments can be seen here -> 
 https://github.com/Kriyszig/magpie


 -----------------------------
 Brief idea of what is to come
 -----------------------------

 This month
 ----------
 * Finish up with structure of DataFrame
 * Finish Terminal Output (What good is data which cannot be 
 seen)
 * Finish writing to CSV
 * Parsing DataFrame from CSV (Both single and multi-indexed)
 * Accessing Elements
 * Accessing Rows and Columns
 * Assignment of element, an entire row or column
 * Binary operation on rows and columns

 Next Month
 ----------
 * groupBy
 * join
 * Begin writing ops for aggregation


 -----------
 Speed Bumps
 -----------

 I am relatively new to D and hail from functional C background. 
 Sometimes (most of the times) my code can start to look more C 
 than D.
 However I am adapting thanks to my mentors Nicholas Wilson and 
 Ilya Yaroshenko. They have helped me a ton - whether it be with 
 debugging errors or me falling back to my functional C past, 
 they have always come for my rescue and I am grateful for their 
 support.


 -------------------------------------
 Addressing Suggestions from Community
 -------------------------------------

 This suggestion comes from Laeeth Isharc
 Source: 
 https://github.com/dlang/projects/issues/15#issuecomment-495831750

 Though this is not on my current road map, I would love to 
 pursue this idea. Adding an easy way to inter operate with 
 other libraries would be very beneficial.
 Although I haven't formally addressed this in the road map, I 
 would love to implement a msgpack based I/O as I continue to 
 develop the library. Also JSON I/O was something on my mind to 
 implement after the data aggregation part.  (I had prioritised 
 JSON as I believed there were much more datasets as JSON 
 compared to any other format)
The outlook looks really great. CSV as a starting point is good. I am not sure wheter adding a second text format like json brings real value. Text formats have 2 disadvantages, string to number and vise versa slows down loading and saving files. If I remember correctly saving a file as CSV needed around 50 seconds while saving the same data as binary file (Parquet) took 3 seconds. Second issue is the file size. A CSV with a size of 580 MB has a size of 280 MB while saving as e.g. Parquet file. The file size isn't an issue on your local file system but it is a big issue while storing these file in the cloud e.g. Amazon S3. The file size will cause longer transfer times. Adding a packed binary format would be great, if possible. Kind regards Andre
May 29
parent reply Laeeth Isharc <laeeth kaleidic.io> writes:
On Wednesday, 29 May 2019 at 18:26:46 UTC, Andre Pany wrote:
 On Wednesday, 29 May 2019 at 18:00:02 UTC, Prateek Nayak wrote:
 Hello everyone,

 I have began work on my Google Summer of Code 2019 project 
 DataFrame for D.

 -----------------
 About the Project
 -----------------

 DataFrames have become a standard while handling and 
 manipulating data. They give a neat representation, access and 
 power to modulate the data in way user wants.
 This project aims at bringing native DataFrame to D one which 
 brings with it:

 * A User Friendly API
 * Multi - Indexing
 * Writing to CSV and parsing from CSV
 * Column binary operation in the form: df["Index1"] = 
 df["Index2"] + df["Index3"];
 * groupBy on an arbitrary number of columns
 * Data Aggregation

 Disclaimer: The entire structuring was inspired by Pandas, the 
 most popular DataFrame library in Python and hence most of the 
 usage will look very similar to the ones in Pandas.

 Main focus of this project is user-friendliness of the API 
 while also maintaining fair amount of speed and power.
 The preliminary road map can be viewed here -> 
 https://docs.google.com/document/d/1Zrf_tFYLauAd_NM4-UMBGt_z-fORhFMrGvW633x8rZs/edit?usp=sharing

 The core developments can be seen here -> 
 https://github.com/Kriyszig/magpie


 -----------------------------
 Brief idea of what is to come
 -----------------------------

 This month
 ----------
 * Finish up with structure of DataFrame
 * Finish Terminal Output (What good is data which cannot be 
 seen)
 * Finish writing to CSV
 * Parsing DataFrame from CSV (Both single and multi-indexed)
 * Accessing Elements
 * Accessing Rows and Columns
 * Assignment of element, an entire row or column
 * Binary operation on rows and columns

 Next Month
 ----------
 * groupBy
 * join
 * Begin writing ops for aggregation


 -----------
 Speed Bumps
 -----------

 I am relatively new to D and hail from functional C 
 background. Sometimes (most of the times) my code can start to 
 look more C than D.
 However I am adapting thanks to my mentors Nicholas Wilson and 
 Ilya Yaroshenko. They have helped me a ton - whether it be 
 with debugging errors or me falling back to my functional C 
 past, they have always come for my rescue and I am grateful 
 for their support.


 -------------------------------------
 Addressing Suggestions from Community
 -------------------------------------

 This suggestion comes from Laeeth Isharc
 Source: 
 https://github.com/dlang/projects/issues/15#issuecomment-495831750

 Though this is not on my current road map, I would love to 
 pursue this idea. Adding an easy way to inter operate with 
 other libraries would be very beneficial.
 Although I haven't formally addressed this in the road map, I 
 would love to implement a msgpack based I/O as I continue to 
 develop the library. Also JSON I/O was something on my mind to 
 implement after the data aggregation part.  (I had prioritised 
 JSON as I believed there were much more datasets as JSON 
 compared to any other format)
The outlook looks really great. CSV as a starting point is good. I am not sure wheter adding a second text format like json brings real value. Text formats have 2 disadvantages, string to number and vise versa slows down loading and saving files. If I remember correctly saving a file as CSV needed around 50 seconds while saving the same data as binary file (Parquet) took 3 seconds. Second issue is the file size. A CSV with a size of 580 MB has a size of 280 MB while saving as e.g. Parquet file. The file size isn't an issue on your local file system but it is a big issue while storing these file in the cloud e.g. Amazon S3. The file size will cause longer transfer times. Adding a packed binary format would be great, if possible. Kind regards Andre
Interoperability with pandas would be important in our use case, and I think probably for quite a few others. So yes I agree that it's not ideal to use JSON but lots of things are not ideal. And I think people use JSON, msgpack and hdf5 for interop with pandas. CSV is more complicated in practice than one might initially think. Finally of course Excel interop. It's not my cup of tea but gzipped JSON is quite compact... I have a little streaming msgpack to our own Variable type deserialiser (it can store a primitive or a variant). It's not long and I could share that. I don't think the initial dataframe needs to have all this stuff in it from day one necessarily. It's worth reusing another JSON implementation. Since you work with Ilya - asdf isn't bad and quite fast though the error messages leave something to be desired. You might see John Colvin repo from his talk on nogc map, filter,fold etc. He didn't do chunkBy yet.
May 29
parent Prateek Nayak <lelouch.cpp gmail.com> writes:
On Wednesday, 29 May 2019 at 22:31:57 UTC, Laeeth Isharc wrote:
 On Wednesday, 29 May 2019 at 18:26:46 UTC, Andre Pany wrote:
 On Wednesday, 29 May 2019 at 18:00:02 UTC, Prateek Nayak wrote:
 [...]
The outlook looks really great. CSV as a starting point is good. I am not sure wheter adding a second text format like json brings real value. Text formats have 2 disadvantages, string to number and vise versa slows down loading and saving files. If I remember correctly saving a file as CSV needed around 50 seconds while saving the same data as binary file (Parquet) took 3 seconds. Second issue is the file size. A CSV with a size of 580 MB has a size of 280 MB while saving as e.g. Parquet file. The file size isn't an issue on your local file system but it is a big issue while storing these file in the cloud e.g. Amazon S3. The file size will cause longer transfer times. Adding a packed binary format would be great, if possible. Kind regards Andre
Interoperability with pandas would be important in our use case, and I think probably for quite a few others. So yes I agree that it's not ideal to use JSON but lots of things are not ideal. And I think people use JSON, msgpack and hdf5 for interop with pandas. CSV is more complicated in practice than one might initially think. Finally of course Excel interop. It's not my cup of tea but gzipped JSON is quite compact... I have a little streaming msgpack to our own Variable type deserialiser (it can store a primitive or a variant). It's not long and I could share that. I don't think the initial dataframe needs to have all this stuff in it from day one necessarily. It's worth reusing another JSON implementation. Since you work with Ilya - asdf isn't bad and quite fast though the error messages leave something to be desired. You might see John Colvin repo from his talk on nogc map, filter,fold etc. He didn't do chunkBy yet.
Interop is important and I see many binary format being suggested as a way of interop. Pandas I/O tools cover some of the popular formats: https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html Apache Arrow was something new that I hadn't heard of before. I'll look into it but then again I couldn't find any way to integrate it with D right away. I would love to hear which binary format does the community want to be added as a way of interop in the future.
May 29
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 29 May 2019 at 18:00:02 UTC, Prateek Nayak wrote:
 [snip]
Glad to see progress being made on this! Somewhat tangentially on the interoperability, I have also made quite a bit of use with R's data.frames. One difference between that and what I have seen of the implementation is that R's data.frames allow for different columns to be different types. This makes certain kinds of analysis of groups very easy. For instance, right now I'm working with a dataset whose columns are doubles, dates, integers, bools, and strings. I can do the equivalent of groupby on the strings as "factors" in R and it's pretty straightforward to get everything working nicely.
May 29
next sibling parent reply Yatheendra <3df4 gmail.ru> writes:
Is it worth considering some binary format that is standard-ish & 
whose toolset can write out text when viewing? I'm thinking 
https://capnproto.org .
May 29
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 29 May 2019 at 19:33:38 UTC, Yatheendra wrote:
 Is it worth considering some binary format that is standard-ish 
 & whose toolset can write out text when viewing? I'm thinking 
 https://capnproto.org .
I have heard of that, but I don't know too much about it. I think there are some hierarchical data formats that have some popularity, like hdf5. I think there's already a D wrapper for the C library. I don't have much experience with this kind of stuff though.
May 29
prev sibling parent Laeeth Isharc <laeeth kaleidic.io> writes:
On Wednesday, 29 May 2019 at 19:33:38 UTC, Yatheendra wrote:
 Is it worth considering some binary format that is standard-ish 
 & whose toolset can write out text when viewing? I'm thinking 
 https://capnproto.org .
Ultimately you need to be able to talk to the world because this kind of thing is social and you may not have a choice about formats. However no point trying to do it all in one summer...
May 29
prev sibling next sibling parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Wednesday, 29 May 2019 at 18:41:28 UTC, jmh530 wrote:
 On Wednesday, 29 May 2019 at 18:00:02 UTC, Prateek Nayak wrote:
 [snip]
Glad to see progress being made on this! Somewhat tangentially on the interoperability, I have also made quite a bit of use with R's data.frames. One difference between that and what I have seen of the implementation is that R's data.frames allow for different columns to be different types. This makes certain kinds of analysis of groups very easy. For instance, right now I'm working with a dataset whose columns are doubles, dates, integers, bools, and strings. I can do the equivalent of groupby on the strings as "factors" in R and it's pretty straightforward to get everything working nicely.
The DataFrame currently uses Mir's ndslice at the core of it which allows for homogeneous data to be stored within it. Right now, we are considering operable data to be homogeneous keeping the API simpler. I'm not sure how something like Variant will play out in this scenario. It may allow for data to be flexible but parsing will probably require an assertion library.
May 29
parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 30 May 2019 at 03:38:50 UTC, Prateek Nayak wrote:
 [snip]

 The DataFrame currently uses Mir's ndslice at the core of it 
 which allows for homogeneous data to be stored within it.
 Right now, we are considering operable data to be homogeneous 
 keeping the API simpler.
 I'm not sure how something like Variant will play out in this 
 scenario. It may allow for data to be flexible but parsing will 
 probably require an assertion library.
It's probably smart to focus on getting the homogeneous case working first. I don't think of it as the entire thing being Variant, so much as a tuple containing 1-dimensional mir slices that are all the same length. The idea is that each column should have its own type. I had done a simple implementation of this a year or so ago and had shown Ilya.
May 30
prev sibling next sibling parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Wednesday, 29 May 2019 at 18:41:28 UTC, jmh530 wrote:
 On Wednesday, 29 May 2019 at 18:00:02 UTC, Prateek Nayak wrote:
 [snip]
Glad to see progress being made on this! Somewhat tangentially on the interoperability, I have also made quite a bit of use with R's data.frames. One difference between that and what I have seen of the implementation is that R's data.frames allow for different columns to be different types. This makes certain kinds of analysis of groups very easy. For instance, right now I'm working with a dataset whose columns are doubles, dates, integers, bools, and strings. I can do the equivalent of groupby on the strings as "factors" in R and it's pretty straightforward to get everything working nicely.
On a second thought, my mentor Nicholas Wilson led me to an interesting Github Gist -> https://gist.github.com/aG0aep6G/a1b87df1ac5930870ffe A similar structure can be used to represent non homogeneous data. The DataFrame structure can be overloaded for such an integration. However homogeneous DataFrame still remain the main objective for now. This integration will definitely happen once the homogeneous DataFrame comes close to looking and working like an actual DataFrame. I'll keep you updated here in case I find anything better for non homogeneous data and when the whole things starts to take shape.
May 29
parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 30 May 2019 at 04:38:09 UTC, Prateek Nayak wrote:
 

 On a second thought, my mentor Nicholas Wilson led me to an 
 interesting Github Gist
 -> https://gist.github.com/aG0aep6G/a1b87df1ac5930870ffe

 A similar structure can be used to represent non homogeneous 
 data. The DataFrame structure can be overloaded for such an 
 integration. However homogeneous DataFrame still remain the 
 main objective for now. This integration will definitely happen 
 once the homogeneous DataFrame comes close to looking and 
 working like an actual DataFrame.

 I'll keep you updated here in case I find anything better for 
 non homogeneous data and when the whole things starts to take 
 shape.
Hmm, my point above was for a tuple of mir slices, which seems to correspond more to a struct of arrays. I wonder if your implementation would be able to take an array of structs approach using built-in mir. I know mir can handle complex numbers, which requires a struct. I don't know off the top of my head how it can handle more generally a slice whose type is a struct.
May 30
prev sibling parent reply welkam <wwwelkam gmail.com> writes:
On Wednesday, 29 May 2019 at 18:41:28 UTC, jmh530 wrote:
 R's data.frames allow for different columns to be different 
 types.
Im not familiar with data frames but this sounds like array of structs
May 30
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 30 May 2019 at 09:06:58 UTC, welkam wrote:
 [snip]

 Im not familiar with data frames but this sounds like array of 
 structs
I think I was thinking of it more like a struct of arrays, but I think an array of structs may also work (see my responses to Prateek)...
May 30
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Thursday, 30 May 2019 at 09:41:59 UTC, jmh530 wrote:
 On Thursday, 30 May 2019 at 09:06:58 UTC, welkam wrote:
 [snip]

 Im not familiar with data frames but this sounds like array of 
 structs
I think I was thinking of it more like a struct of arrays, but I think an array of structs may also work (see my responses to Prateek)...
Due to the popularity of heterogeneous DataFrames, we decided to take care of it the early stages of development before it's too late. The heterogeneous DataFrame is now live at: https://github.com/Kriyszig/magpie/tree/experimental Some parts are still under development but the goals in the road maps will be reached on time. --------------------------------- Summing up the first week of GSoC --------------------------------- * Base and file I/O ops were built for homogeneous DataFrame * Based on the type of data community has worked with, it seemed evident homogeneous DataFrames weren't gonna cut it. So a rebuild was initiated over the weekend to allow for heterogeneous data. * The API was overhauled to allow for Heterogeneous DataFrames. * New parser that can parse selective columns. The code will land in master once it's cleaned up and is deemed stable. ----------------------------------- Things that will be dealt this week ----------------------------------- This week will be for: * Improving Parser * Overhaul code structure (in Experimental) * Adding setters for data and index in DataFrame * Adding functions to create a multi-indexed DataFrame, the same way one can do in Python. * Adding Documentation and examples * Index Operations * Retrieve rows and columns The last one will set in motion the implementation of Column Binary ops of the form: df["Index1"] = df["Index2"] + df["Index3"]; Meanwhile if you guys have any more suggestion please feel free to contact me - you can use this thread, open an issue on Github, reach out to me on slack (Prateek Nayak) or you can email me directly (lelouch.cpp gmail.com)
Jun 03
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 4 June 2019 at 03:13:03 UTC, Prateek Nayak wrote:
 [snip]

 Due to the popularity of heterogeneous DataFrames, we decided 
 to take care of it the early stages of development before it's 
 too late.
Excellent!
Jun 04
prev sibling next sibling parent reply James Blachly <james.blachly gmail.com> writes:
On 6/3/19 11:13 PM, Prateek Nayak wrote:
 Due to the popularity of heterogeneous DataFrames, we decided to take 
 care of it the early stages of development before it's too late.
 
 The heterogeneous DataFrame is now live at: 
 https://github.com/Kriyszig/magpie/tree/experimental
Amazing, thanks! experimental branch readme code snippet has typo; var name should be `heterogeneous` ``` // Creating a heterogeneous DataFrame of 10 integer columns and 10 double columns DataFrame!(int, 10, double, 10) homogeneous; ``` Again I cannot thank you enough!
Jun 05
parent Prateek Nayak <lelouch.cpp gmail.com> writes:
On Thursday, 6 June 2019 at 00:24:17 UTC, James Blachly wrote:
 On 6/3/19 11:13 PM, Prateek Nayak wrote:
 Due to the popularity of heterogeneous DataFrames, we decided 
 to take care of it the early stages of development before it's 
 too late.
 
 The heterogeneous DataFrame is now live at: 
 https://github.com/Kriyszig/magpie/tree/experimental
Amazing, thanks! experimental branch readme code snippet has typo; var name should be `heterogeneous` ``` // Creating a heterogeneous DataFrame of 10 integer columns and 10 double columns DataFrame!(int, 10, double, 10) homogeneous; ``` Again I cannot thank you enough!
Oops! Thanks for spotting that. I'll update it today with a complete example for the usage and a snippet each for the functions added since it was last modified.
Jun 05
prev sibling parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Tuesday, 4 June 2019 at 03:13:03 UTC, Prateek Nayak wrote:
 On Thursday, 30 May 2019 at 09:41:59 UTC, jmh530 wrote:
 On Thursday, 30 May 2019 at 09:06:58 UTC, welkam wrote:
 [snip]

 Im not familiar with data frames but this sounds like array 
 of structs
I think I was thinking of it more like a struct of arrays, but I think an array of structs may also work (see my responses to Prateek)...
Due to the popularity of heterogeneous DataFrames, we decided to take care of it the early stages of development before it's too late. The heterogeneous DataFrame is now live at: https://github.com/Kriyszig/magpie/tree/experimental Some parts are still under development but the goals in the road maps will be reached on time. --------------------------------- Summing up the first week of GSoC --------------------------------- * Base and file I/O ops were built for homogeneous DataFrame * Based on the type of data community has worked with, it seemed evident homogeneous DataFrames weren't gonna cut it. So a rebuild was initiated over the weekend to allow for heterogeneous data. * The API was overhauled to allow for Heterogeneous DataFrames. * New parser that can parse selective columns. The code will land in master once it's cleaned up and is deemed stable. ----------------------------------- Things that will be dealt this week ----------------------------------- This week will be for: * Improving Parser * Overhaul code structure (in Experimental) * Adding setters for data and index in DataFrame * Adding functions to create a multi-indexed DataFrame, the same way one can do in Python. * Adding Documentation and examples * Index Operations * Retrieve rows and columns The last one will set in motion the implementation of Column Binary ops of the form: df["Index1"] = df["Index2"] + df["Index3"]; Meanwhile if you guys have any more suggestion please feel free to contact me - you can use this thread, open an issue on Github, reach out to me on slack (Prateek Nayak) or you can email me directly (lelouch.cpp gmail.com)
I have decided to make weekly updates for the community to know the progress of the project. ------------------------------- Summing up last week's progress ------------------------------- * Brought Heterogeneous DataFrames to the same point as the previous Homogeneous DataFrame development. * Assignment ops - both direct and indexed. * Retrieve an entire column using index operation. * Retrieving entire row using index operation. * Small redesign here and there to reduce code size. The index op for rows and columns will return the value as an Axis structure. Binary Ops on Axis structure will basically translate to Column and Row binary operation on DataFrame. ---------------------------------- Tasks that will be dealt this week ---------------------------------- * Column and row binary operation * There are a few places where O(n) operations can be converted to O(log n) operations - these few optimisations will be done. * Updating the Documentation with the developments of the week. So far no major roadblocks have been encountered. P.S. Found this interesting Reddit post regarding file formats on r/datasets: https://www.reddit.com/r/datasets/comments/bymru3/are_there_any_data_formats_for_storing_text_worth/
Jun 10
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Tuesday, 11 June 2019 at 04:35:22 UTC, Prateek Nayak wrote:
 On Tuesday, 4 June 2019 at 03:13:03 UTC, Prateek Nayak wrote:
 On Thursday, 30 May 2019 at 09:41:59 UTC, jmh530 wrote:
 On Thursday, 30 May 2019 at 09:06:58 UTC, welkam wrote:
 [snip]

 Im not familiar with data frames but this sounds like array 
 of structs
I think I was thinking of it more like a struct of arrays, but I think an array of structs may also work (see my responses to Prateek)...
Due to the popularity of heterogeneous DataFrames, we decided to take care of it the early stages of development before it's too late. The heterogeneous DataFrame is now live at: https://github.com/Kriyszig/magpie/tree/experimental Some parts are still under development but the goals in the road maps will be reached on time. --------------------------------- Summing up the first week of GSoC --------------------------------- * Base and file I/O ops were built for homogeneous DataFrame * Based on the type of data community has worked with, it seemed evident homogeneous DataFrames weren't gonna cut it. So a rebuild was initiated over the weekend to allow for heterogeneous data. * The API was overhauled to allow for Heterogeneous DataFrames. * New parser that can parse selective columns. The code will land in master once it's cleaned up and is deemed stable. ----------------------------------- Things that will be dealt this week ----------------------------------- This week will be for: * Improving Parser * Overhaul code structure (in Experimental) * Adding setters for data and index in DataFrame * Adding functions to create a multi-indexed DataFrame, the same way one can do in Python. * Adding Documentation and examples * Index Operations * Retrieve rows and columns The last one will set in motion the implementation of Column Binary ops of the form: df["Index1"] = df["Index2"] + df["Index3"]; Meanwhile if you guys have any more suggestion please feel free to contact me - you can use this thread, open an issue on Github, reach out to me on slack (Prateek Nayak) or you can email me directly (lelouch.cpp gmail.com)
I have decided to make weekly updates for the community to know the progress of the project. ------------------------------- Summing up last week's progress ------------------------------- * Brought Heterogeneous DataFrames to the same point as the previous Homogeneous DataFrame development. * Assignment ops - both direct and indexed. * Retrieve an entire column using index operation. * Retrieving entire row using index operation. * Small redesign here and there to reduce code size. The index op for rows and columns will return the value as an Axis structure. Binary Ops on Axis structure will basically translate to Column and Row binary operation on DataFrame. ---------------------------------- Tasks that will be dealt this week ---------------------------------- * Column and row binary operation * There are a few places where O(n) operations can be converted to O(log n) operations - these few optimisations will be done. * Updating the Documentation with the developments of the week. So far no major roadblocks have been encountered. P.S. Found this interesting Reddit post regarding file formats on r/datasets: https://www.reddit.com/r/datasets/comments/bymru3/are_there_any_data_formats_for_storing_text_worth/
------------- Weekly Update ------------- This week, the development was bit slower compared to the last couple of weeks - I had to attend college for a couple of days and it took more time than I would have liked. However, that said, everything from the last week's goal is achieved. ----------------------- What happened last week ----------------------- * Redesigned Axis - the structure that returns the values during column binary operations * Added binary operations for Axis [This was equivalent of binary operations on DataFrame] * Tested whether the Binary Operations worked fine on DataFrames * Fixed couple of tiny bugs here and there. * Added more ways to build index. * HashMap like implementation to check for duplicates in index. ------------------- Goals for this week ------------------- * Work on apply - That applies a function to values in a row/column or the entire DataFrame * Add some inbuilt operations [This will be operations like mean, median - operations that are essential] * Optimize parts of DataFrame. * Addition of helpers that will eventually trigger the development of groupBy ---------- Roadblocks ---------- There were a few moments where I ran into trouble while returning Axis structure because of the variable return type. My mentors helped me a lot when I got stuck with the implementation. There ware also a couple of bugs which were simple but still took a while to solve. Other than that, things went really smooth. --------------------- Community Suggestions --------------------- Petar Kirov [ZombineDev] suggested using static arrays as a way to declare a DataFrame. Soon it will be possible to declare DataFrame as: DataFrame!(int[5], double[10]) df; [Implemented in BinaryOps branch - Will land in master with BinaryOps implementation soon] Thank you for the suggestion.
Jun 17
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
 [snip]
------------- Week 4 Update ------------- This marks the completion of Stage I of Google Summer of Code 2019. It seems like it was only yesterday when I started working on this project and it has already been a month. -------------------------- So what happened last week -------------------------- * apply - to apply a function on a row/column * function to convert a column of data to level of Index * drop - to drop a row/column Going back to the original proposal, I had allocated some time for optimisations in case there was time: I was testing old parser with large files and it failed miserably. So I redesigned the from_csv function and added it to the library as fastCSV. fastCSV gives 40x speed improvement over from_csv and fastCSV will eventually replace from_csv * fastCSV was added to the library. ------------------- Plans for this week ------------------- Plans for this entire stage isn't strictly on a week by week timeline but the following things will be dealt sequentially throughout this stage: This stage is reserved for implementation of groupBy. So for the beginning the internal structure and grouping will be decided. Later things like display and combining into a DataFrame struct will be dealt with. These tasks were scheduled for Stage-III but will again fall under sequential implementation. If the above tasks are done. The following tasks will be dealt with: * Aggregate [with complete set of popular operations] * Join will be implemented to merge two DataFrame. Aggregate was reserved for later stages to support implementation for both normal DataFrame and groubBy at once. ---------- Roadblocks ---------- This week there hasn't been any roadblocks. I needed the help of my mentors to solve a couple errors here and there but other than that things were smooth. As for the future roadblocks, I cannot see any apparent ones but then again they show up when you are least expecting them :(
Jun 25
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 25 June 2019 at 17:25:34 UTC, Prateek Nayak wrote:
 [snip]
------------- Week 4 Update ------------- This marks the completion of Stage I of Google Summer of Code 2019. It seems like it was only yesterday when I started working on this project and it has already been a month. -------------------------- So what happened last week -------------------------- * apply - to apply a function on a row/column * function to convert a column of data to level of Index * drop - to drop a row/column
[snip] Glad to see you're still making great progress. I had worked on the byDim function in mir.ndslice.topology is byDim because I had wanted the same sort of functionality as R's apply. It works a little differently than R's, but I find it very good for a lot of things. Your version of apply (I'm looking at the apply branch of magpie) looks like it operates a bit like a byDim chained with an each, so byDim!N.each!f. However, it also has this index variable allowing it to skip rows or something (I'm not really sure if this feature pulls its weight...). So I have two questions: 1) does byDim also work with dataframes?, 2) can you add an overload that is apply(f, axis) without the index parameter? One of my take-a-ways from looking at the apply function (again just looking at that apply branch) is that you might benefit from using more of what is Ilya has already put in mir.ndslice where available. For instance, the overload of apply that is just apply!f is basically the same as mir's each, but each has more features.
Jun 25
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 25 June 2019 at 17:54:36 UTC, jmh530 wrote:
 On Tuesday, 25 June 2019 at 17:25:34 UTC, Prateek Nayak wrote:
 [snip]
------------- Week 4 Update ------------- This marks the completion of Stage I of Google Summer of Code 2019. It seems like it was only yesterday when I started working on this project and it has already been a month. -------------------------- So what happened last week -------------------------- * apply - to apply a function on a row/column * function to convert a column of data to level of Index * drop - to drop a row/column
[snip] Glad to see you're still making great progress. I had worked on the byDim function in mir.ndslice.topology is byDim because I had wanted the same sort of functionality as R's apply. It works a little differently than R's, but I find it very good for a lot of things. Your version of apply (I'm looking at the apply branch of magpie) looks like it operates a bit like a byDim chained with an each, so byDim!N.each!f. However, it also has this index variable allowing it to skip rows or something (I'm not really sure if this feature pulls its weight...). So I have two questions: 1) does byDim also work with dataframes?, 2) can you add an overload that is apply(f, axis) without the index parameter? One of my take-a-ways from looking at the apply function (again just looking at that apply branch) is that you might benefit from using more of what is Ilya has already put in mir.ndslice where available. For instance, the overload of apply that is just apply!f is basically the same as mir's each, but each has more features.
Stupid typos. I had worked on the byDim function in mir.ndslice.topology because ..."
Jun 25
prev sibling parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Tuesday, 25 June 2019 at 17:54:36 UTC, jmh530 wrote:
 Glad to see you're still making great progress.

 I had worked on the byDim function in mir.ndslice.topology is 
 byDim because I had wanted the same sort of functionality as 
 R's apply. It works a little differently than R's, but I find 
 it very good for a lot of things. Your version of apply (I'm 
 looking at the apply branch of magpie) looks like it operates a 
 bit like a byDim chained with an each, so byDim!N.each!f. 
 However, it also has this index variable allowing it to skip 
 rows or something (I'm not really sure if this feature pulls 
 its weight...).

 So I have two questions: 1) does byDim also work with 
 dataframes?, 2) can you add an overload that is apply(f, axis) 
 without the index parameter?

 One of my take-a-ways from looking at the apply function (again 
 just looking at that apply branch) is that you might benefit 
 from using more of what is Ilya has already put in mir.ndslice 
 where available. For instance, the overload of apply that is 
 just apply!f is basically the same as mir's each, but each has 
 more features.
1) Current, byDim doesn't work on DataFrame DataFrame. 2) Sure, the overload can be made but what are you specifically looking for? apply(f, axis)(indexes) ? You are right, apply works like byDim!axis.each on particular columns/rows. I'll look into Mir's implementation. Thanks for that advice. I do believe apply can be strengthened to account for different use cases. When the heterogeneous DataFrame support came, mir-algorithms was dropped from dependencies and Structure of Array implementation was taken up using TypeTuples. Once the basic working is solid, I'll port useful features from Mir to Magpie.
Jun 25
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 25 June 2019 at 20:44:43 UTC, Prateek Nayak wrote:
 [snip]
 2) Sure, the overload can be made but what are you specifically 
 looking for?
 apply(f, axis)(indexes) ?
 [snip]
I see void apply(alias Fn, int axis, T)(T index) and void apply(alias Fn)() in the current implementation. I think you interpreted what I am asking as something like void apply(alias Fn, int axis, T[])(T[] indices) which also might make sense. But I guess I was suggesting a little simpler as void apply(alias Fn, int axis)() so that it applies to all the rows or columns. This is particularly relevant in the homogeneous data case. My motivation reflects a common use case of the apply function in R to calculate summary statistics of an array/matrix by column or row. For instance, I might want to calculate the standard deviation of every column.
Jun 25
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Tuesday, 25 June 2019 at 21:07:35 UTC, jmh530 wrote:
 On Tuesday, 25 June 2019 at 20:44:43 UTC, Prateek Nayak wrote:
 [snip]
 2) Sure, the overload can be made but what are you 
 specifically looking for?
 apply(f, axis)(indexes) ?
 [snip]
I see void apply(alias Fn, int axis, T)(T index) and void apply(alias Fn)() in the current implementation. I think you interpreted what I am asking as something like void apply(alias Fn, int axis, T[])(T[] indices) which also might make sense. But I guess I was suggesting a little simpler as void apply(alias Fn, int axis)() so that it applies to all the rows or columns. This is particularly relevant in the homogeneous data case. My motivation reflects a common use case of the apply function in R to calculate summary statistics of an array/matrix by column or row. For instance, I might want to calculate the standard deviation of every column.
The apply right now works exactly as void apply(alias Fn, int axis, T)(T indices) indices can be an array of integer or a 2D array of string index https://github.com/Kriyszig/magpie/blob/dec86d1942f9c9b4db31438407798329af0aed96/source/magpie/dataframe.d#L1200 The overload you need also exists: apply(Fn) https://github.com/Kriyszig/magpie/blob/dec86d1942f9c9b4db31438407798329af0aed96/source/magpie/dataframe.d#L1246 Unittest for apply - https://github.com/Kriyszig/magpie/blob/dec86d1942f9c9b4db31438407798329af0aed96/source/magpie/dataframe.d#L2821 I agree, things like mean and standard deviation calculations are of utmost importance in data science. Aggregate will bring such features as inbuilt functions. Count, Min, Max, Mean, SD, Variance, etc. This will be added soon (by soon I mean somewhere between the final week of this stage [possibly sooner] and the fist week of the next - As soon as groupBy is stable, I will get onto aggregate) Sorry for the inconvenience.
Jun 25
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 26 June 2019 at 05:41:48 UTC, Prateek Nayak wrote:
 [snip]

 I agree, things like mean and standard deviation calculations 
 are of utmost importance in data science. Aggregate will bring 
 such features as inbuilt functions. Count, Min, Max, Mean, SD, 
 Variance, etc.
 This will be added soon (by soon I mean somewhere between the 
 final week of this stage [possibly sooner] and the fist week of 
 the next - As soon as groupBy is stable, I will get onto 
 aggregate)
 Sorry for the inconvenience.
By no means do you need to apologize for any inconvenience. I suppose what I am thinking is more about leveraging work that is already done as much as possible. For instance, I know that count/sum/min/max are part of mir-algorithm already and I had helped add sd and variance to numir. Do you mind if I send you an email?
Jun 26
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Wednesday, 26 June 2019 at 12:50:23 UTC, jmh530 wrote:
 On Wednesday, 26 June 2019 at 05:41:48 UTC, Prateek Nayak wrote:
 [snip]

 I agree, things like mean and standard deviation calculations 
 are of utmost importance in data science. Aggregate will bring 
 such features as inbuilt functions. Count, Min, Max, Mean, SD, 
 Variance, etc.
 This will be added soon (by soon I mean somewhere between the 
 final week of this stage [possibly sooner] and the fist week 
 of the next - As soon as groupBy is stable, I will get onto 
 aggregate)
 Sorry for the inconvenience.
By no means do you need to apologize for any inconvenience. I suppose what I am thinking is more about leveraging work that is already done as much as possible. For instance, I know that count/sum/min/max are part of mir-algorithm already and I had helped add sd and variance to numir. Do you mind if I send you an email?
I'm sorry I couldn't reply sooner - I was sick for past couple of days. I don't mind emails one bit. The email id is: lelouch.cpp gmail.com [I know its weird :)] I'll reply as soon as I see the mail [At worse it will take 12hrs to get a reply for me when my phone doesn't notify me of a new mail. At best I'll reply immediately] Again, sorry for the delayed response. Hope to hear from you soon :)
Jun 28
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Saturday, 29 June 2019 at 04:39:39 UTC, Prateek Nayak wrote:
 [snip]
------------- Weekly Update ------------- I caught a flu this week which was really unfortunate. However I'm getting better and the work is going forward :) ----------------------- What happened last week ----------------------- I mostly dealt with the internal structure of `Group` - the structure that is returned during groupBy operation. First I thought an array of DataFrames might be a good idea but soon dropped the idea as it would mean some parts - like the column index remain same but need to be copied to every DataFrame structure in the array and its just a waste of space at that point. The implementation now looks somewhat similar to the DataFrame structure itself - there is an `Index` and `data`. Indexes are sorted based on the groups formed as a result of groupBy. There are few places where optimizations can be made [mostly wrt space used] and I'll work on it this week. Some of the functionality added to `Group` so far: * display - User can choose to display a singe group or multiple groups * combine - returns a DataFrame combining the groups user would like At this point there was a need fora function in DataFrame which could convert a level of indexing to a column of operable data if required. This is because combine on groupBy doesn't remember the position from where the data was extracted. Hence if a level of data is used for groupBy, it would automatically be converted to a level of index in the result of combine. Hence `indexToData` was added to revert the result of this if the user desires so. There were a few minor updates here and there. Nothing major. They include a new argument for `extends` in `Index` which can now insert the index at the position of user's choice. The other was stripping of trailing white spaces which appeared in display. -------------------------- What will happen this week -------------------------- This week will deal with optimizations of Group, add binary operations to `Group` which may be helpful. Document the changes once stability is reached. Start work on aggregate/join. ---------- Roadblocks ---------- I can't spot any major roadblocks up ahead. Work should go smoothly this week :) -> Thank you jmh530 for sharing your work. This should help in improving the functionality of DataFrames further.
Jul 02
parent reply Prateeek Nayak <lelouch.cpp gmail.com> writes:
On Wednesday, 3 July 2019 at 05:04:20 UTC, Prateek Nayak wrote:
 On Saturday, 29 June 2019 at 04:39:39 UTC, Prateek Nayak wrote:
 [snip]
[snip]
--------------- Progress Update --------------- The past couple of weeks went as expected without any Roadblocks * groupBy can group a DataFrame based on arbitrary number of columns * groupBy returns a Group structure which supports binary operations * Retrieve single/multiple group as DataFrame. * Merge two/more Group into a single DataFrame. * Index operations on DataFrame. An entire column/row is returned as Axis the same way Index operation on DataFrame is implemented. * Display Group on Terminal Works on DataFrame * Added short hand data operations which I missed before! \(°^°)/ * Added function to convert index to an operable data column and vice versa --------------------- What is due this week --------------------- This week was mostly reserved for refactor. Mr. Wilson introduced me to the beautiful lockstep in range and I worked it in the codebase wherever it's necessary. This week I as adding ways to retrieve data as a Slice and assign Slice to DataFrame. This IMHO is important as ndslice is used widely and it opens a lot of doors for data computations. A way to easily retrieve data as Slice operate on it and assign the data back to DataFrame sees valuable. I hope to get the initial PR ready by the beginning of next week. After this will bring Aggregate - on whole Frame/Group on a selected column/row of DataFrame or Group, selective operation on selective columns/rows. ----------------- Future Roadblocks ----------------- I can't see any obvious roadblocks but then you never do see them coming ¯\_(ツ)_/¯
Jul 17
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 18 July 2019 at 05:03:38 UTC, Prateeek Nayak wrote:
 [snip]
Thanks for the update. I'm glad you're still making good progress. I'm just looking over the readme.md. I noticed the "at" function has a signature like at!(row, column)(). Because it uses a template, doesn't that imply that the row and column parameters must be known at compile-time? What if we want run-time access using a function style instead of like df[0, 0]? mir's ndslice also has a set of select functions that are also useful for access. There's also a typo in the GroupBy text: "Group DataFrame based on na arbitrary number of columns." I noticed that you make a lot of use of static foreach's over RowType in dataframe.d. Does that this means that this means there isn't any extra cost if you use a homogeneous dataframe with RowType.length == 1? If you can advertise that it doesn't have any additional overhead for working with homogeneous, then that's probably a win. You might also add a trait for isHomogeneous that checks if RowType.length == 1.
Jul 18
parent reply Prateeek Nayak <lelouch.cpp gmail.com> writes:
On Thursday, 18 July 2019 at 10:55:55 UTC, jmh530 wrote:
 On Thursday, 18 July 2019 at 05:03:38 UTC, Prateeek Nayak wrote:
 [snip]
Thanks for the update. I'm glad you're still making good progress. I'm just looking over the readme.md. I noticed the "at" function has a signature like at!(row, column)(). Because it uses a template, doesn't that imply that the row and column parameters must be known at compile-time? What if we want run-time access using a function style instead of like df[0, 0]? mir's ndslice also has a set of select functions that are also useful for access. There's also a typo in the GroupBy text: "Group DataFrame based on na arbitrary number of columns." I noticed that you make a lot of use of static foreach's over RowType in dataframe.d. Does that this means that this means there isn't any extra cost if you use a homogeneous dataframe with RowType.length == 1? If you can advertise that it doesn't have any additional overhead for working with homogeneous, then that's probably a win. You might also add a trait for isHomogeneous that checks if RowType.length == 1.
* "at" was for a fast access to element. It's only necessary to know one of the two argument at compile time to be honest but df[i1, i2] has to be written as at!(i2)(i1) which reverses the two position hence I thought at!(i1, i2) could reduce some mishap that position reversal can cause. I agree a method to access the element at runtime. I will overload at for that. * Sorry about the typo, will fix it soon (^_^) * The data in DataFrame is stored as TypeTuple which requires the column index to be known statically. When trying to do a runtime operation on data, I was forced to traverse the tuple statically to find the particular index. Homogeneous DataFrame defined as DataFrame!(int, 5) will give RowType as (int, int, int, int, int). For now that overhead still exists but I think isHomogeneous template can open some new door for optimization. I will definitely look into this over the next week. Thanks for bringing it to my notice.
Jul 18
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 18 July 2019 at 15:34:32 UTC, Prateeek Nayak wrote:
 [snip]

 * The data in DataFrame is stored as TypeTuple which requires 
 the column index to be known statically. When trying to do a 
 runtime operation on data, I was forced to traverse the tuple 
 statically to find the particular index. Homogeneous DataFrame 
 defined as DataFrame!(int, 5) will give RowType as (int, int, 
 int, int, int).
 For now that overhead still exists but I think isHomogeneous 
 template can open some new door for optimization. I will 
 definitely look into this over the next week. Thanks for 
 bringing it to my notice.
Ah, so what you would want to check is that all the RowTypes are the same instead.
Jul 18
parent Prateeek Nayak <lelouch.cpp gmail.com> writes:
On Thursday, 18 July 2019 at 16:23:20 UTC, jmh530 wrote:
 On Thursday, 18 July 2019 at 15:34:32 UTC, Prateeek Nayak wrote:
 [snip]
Ah, so what you would want to check is that all the RowTypes are the same instead.
Yes. Will require a small redesign in the internal structure and some optimizations here and there but can seriously cut down overheads.
Jul 18
prev sibling next sibling parent reply James Blachly <james.blachly gmail.com> writes:
On 5/29/19 2:00 PM, Prateek Nayak wrote:
 (snip)
Outstanding, and greatly needed. Congratulations to you and your mentors. Our lab has transitioned to D for new software but still relies on python+pandas for some analytics pipelines. I second the notions about importance of interoperability. An interesting in-memory interop framework I haven't seen mentioned here yet is Apache Arrow. In this 2017 blog post, Wes McKinney, the author of Pandas, discusses it in the context of mistakes made designing pandas; recommended reading if you have not: https://wesmckinney.com/blog/apache-arrow-pandas-internals/
May 29
parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 30 May 2019 at 02:16:16 UTC, James Blachly wrote:
 [snip]

 https://wesmckinney.com/blog/apache-arrow-pandas-internals/
This was a good read. The columnar data structures he describes sound more like struct of arrays than array of structs.
May 30
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
Looking at the readme, I see the following example for accessing 
elements by name:

df[["1"], ["0"]];

Why can't that instead be

df["1", "0"];

Something that gets in the way of adoption is verbose notation, 
and I'm not seeing any advantage to the array notation.

Also, for this example:

Index indx;
indx.setIndex([1, 2, 3, 4], ["Row Index"], [1, 2, 3], ["Column 
Index"]);

That's pretty verbose/hard to parse compared to

rownames(x) = [1, 2, 3, 4];
colnames(x) = [1, 2, 3];
Jul 18
parent Prateek Nayak <lelouch.cpp gmail.com> writes:
On Thursday, 18 July 2019 at 16:02:40 UTC, bachmeier wrote:
 Looking at the readme, I see the following example for 
 accessing elements by name:

 df[["1"], ["0"]];

 Why can't that instead be

 df["1", "0"];

 Something that gets in the way of adoption is verbose notation, 
 and I'm not seeing any advantage to the array notation.

 Also, for this example:

 Index indx;
 indx.setIndex([1, 2, 3, 4], ["Row Index"], [1, 2, 3], ["Column 
 Index"]);

 That's pretty verbose/hard to parse compared to

 rownames(x) = [1, 2, 3, 4];
 colnames(x) = [1, 2, 3];
I'm really sorry I overlooked this. Sorry about that (-‸ლ) I'll fix the first case in the PR where I'll make optimizations for homogeneous DataFrame. I'll address the second problem of verbosity soon but not in the immediate PR Thanks for the feedback ٩(^‿^)۶
Jul 23
prev sibling parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
On Wednesday, 29 May 2019 at 18:00:02 UTC, Prateek Nayak wrote:
 Hello everyone,

 I have began work on my Google Summer of Code 2019 project 
 DataFrame for D.
Really glad to see someone working on that. I hope you will have time to implement a good CSV/TSV reader/writer based on the fantastic iopipe project (that should IMHO go into Phobos this way or another)...
Jul 19
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Friday, 19 July 2019 at 14:50:35 UTC, Dejan Lekic wrote:
 On Wednesday, 29 May 2019 at 18:00:02 UTC, Prateek Nayak wrote:
 Hello everyone,

 I have began work on my Google Summer of Code 2019 project 
 DataFrame for D.
Really glad to see someone working on that. I hope you will have time to implement a good CSV/TSV reader/writer based on the fantastic iopipe project (that should IMHO go into Phobos this way or another)...
Right now there is a CSV reader in Magpie but it isn't perfect enough to go into Phobos yet. I'll improve the parser and when I'm happy with the read speed, I'll send a PR (^_^)
Jul 23
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 23 July 2019 at 18:21:30 UTC, Prateek Nayak wrote:
 [snip]

 Right now there is a CSV reader in Magpie but it isn't perfect 
 enough to go into Phobos yet. I'll improve the parser and when 
 I'm happy with the read speed, I'll send a PR (^_^)
mir was originally intended to be included in Phobos, but got split off to its own library. If anything, Magpie has a better place in mir than Phobos. However, I think there is probably value in splitting off the csv reader to a separate project and just putting that up on dub when it is ready for broader use.
Jul 23
prev sibling parent reply Suliman <evermind live.ru> writes:
Could you do any benchmarks against Python Pandas.
Jul 25
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Thursday, 25 July 2019 at 11:55:48 UTC, Suliman wrote:
 Could you do any benchmarks against Python Pandas.
As soon as the aggregate is done, I'll get onto this. It's best to benchmark with real world examples IMHO and aggregate brings most of the analytics functionality. I'll keep the thread updated (^_^)
Jul 25
parent reply Prateeek Nayak <lelouch.cpp gmail.com> writes:
-----------
Update Time
-----------

Pardon me for the delay, my university just started and it has 
been a busy first week. However I have some good news

* Aggregate implementation is under review - The preliminary 
implementation restricted the set of operations that aggregate 
could do but then Mr. Wilson suggested there should be a way to 
expand it's usability so we worked on a revamp which takes the 
function you desire as input and operates them on row/column of 
DataFrame
* There is a new way set index using index operation
* to_csv supports setting precision for floating point numbers - 
this was a problem I knew existed but I hadn't addressed it till 
now. Better late then never.
* Homogeneous DataFrame don't use TypeTuple anymore
* at overload coming soon


--------------------
What is to come next
--------------------

* The first few responses from the community were mostly 
regarding bringing binary file I/O support because of their lean 
size and fast read/write. I will explore more regarding this.
* Time Series is gaining importance with the rise of Machine 
Learning. I would like to implement something along the lines of 
time series functionality Pandas has.
* Something you would line to see. I am open to suggestions (^_^)

--------------
Problems faced
--------------

There remains a small implementation detail that remains - a 
dispatch function. Given non-homogeneous cases still require 
traversal to a column, a function to apply an alias statically or 
non-statically depending on the DataFrame is under discussion.
This will reduce code redundancy however my preliminary attempts 
to tackle this have ended in failure. I will try to finish it by 
the weekend. If I cannot solve it by then, I will seek your help 
in the Learn section (^_^)
Thank you
Aug 08
next sibling parent reply bioinfornatics <bioinfornatics fedoraproject.org> writes:
On Thursday, 8 August 2019 at 16:49:09 UTC, Prateeek Nayak wrote:
 -----------
 Update Time
 -----------

 Pardon me for the delay, my university just started and it has 
 been a busy first week. However I have some good news

 * Aggregate implementation is under review - The preliminary 
 implementation restricted the set of operations that aggregate 
 could do but then Mr. Wilson suggested there should be a way to 
 expand it's usability so we worked on a revamp which takes the 
 function you desire as input and operates them on row/column of 
 DataFrame
 * There is a new way set index using index operation
 * to_csv supports setting precision for floating point numbers 
 - this was a problem I knew existed but I hadn't addressed it 
 till now. Better late then never.
 * Homogeneous DataFrame don't use TypeTuple anymore
 * at overload coming soon


 --------------------
 What is to come next
 --------------------

 * The first few responses from the community were mostly 
 regarding bringing binary file I/O support because of their 
 lean size and fast read/write. I will explore more regarding 
 this.
 * Time Series is gaining importance with the rise of Machine 
 Learning. I would like to implement something along the lines 
 of time series functionality Pandas has.
 * Something you would line to see. I am open to suggestions 
 (^_^)

 --------------
 Problems faced
 --------------

 There remains a small implementation detail that remains - a 
 dispatch function. Given non-homogeneous cases still require 
 traversal to a column, a function to apply an alias statically 
 or non-statically depending on the DataFrame is under 
 discussion.
 This will reduce code redundancy however my preliminary 
 attempts to tackle this have ended in failure. I will try to 
 finish it by the weekend. If I cannot solve it by then, I will 
 seek your help in the Learn section (^_^)
 Thank you
Dear D community, Thanks, Prateeek Nayak for your works. As currently, I am working with pandas (python, dataframe ...) . They are an extra feature that I appreciate a lot, it is the IO tool part: * SQL method: read_sql and to_sql Description: which allow to read and save from a DataBase. These methods combined with SqlAlchemy are awesome. * Parquet method: read_parquet and to_parquet Description: In BigData environment Parquet is a file format often used These abilities made Panda and its Dataframe API a core library to have. Using like this, allow standardizing data structured used into our application and in same time offer rich statistics API. Indeed it is important for tho code maintainability. And the FairData point that an application is a set of input data + program's feature = result. Thus put data structured as the first component to think how to develop an application is important. The application is more robust and flexible as we can handle multiple input data file format. I hope to see such features in D. Best regards Source: - https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql.html - https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html - https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#parquet
Aug 09
parent reply Prateek Nayak <lelouch.cpp gmail.com> writes:
On Friday, 9 August 2019 at 08:08:39 UTC, bioinfornatics wrote:
 Dear D community,

 Thanks, Prateeek Nayak for your works.
 As currently, I am working with pandas (python, dataframe ...) 
 . They are an extra feature that I appreciate a lot, it is the 
 IO tool part:

 * SQL
 method: read_sql and to_sql
 Description: which allow to read and save from a DataBase. 
 These methods combined with SqlAlchemy are awesome.

 * Parquet
 method: read_parquet and to_parquet
 Description: In BigData environment Parquet is a file format 
 often used

 These abilities made Panda and its Dataframe API a core library 
 to have. Using like this, allow standardizing data structured 
 used into our application and in same time offer rich 
 statistics API.

 Indeed it is important for tho code maintainability. And the 
 FairData point that an application is a set of input data + 
 program's feature = result. Thus put data structured as the 
 first component to think how to develop an application is 
 important.
 The application is more robust and flexible as we can handle 
 multiple input data file format.

 I hope to see such features in D.


 Best regards

 Source:
 - 
 https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_sql.html
 - 
 https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html
 - 
 https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#parquet
I was looking into Parquet and it even came up in the reddit post i had linked to earlier on - smaller file size and better I/O makes it really good for industrial use. A quick search on DUB didn't give any result for a parser so I'll probably work on a library to work with Parquet files. I looked into Cap'n Proto too - it looks promising but its missing from Pandas I/O section which was disappointing. Thanks for mentioning SQL. I will start working on these features soon.
 Again, thank you so much for working on this!
 We will be excited to put Magpie through its paces in our lab, 
 but it is missing* a few key (really, basic IMO) features we 
 make heavy use of in pandas.
 * I have read the README and glanced at code but not used 
 Magpie yet, so if I am > wrong about below please correct me!
 Since you are soliciting ideas:
 1. Selecting/indexing into data with boolean vectors. e.g:
 df[df.A > 30 && df.B != "ignore"]
 1a. This really means returning a boolean vector for df.COL 
 <op> <operand>
 1b. ...and being able to subset data by a bool vector
 2. We make heavy use of "pivot" functionality.
 Kind regards
I was thinking of the same feature as 1 - a filter like function for DataFrame and Group - finding possible ways to implement it I'm really embarrassed to admit I never even thought about Pivot. I looks like a beautiful feature to have - will definitely add to Magpie soon (possibly over the next couple of weeks - I'm a bit tied down right now with commencement of University academics but it will definitely come soon)
Aug 09
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 10 August 2019 at 04:10:31 UTC, Prateek Nayak wrote:
 I was looking into Parquet and it even came up in the reddit 
 post i had linked to earlier on - smaller file size and better 
 I/O makes it really good for industrial use.
 A quick search on DUB didn't give any result for a parser so 
 I'll probably work on a library to work with Parquet files.
 I looked into Cap'n Proto too - it looks promising but its 
 missing from Pandas I/O section which was disappointing.
 Thanks for mentioning SQL. I will start working on these 
 features soon.
It's clearly important that your project supports the same data exchange formats as pandas, but it doesn't seem inherently a problem to support other formats as well, assuming you have the time and inclination to do so.
Aug 10
parent Prateek Nayak <lelouch.cpp gmail.com> writes:
On Saturday, 10 August 2019 at 12:38:19 UTC, Joseph Rushton 
Wakeling wrote:
 It's clearly important that your project supports the same data 
 exchange formats as pandas, but it doesn't seem inherently a 
 problem to support other formats as well, assuming you have the 
 time and inclination to do so.
It is never an inherent problem to support a new file format but the initial comments from the community was mainly regarding easy interop with python. That is why I was thinking of Parquet support. Cap'n Proto is great and I'll love to implement a Cap'n Proto I/O sooner or later but Parquet seems to have a heavier presence due to popularity of Pandas so I decided to look into it first.
Aug 10
prev sibling parent James Blachly <james.blachly gmail.com> writes:
On 8/8/19 12:49 PM, Prateeek Nayak wrote:
 --------------------
 What is to come next
 --------------------
 
 * The first few responses from the community were mostly regarding 
 bringing binary file I/O support because of their lean size and fast 
 read/write. I will explore more regarding this.
 * Time Series is gaining importance with the rise of Machine Learning. I 
 would like to implement something along the lines of time series 
 functionality Pandas has.
 * Something you would line to see. I am open to suggestions (^_^)
Again, thank you so much for working on this! We will be excited to put Magpie through its paces in our lab, but it is missing* a few key (really, basic IMO) features we make heavy use of in pandas. * I have read the README and glanced at code but not used Magpie yet, so if I am wrong about below please correct me! Since you are soliciting ideas: 1. Selecting/indexing into data with boolean vectors. e.g: df[df.A > 30 && df.B != "ignore"] 1a. This really means returning a boolean vector for df.COL <op> <operand> 1b. ...and being able to subset data by a bool vector 2. We make heavy use of "pivot" functionality. Kind regards
Aug 09