www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [GSOC] Database API

reply Christian Manning <cmanning999 gmail.com> writes:
Hi,
I'm a second year student at De Montfort University studying Computer 
Science. I am very much interested in working on the database API idea 
that is proposed at 
http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DatabaseAPI (I was 
also quite interested in the containers idea, but it looks like someone 
else wants to do that)

I have limited experience with database libraries but have begun to 
study the JDBC API (in my small amount of spare time only as term 
doesn't finish for another 2 weeks for me and I still have assignments).

Has this idea/project been assigned a mentor? I'd like to ask them and 
the list, what's the best thing for me to do right now to prepare for this?

Thanks
Chris
Mar 24 2011
next sibling parent reply Trass3r <un known.com> writes:
 Has this idea/project been assigned a mentor? I'd like to ask them and  
 the list, what's the best thing for me to do right now to prepare for  
 this?


You could also have a look at http://dsource.org/projects/ddbi
This shows some past efforts to create database interfaces.
Mar 24 2011
parent reply Christian Manning <cmanning999 gmail.com> writes:
On 24/03/2011 17:58, Trass3r wrote:

 Has this idea/project been assigned a mentor? I'd like to ask them and
 the list, what's the best thing for me to do right now to prepare for
 this?


 You could also have a look at http://dsource.org/projects/ddbi
 This shows some past efforts to create database interfaces.


This can be of some inspiration and also brings up a point about how 
this project would be executed, ie. is this to be part of phobos or a 
totally separate project? I'm guessing to be Digital Mars supported it's 
more likely wanted to be a part of phobos, but could someone clarify on 
this?

Another question about Digital Mars' GSOC involvement: when it's 
application time do I submit a proposal to someone directly, to this 
list, or to the GSOC site at http://www.google-melange.com ?

Thanks
Chris
Mar 25 2011
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
 On 24/03/2011 17:58, Trass3r wrote:

 Has this idea/project been assigned a mentor? I'd like to ask them and
 the list, what's the best thing for me to do right now to prepare for
 this?


 
 You could also have a look at http://dsource.org/projects/ddbi
 This shows some past efforts to create database interfaces.


 
 This can be of some inspiration and also brings up a point about how
 this project would be executed, ie. is this to be part of phobos or a
 totally separate project? I'm guessing to be Digital Mars supported it's
 more likely wanted to be a part of phobos, but could someone clarify on
 this?


What I would expect to be the case would be that it would be implemented with 
the idea that it would ideally be put into Phobos but that it wouldn't 
necessarily make it into Phobos. Any major new functionality that gets added 
to Phobos at this point needs to be reviewed by the folks on this newsgroup 
and get voted in. So, it's possible that a fairly solid solution could be 
created which would not be acceptable in Phobos for one reason or another, at 
which point it would have to either be fixed so that it would be acceptable in 
Phobos or be left as a 3rd party library.

So, ideally stuff like a good database solution for D would make it into 
Phobos, but whether it actually makes it in depends a great deal on the actual 
solution.

- Jonathan M Davis
Mar 25 2011
parent Christian Manning <cmanning999 gmail.com> writes:
On 25/03/2011 18:45, Jonathan M Davis wrote:

 On 24/03/2011 17:58, Trass3r wrote:

 Has this idea/project been assigned a mentor? I'd like to ask them and
 the list, what's the best thing for me to do right now to prepare for
 this?


 You could also have a look at http://dsource.org/projects/ddbi
 This shows some past efforts to create database interfaces.


 This can be of some inspiration and also brings up a point about how
 this project would be executed, ie. is this to be part of phobos or a
 totally separate project? I'm guessing to be Digital Mars supported it's
 more likely wanted to be a part of phobos, but could someone clarify on
 this?


 What I would expect to be the case would be that it would be implemented with
 the idea that it would ideally be put into Phobos but that it wouldn't
 necessarily make it into Phobos. Any major new functionality that gets added
 to Phobos at this point needs to be reviewed by the folks on this newsgroup
 and get voted in. So, it's possible that a fairly solid solution could be
 created which would not be acceptable in Phobos for one reason or another, at
 which point it would have to either be fixed so that it would be acceptable in
 Phobos or be left as a 3rd party library.

 So, ideally stuff like a good database solution for D would make it into
 Phobos, but whether it actually makes it in depends a great deal on the actual
 solution.

 - Jonathan M Davis


Understood.

My last deadline is Monday, so after that I should have much more time 
to concentrate on this to plan out what I'd like to do and submit a 
proper proposal.

Chris
Mar 25 2011
prev sibling next sibling parent reply Trass3r <un known.com> writes:
 Another question about Digital Mars' GSOC involvement: when it's  
 application time do I submit a proposal to someone directly, to this  
 list, or to the GSOC site at http://www.google-melange.com ?


You always have to apply via the google site.
I'm sure I read that somewhere in the FAQs.
Mar 25 2011
parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 26/03/2011 00:04, Trass3r wrote:

 Another question about Digital Mars' GSOC involvement: when it's
 application time do I submit a proposal to someone directly, to this
 list, or to the GSOC site at http://www.google-melange.com ?


 You always have to apply via the google site.
 I'm sure I read that somewhere in the FAQs.


Yeah, exactly. Technically you don't strictly need to contact the 
organization before hand to submit the proposal, nor does the proposal 
have to be related to the ideas presented by the organization, the 
student can bring forth his own ideas.
But later the organization will rank each student proposal to eventually 
decide which ones get accepted, so of course it helps that the student 
has discussed the proposal before hand.

-- 
Bruno Medeiros - Software Engineer
Mar 26 2011
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 03/25/2011 09:02 AM, Christian Manning wrote:

 On 24/03/2011 17:58, Trass3r wrote:

 Has this idea/project been assigned a mentor? I'd like to ask them and
 the list, what's the best thing for me to do right now to prepare for
 this?


 You could also have a look at http://dsource.org/projects/ddbi
 This shows some past efforts to create database interfaces.


 This can be of some inspiration and also brings up a point about how
 this project would be executed, ie. is this to be part of phobos or a
 totally separate project? I'm guessing to be Digital Mars supported it's
 more likely wanted to be a part of phobos, but could someone clarify on
 this?


Either way would be acceptable.


 Another question about Digital Mars' GSOC involvement: when it's
 application time do I submit a proposal to someone directly, to this
 list, or to the GSOC site at http://www.google-melange.com ?


You'll submit via Google's interface. See 
http://d-programming-language.org/gsoc2011.html for details. Submission 
opens tomorrow.


Andrei
Mar 27 2011
prev sibling next sibling parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Christian Manning wrote:

 Hi,
 I'm a second year student at De Montfort University studying Computer
 Science. I am very much interested in working on the database API idea
 that is proposed at
 http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DatabaseAPI (I was
 also quite interested in the containers idea, but it looks like someone
 else wants to do that)

 I have limited experience with database libraries but have begun to
 study the JDBC API (in my small amount of spare time only as term
 doesn't finish for another 2 weeks for me and I still have assignments).

 Has this idea/project been assigned a mentor? I'd like to ask them and
 the list, what's the best thing for me to do right now to prepare for this?

 Thanks
 Chris


I'm at finishing stage of my PostgreSQL client, written entirely in D. I 
will post sources in few days.

Before adding database support to Phobos, there are few important issues 
which obviously need further discussion. One of this issues are handling 
NULL values - essential thing in relational databases.

Although, there were some efforts to introduce standard Nullable or 
Optional type, they were aborted.

Currently I'm using this simple definition:

struct Nullable(T)
{
     Algebraic!(T, void*) Nullable;
     alias Nullable this;

     bool hasValue()
     {
         return cast(bool)Nullable.peek!(T);
     }

     bool isNull()
     {
         return cast(bool)!Nullable.peek!(T);
     }

     string toString()
     {
         T* t = Nullable.peek!T;
         return t ? to!string(*t) : "null";
     }
}

It flawlessy cooperates with Variant, which - I think - should 
officially support nulls as well. I would propose that standard Nullable 
type, should be a subtype of Algebraic union as above.

During writing my PostgreSQL interface, I also had some problems with 
Variant and std.conv.to functions. Currently std.conv doesn't support 
Variants, f.i. when using to!Variant(5), there are more than one toImpl 
match.

In case of DB api, Variants are also essential, because field types 
returned by database server aren't known at compile time.

To sum up, Phobos need some additional changes for database support, 
mainly addition of Nullable.
Mar 25 2011
parent reply Christian Manning <cmanning999 gmail.com> writes:
On 25/03/2011 14:56, Piotr Szturmaj wrote:

 To sum up, Phobos need some additional changes for database support,
 mainly addition of Nullable.


For now, couldn't this just be included with the database module?
Mar 25 2011
parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Christian Manning wrote:

 On 25/03/2011 14:56, Piotr Szturmaj wrote:

 To sum up, Phobos need some additional changes for database support,
 mainly addition of Nullable.


 For now, couldn't this just be included with the database module?


Yes. I think, currently this is the only option.
Mar 26 2011
prev sibling next sibling parent reply "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Fri, 25 Mar 2011 02:48:14 +0900, Christian Manning  
<cmanning999 gmail.com> wrote:


 Hi,
 I'm a second year student at De Montfort University studying Computer  
 Science. I am very much interested in working on the database API idea  
 that is proposed at  
 http://prowiki.org/wiki4d/wiki.cgi?GSOC_2011_Ideas#DatabaseAPI (I was  
 also quite interested in the containers idea, but it looks like someone  
 else wants to do that)

 I have limited experience with database libraries but have begun to  
 study the JDBC API (in my small amount of spare time only as term  
 doesn't finish for another 2 weeks for me and I still have assignments).

 Has this idea/project been assigned a mentor? I'd like to ask them and  
 the list, what's the best thing for me to do right now to prepare for  
 this?


Currently, many databases exist.

* SQL based: MySQL, PostgresSQL, SQLite, etc..
* KVS: Cassandra, HBase, Kumofs, Redis, etc...
* Document Oriented: MongoDB, CouchDB, etc...

I think next Database API should support these types.
AFAIK, DDBI supports only SQL based interface.

In Ruby, Rails 3 has Arel.
Arel seems to be a good query abstraction.

https://github.com/rails/arel

Arel's concept may help you :)


Masahiro
Mar 25 2011
next sibling parent Christian Manning <cmanning999 gmail.com> writes:
On 25/03/2011 23:44, Masahiro Nakagawa wrote:

 In Ruby, Rails 3 has Arel.
 Arel seems to be a good query abstraction.
 
 https://github.com/rails/arel
 
 Arel's concept may help you :)
 
 
 Masahiro


Now that's an interesting way of going about things. Although, its
description suggests that it's a framework for building frameworks which
isn't really what I'm after as the database API should be able to be
used by anybody easily. I don't know Ruby at all but in some of those
examples, the syntax is quite neat.

Chris
Mar 26 2011
prev sibling parent reply David Nadlinger <see klickverbot.at> writes:
On 3/26/11 12:44 AM, Masahiro Nakagawa wrote:

 Currently, many databases exist.

 * SQL based: MySQL, PostgresSQL, SQLite, etc..
 * KVS: Cassandra, HBase, Kumofs, Redis, etc...
 * Document Oriented: MongoDB, CouchDB, etc...

 I think next Database API should support these types.


I don't think this is the way to go, as there is a huge conceptional 
difference between relational database systems and »NoSQL« systems such 
as, say, CouchDB or Cassandra.

How would such a unified API look like?

David
Mar 26 2011
parent reply "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Sun, 27 Mar 2011 07:11:01 +0900, David Nadlinger <see klickverbot.at>=
  =

wrote:


 On 3/26/11 12:44 AM, Masahiro Nakagawa wrote:

 Currently, many databases exist.

 * SQL based: MySQL, PostgresSQL, SQLite, etc..
 * KVS: Cassandra, HBase, Kumofs, Redis, etc...
 * Document Oriented: MongoDB, CouchDB, etc...

 I think next Database API should support these types.


 I don't think this is the way to go, as there is a huge conceptional  =



 difference between relational database systems and =C2=BBNoSQL=C2=AB s=


ystems such  =


 as, say, CouchDB or Cassandra.


It's an open question.

Ruby / Rails has Arel (note that Arel is data access abstraction,
not database access abstraction. But Database access is a part of data  =

access).
Haskell has Database.Persistent.
I think other languages will support similar library in the future.


 How would such a unified API look like?


I don't have a detail.
I already implemented it if I had a detailed design.
In my D tasks, database related task is low priority...
Mar 28 2011
parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 29.03.2011 01:18, schrieb Masahiro Nakagawa:

 On Sun, 27 Mar 2011 07:11:01 +0900, David Nadlinger <see klickverbot.at>
 wrote:


 On 3/26/11 12:44 AM, Masahiro Nakagawa wrote:

 Currently, many databases exist.

 * SQL based: MySQL, PostgresSQL, SQLite, etc..
 * KVS: Cassandra, HBase, Kumofs, Redis, etc...
 * Document Oriented: MongoDB, CouchDB, etc...

 I think next Database API should support these types.


 I don't think this is the way to go, as there is a huge conceptional
 difference between relational database systems and »NoSQL« systems
 such as, say, CouchDB or Cassandra.


 It's an open question.

 Ruby / Rails has Arel (note that Arel is data access abstraction,
 not database access abstraction. But Database access is a part of data
 access).
 Haskell has Database.Persistent.
 I think other languages will support similar library in the future.


I think proper support for relational databases with SQL is more 
important right now.
Once it is done some abstraction for generic data access can be 
developed, using the relational database support for these databases and 
something else (probably more specific to each NoSQL-Database, because 
AFAIK they are even less compatible to each other than relational DBs) 
for NoSQL-DBs.

Cheers,
- Daniel
Mar 29 2011
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
Christian Manning Wrote:


 On 25/03/2011 14:56, Piotr Szturmaj wrote:

 To sum up, Phobos need some additional changes for database support,
 mainly addition of Nullable.


 
 For now, couldn't this just be included with the database module?


Why not just use Variant? It should be able to hold anything, so Nullable
should be not needed.
Mar 26 2011
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Kagamin wrote:

 Christian Manning Wrote:


 On 25/03/2011 14:56, Piotr Szturmaj wrote:

 To sum up, Phobos need some additional changes for database support,
 mainly addition of Nullable.


 For now, couldn't this just be included with the database module?


 Why not just use Variant? It should be able to hold anything, so Nullable
should be not needed.


I think it should, for the same reason. Many times you want more 
constrained data type than Variant. Say, some interface expect int or 
null. Nullable!int is perfect for it. But if you use Variant, you may 
end up having some struct inside or maybe some string, like 'abc' :)
In other words, base type of Nullable (for example int) is enforceable 
at compile time, unlike Variant.

Anyway my probosal is compatible with Variant. Variant is an alias for 
backend type VariantN. Quote from std.variant:


"VariantN is a back-end type seldom used directly by user code. Two 
commonly-used types using VariantN as back-end are:

Algebraic: A closed discriminated union with a limited type universe 
(e.g., Algebraic!(int, double, string) only accepts these three types 
and rejects anything else).
Variant: An open discriminated union allowing an unbounded set of types. 
The restriction is that the size of the stored type cannot be larger 
than the largest built-in type. This means that Variant can accommodate 
all primitive types and all user-defined types except for large structs.

Both Algebraic and Variant share VariantN's interface. (See their 
respective documentations below.)"


I propose that, Nullable should be an alias (or subtype) of Algebraic, 
for example:

template Nullable(T)
{
     alias Algebraic!(T, void*) Nullable;
}

This way it will share the same backend as Variant, so both types will 
be compatible.
Mar 26 2011
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/26/11 2:11 PM, Piotr Szturmaj wrote:

 I propose that, Nullable should be an alias (or subtype) of Algebraic,
 for example:

 template Nullable(T)
 {
 alias Algebraic!(T, void*) Nullable;
 }


Should be Algebraic!(T, void). If that doesn't currently compile, we 
should make it compile.

Andrei
Mar 26 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 26 Mar 2011 15:58:18 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 3/26/11 2:11 PM, Piotr Szturmaj wrote:

 I propose that, Nullable should be an alias (or subtype) of Algebraic,
 for example:

 template Nullable(T)
 {
 alias Algebraic!(T, void*) Nullable;
 }


 Should be Algebraic!(T, void). If that doesn't currently compile, we  
 should make it compile.

 Andrei


For what it's worth, both Algebraic!(T, void*) and Algebraic!(T, void)  
compile on my branch of std.variant.
Mar 26 2011
next sibling parent reply Kagamin <spam here.lot> writes:
Robert Jacques Wrote:


 Should be Algebraic!(T, void). If that doesn't currently compile, we  
 should make it compile.

 Andrei


 
 For what it's worth, both Algebraic!(T, void*) and Algebraic!(T, void)  
 compile on my branch of std.variant.


can't assign null

	alias Algebraic!(int,void) nint;
	nint n1=5;
	assert(n1==5);
	n1=null;

\std\variant.d:497: Error: static assert  "Cannot store a void* in a
VariantN!(maxSize,int,void). Valid types are (int, void)"
test.d:45:        instantiated from here: opAssign!(void*)
Mar 26 2011
next sibling parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Kagamin wrote:

 Robert Jacques Wrote:


 Should be Algebraic!(T, void). If that doesn't currently compile, we
 should make it compile.

 Andrei


 For what it's worth, both Algebraic!(T, void*) and Algebraic!(T, void)
 compile on my branch of std.variant.


 can't assign null

 	alias Algebraic!(int,void) nint;
 	nint n1=5;
 	assert(n1==5);
 	n1=null;

 \std\variant.d:497: Error: static assert  "Cannot store a void* in a
VariantN!(maxSize,int,void). Valid types are (int, void)"
 test.d:45:        instantiated from here: opAssign!(void*)


Yes, compiler qualifies null as void*.

I was playing with void* version few weeks ago and I can confirm that 
both assignment and equality testing works for nulls.
One thing I missed was lack of conversion to string "null". For example 
array of Algebraic!(int, void*) was printed as [0, 0, 2], when element 1 
was null. I think it should rather be [0, null, 2].
Mar 26 2011
prev sibling parent Kagamin <spam here.lot> writes:
Kagamin Wrote:


 Robert Jacques Wrote:
 

 Should be Algebraic!(T, void). If that doesn't currently compile, we  
 should make it compile.

 Andrei


 
 For what it's worth, both Algebraic!(T, void*) and Algebraic!(T, void)  
 compile on my branch of std.variant.


 
 can't assign null
 
 	alias Algebraic!(int,void) nint;
 	nint n1=5;
 	assert(n1==5);
 	n1=null;
 
 \std\variant.d:497: Error: static assert  "Cannot store a void* in a
VariantN!(maxSize,int,void). Valid types are (int, void)"
 test.d:45:        instantiated from here: opAssign!(void*)


Ahh, there's a workaround:

	alias Algebraic!(int,void) nint;
	nint n1=5;
	assert(n1.get!(int)==5);
	n1=nint(); //uninitialized Variant, sort of null
	assert(!n1.hasValue);
	assert(n1.peek!(int)==null);
	writeln("error here:");
	assert(n1.get!(int)==0);
Mar 26 2011
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/26/11 4:20 PM, Robert Jacques wrote:

 On Sat, 26 Mar 2011 15:58:18 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 On 3/26/11 2:11 PM, Piotr Szturmaj wrote:

 I propose that, Nullable should be an alias (or subtype) of Algebraic,
 for example:

 template Nullable(T)
 {
 alias Algebraic!(T, void*) Nullable;
 }


 Should be Algebraic!(T, void). If that doesn't currently compile, we
 should make it compile.

 Andrei


 For what it's worth, both Algebraic!(T, void*) and Algebraic!(T, void)
 compile on my branch of std.variant.


Great. I'd prefer the one using void because storing a void* can be a 
legitimate need.

Andrei
Mar 26 2011
prev sibling parent KennyTM~ <kennytm gmail.com> writes:
On Mar 27, 11 03:11, Piotr Szturmaj wrote:

 I propose that, Nullable should be an alias (or subtype) of Algebraic,
 for example:

 template Nullable(T)
 {
      alias Algebraic!(T, void*) Nullable;
 }

 This way it will share the same backend as Variant, so both types will
 be compatible.


template Nullable(T) if (!__traits(compiles, {T a=null;}) { ... }

I don't think it makes sense to Nullable something that can already take 
null.
Mar 26 2011
prev sibling parent reply Kagamin <spam here.lot> writes:
Kagamin Wrote:


 Kagamin Wrote:
 

 Robert Jacques Wrote:
 

 Should be Algebraic!(T, void). If that doesn't currently compile, we  
 should make it compile.

 Andrei


 
 For what it's worth, both Algebraic!(T, void*) and Algebraic!(T, void)  
 compile on my branch of std.variant.


 
 can't assign null
 
 	alias Algebraic!(int,void) nint;
 	nint n1=5;
 	assert(n1==5);
 	n1=null;
 
 \std\variant.d:497: Error: static assert  "Cannot store a void* in a
VariantN!(maxSize,int,void). Valid types are (int, void)"
 test.d:45:        instantiated from here: opAssign!(void*)


 
 Ahh, there's a workaround:
 
 	alias Algebraic!(int,void) nint;
 	nint n1=5;
 	assert(n1.get!(int)==5);
 	n1=nint(); //uninitialized Variant, sort of null
 	assert(!n1.hasValue);
 	assert(n1.peek!(int)==null);
 	writeln("error here:");
 	assert(n1.get!(int)==0);


this also works without allowing void!

	alias Algebraic!(int) nint;
	nint n1=5;
	assert(n1.get!(int)==5);
	n1=nint(); //still null
	assert(!n1.hasValue);
	assert(n1.peek!(int)==null);
	writeln("error:");
	assert(n1.get!(int)==0);
Mar 26 2011
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/26/11 5:33 PM, Kagamin wrote:

 Kagamin Wrote:


 Kagamin Wrote:


 Robert Jacques Wrote:


 Should be Algebraic!(T, void). If that doesn't currently compile, we
 should make it compile.

 Andrei


 For what it's worth, both Algebraic!(T, void*) and Algebraic!(T, void)
 compile on my branch of std.variant.


 can't assign null

 	alias Algebraic!(int,void) nint;
 	nint n1=5;
 	assert(n1==5);
 	n1=null;

 \std\variant.d:497: Error: static assert  "Cannot store a void* in a
VariantN!(maxSize,int,void). Valid types are (int, void)"
 test.d:45:        instantiated from here: opAssign!(void*)


 Ahh, there's a workaround:

 	alias Algebraic!(int,void) nint;
 	nint n1=5;
 	assert(n1.get!(int)==5);
 	n1=nint(); //uninitialized Variant, sort of null
 	assert(!n1.hasValue);
 	assert(n1.peek!(int)==null);
 	writeln("error here:");
 	assert(n1.get!(int)==0);


 this also works without allowing void!

 	alias Algebraic!(int) nint;
 	nint n1=5;
 	assert(n1.get!(int)==5);
 	n1=nint(); //still null
 	assert(!n1.hasValue);
 	assert(n1.peek!(int)==null);
 	writeln("error:");
 	assert(n1.get!(int)==0);


Awesome. Talk about forgetting my own code :o).

Andrei
Mar 26 2011