www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Proposal: Database Engine for D

reply Piotrek <nodata nodata.pl> writes:
The goal of this post is to measure the craziness of an idea to 
embed a database engine into the D language ;)

I think about a database engine which would meet my three main 
requirements:
   - integrated with D (ranges)
   - ACID
   - fast

Since the days when I was working on financing data SW I become 
allergic to SQL. I though that NoSQL databases would fill the 
bill. Unfortunately they didn't. And I want to have an ability to 
write a code like this without too much effort:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);

And other things like updating and deleting from db. I think you 
get my point.

So I started a PoC project based on SQLite design:
https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

The PoC code: 
https://github.com/PiotrekDlang/AirLock/tree/master/src/database

Can you please share your thoughts and experience on the topic? 
Has anyone tried similar things?

Piotrek
Dec 31 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 01/01/16 6:14 AM, Piotrek wrote:

 The goal of this post is to measure the craziness of an idea to embed a
 database engine into the D language ;)

 I think about a database engine which would meet my three main
 requirements:
    - integrated with D (ranges)
    - ACID
    - fast

 Since the days when I was working on financing data SW I become allergic
 to SQL. I though that NoSQL databases would fill the bill. Unfortunately
 they didn't. And I want to have an ability to write a code like this
 without too much effort:

    struct Person
    {
     string name;
     string surname;
     ubyte age;
     Address address;
    }

   DataBase db = new DataBase("file.db");
   auto coll = db.collection!Person("NSA.Registry");
   auto visitationList = coll.filter!(p => p.name == "James");
   writeln (visitationList);

 And other things like updating and deleting from db. I think you get my
 point.

 So I started a PoC project based on SQLite design:
 https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture


 The PoC code:
 https://github.com/PiotrekDlang/AirLock/tree/master/src/database

 Can you please share your thoughts and experience on the topic? Has
 anyone tried similar things?

 Piotrek


You've just introduced two topics.
The first is a database engine, abstracting away the drivers.
And second an ORM.
Dec 31 2015
parent Piotrek <nodata nodata.pl> writes:
On Friday, 1 January 2016 at 01:34:53 UTC, Rikki Cattermole wrote:

 You've just introduced two topics.
 The first is a database engine, abstracting away the drivers.
 And second an ORM.


And maybe even an object-oriented database management system to 
some extent. OTOH, I removed SQL from the stack.

Piotrek
Jan 02 2016
prev sibling next sibling parent reply tcak <1ltkrs+3wyh1ow7kzn1k sharklasers.com> writes:
On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:

 The goal of this post is to measure the craziness of an idea to 
 embed a database engine into the D language ;)

 I think about a database engine which would meet my three main 
 requirements:
   - integrated with D (ranges)
   - ACID
   - fast

 Since the days when I was working on financing data SW I become 
 allergic to SQL. I though that NoSQL databases would fill the 
 bill. Unfortunately they didn't. And I want to have an ability 
 to write a code like this without too much effort:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);

 And other things like updating and deleting from db. I think 
 you get my point.

 So I started a PoC project based on SQLite design:
 https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

 The PoC code: 
 https://github.com/PiotrekDlang/AirLock/tree/master/src/database

 Can you please share your thoughts and experience on the topic? 
 Has anyone tried similar things?

 Piotrek


You know someone needs to maintain all that code base 
continuously. When SQLite is a separate project, it has its own 
developers and we just bind to its library; it is same for other 
DBs. Your proposal is nice, but creating another standard, and a 
group of people needs to take care of it in both documentation 
and coding wise continuously are biggest issues.
Dec 31 2015
parent Piotrek <nodata nodata.pl> writes:
On Friday, 1 January 2016 at 04:20:19 UTC, tcak wrote:

 You know someone needs to maintain all that code base 
 continuously. When SQLite is a separate project, it has its own 
 developers and we just bind to its library; it is same for 
 other DBs. Your proposal is nice, but creating another 
 standard, and a group of people needs to take care of it in 
 both documentation and coding wise continuously are biggest 
 issues.


Agree. Lack of resources is an enemy of every project.

BTW. I don't introduce any new standard except a new file format.

Piotrek
Jan 02 2016
prev sibling next sibling parent reply Kapps <opantm2+spam gmail.com> writes:
On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);


This example shows the difficulty of doing this in D. You can't 
really have something like `p.Name == "James"`, or `p.Age < 21` 
translate to SQL properly without language changes, which I 
believe Walter or Andrei were against. This has been the key 
problem when things like Linq to Sql for D have been brought up 
before.
Jan 01 2016
next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 2016-01-01 at 10:00 +0000, Kapps via Digitalmars-d wrote:

 On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:

=20
 =C2=A0 struct Person
 =C2=A0 {
 =C2=A0=C2=A0=C2=A0string name;
 =C2=A0=C2=A0=C2=A0string surname;
 =C2=A0=C2=A0=C2=A0ubyte age;
 =C2=A0=C2=A0=C2=A0Address address;
 =C2=A0 }
=20
 =C2=A0DataBase db =3D new DataBase("file.db");
 =C2=A0auto coll =3D db.collection!Person("NSA.Registry");
 =C2=A0auto visitationList =3D coll.filter!(p =3D> p.name =3D=3D "James"=




);

 =C2=A0writeln (visitationList);


=20
 This example shows the difficulty of doing this in D. You can't=C2=A0
 really have something like `p.Name =3D=3D "James"`, or `p.Age < 21`=C2=A0
 translate to SQL properly without language changes, which I=C2=A0
 believe Walter or Andrei were against. This has been the key=C2=A0
 problem when things like Linq to Sql for D have been brought up=C2=A0
 before.


Why does it need language changes?

Having the ability to have an internal DSL instead of SQL string
fiddling is one of the major wins for SQLAlchemy. If it can be done in
Python why can't it be done in D?

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Jan 01 2016
next sibling parent reply Kapps <opantm2+spam gmail.com> writes:
On Friday, 1 January 2016 at 10:26:14 UTC, Russel Winder wrote:

 On Fri, 2016-01-01 at 10:00 +0000, Kapps via Digitalmars-d 
 wrote:

 On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:

 
   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }
 
  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);


 
 This example shows the difficulty of doing this in D. You 
 can't really have something like `p.Name == "James"`, or 
 `p.Age < 21` translate to SQL properly without language 
 changes, which I believe Walter or Andrei were against. This 
 has been the key problem when things like Linq to Sql for D 
 have been brought up before.


 Why does it need language changes?

 Having the ability to have an internal DSL instead of SQL 
 string fiddling is one of the major wins for SQLAlchemy. If it 
 can be done in Python why can't it be done in D?


Someone else can explain better / more correctly than me, but I 
believe the issue lies with opCmp and opEquals. You can make 
expressions like p.Name.equals("James") work (I believe using 
opDispatch), but because all you have is opEquals, you can't know 
if the user put in 'p.Name == "James"` or `p.Name != "James"`, as 
they both simply call opEquals. In order to do that, you would 
need things like opLessThan, opEquals, opNotEquals, 
opGreaterThan, etc, which would (with improper use or bugs) cause 
other issues, like a < b && a > b and a == b && a != b to be 
true, or a == b || a != b to be false.

I'm also not certain how you could implement `p => p.Name == 
"James" || p.Name == "Bob"`, but there might be a way? I think 
this is the gist of it, but I'm likely wrong on some aspects of 
this, so it would be good if someone else clarified..
Jan 01 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Friday, 1 January 2016 at 10:40:59 UTC, Kapps wrote:


 Someone else can explain better / more correctly than me, but I 
 believe the issue lies with opCmp and opEquals. You can make 
 expressions like p.Name.equals("James") work (I believe using 
 opDispatch), but because all you have is opEquals, you can't 
 know if the user put in 'p.Name == "James"` or `p.Name != 
 "James"`, as they both simply call opEquals. In order to do 
 that, you would need things like opLessThan, opEquals, 
 opNotEquals, opGreaterThan, etc, which would (with improper use 
 or bugs) cause other issues, like a < b && a > b and a == b && 
 a != b to be true, or a == b || a != b to be false.

 I'm also not certain how you could implement `p => p.Name == 
 "James" || p.Name == "Bob"`, but there might be a way? I think 
 this is the gist of it, but I'm likely wrong on some aspects of 
 this, so it would be good if someone else clarified..


That's exactly the problem. See an issue reported [1] and a 
related thread [2]. AST macros could solve the problem as well 
[3] (and many other problems).

[1] https://issues.dlang.org/show_bug.cgi?id=14593
[2] http://forum.dlang.org/thread/msvapl$2rmn$1 digitalmars.com
[3] http://wiki.dlang.org/DIP50

--
/Jacob Carlborg
Jan 01 2016
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 2016-01-01 at 11:03 +0000, Jacob Carlborg via Digitalmars-d
wrote:

=20


[=E2=80=A6]

 That's exactly the problem. See an issue reported [1] and a=C2=A0
 related thread [2]. AST macros could solve the problem as well=C2=A0
 [3] (and many other problems).
=20
 [1] https://issues.dlang.org/show_bug.cgi?id=3D14593
 [2] http://forum.dlang.org/thread/msvapl$2rmn$1 digitalmars.com
 [3] http://wiki.dlang.org/DIP50


Or alternative 4, fix D so that proper operator definition can be
achieved.
=C2=A0
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Jan 01 2016
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-01-01 12:33, Russel Winder via Digitalmars-d wrote:

 On Fri, 2016-01-01 at 11:03 +0000, Jacob Carlborg via Digitalmars-d
 wrote:


 […]

 That's exactly the problem. See an issue reported [1] and a
 related thread [2]. AST macros could solve the problem as well
 [3] (and many other problems).

 [1] https://issues.dlang.org/show_bug.cgi?id=14593
 [2] http://forum.dlang.org/thread/msvapl$2rmn$1 digitalmars.com
 [3] http://wiki.dlang.org/DIP50


 Or alternative 4, fix D so that proper operator definition can be
 achieved.


That would be the first link and the second link discussing that issue.

-- 
/Jacob Carlborg



 Jan 02 2016



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 1/1/2016 3:33 AM, Russel Winder via Digitalmars-d wrote:

 Or alternative 4, fix D so that proper operator definition can be
 achieved.


The way D's operator overloading is designed is not a mistake. It's limitations 
are a feature, not a bug. It was deliberately set up to:

1. Make doing C++ style iostreams hard.

2. Prevent clever use of operator overloading and expression templates to
create 
languages that look like D, but are NOT.

3. Work well when using operator overloading to implement arithmetic types.

For example, I've seen operator overloading used in C++ to turn it into a 
sort-of regex language. The failures of it are:

1. Sort-of because C++ operator precedence and prefix/postfix grammar is 
different than that of regex, so it can't be emulated correctly.

2. It is visually indistinguishable from C++ code. You simply cannot look at a 
piece of code and tell it is regex with utterly different meaning, rather than 
the usual meanings.

3. Any error messages from misuse are utterly and totally incomprehensible, 
because the compiler is designed to compile C++ and gives C++ messages, not 
regex messages.

4. C++ ETs are legendary in their tendency to consume all the memory in the 
computer and take incredibly long to compile. Most fail when the expressions 
exceed a rather small level of complexity because of this.

My not-so-humble opinion is these sorts of DSLs are technical demonstrations, 
but not useful nor desirable tools.

***************************************
So, what does D do?

    http://dlang.org/phobos/std_regex.html

D enables CTFE to write a PROPER compile time regex language, with correct
regex 
grammar, correct regex tokens, regex-specific error messages, etc. And it works 
great! It's not a hack, compromise, or workaround. It's a real, embedded DSL. 
And it's faster than any other regex engine on the market.



 Jan 04 2016



 next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Monday, 4 January 2016 at 09:44:35 UTC, Walter Bright wrote:

 My not-so-humble opinion is these sorts of DSLs are technical 
 demonstrations, but not useful nor desirable tools.


Well, this is wrong. They are desirable and work with tooling, 
editors, code completion etc. And also allow user extensions if 
done right.

Symbols should not be referenced as strings. That's bad. That's 
textual macro processing.



 Jan 04 2016



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 1/4/2016 2:34 AM, Ola Fosheim Grøstad wrote:

 On Monday, 4 January 2016 at 09:44:35 UTC, Walter Bright wrote:

 My not-so-humble opinion is these sorts of DSLs are technical demonstrations,
 but not useful nor desirable tools.


 Well, this is wrong. They are desirable and work with tooling, editors, code
 completion etc. And also allow user extensions if done right.


You didn't address my points. Have you used C++ ETs for production? They fall 
apart in real world use for the reasons I mentioned. It's one thing to read a 
paper about something, and another to use it for something larger than will fit 
on a slide.



 Jan 04 2016



  parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Monday, 4 January 2016 at 11:19:38 UTC, Walter Bright wrote:

ou didn't address my points. Have you used C++ ETs for
 production? They fall apart in real world use for the reasons I 
 mentioned. It's one thing to read a paper about something, and 
 another to use it for something larger than will fit on a slide.


You are assuming template expressions. I am not. The main 
overhead when interacting with databases is network traffic, 
building the query should be done at runtime. No need for 
templates.

I am in favour of having a clean core language (minimal language 
after desugaring/lowering) that can be manipulated using term 
rewriting/symbolic manipulation (or ast macros). But that sounds 
more like D4 than D2...?



 Jan 04 2016



prev sibling next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2016-01-04 10:44, Walter Bright wrote:


 So, what does D do?

     http://dlang.org/phobos/std_regex.html

 D enables CTFE to write a PROPER compile time regex language, with
 correct regex grammar, correct regex tokens, regex-specific error
 messages, etc. And it works great! It's not a hack, compromise, or
 workaround. It's a real, embedded DSL. And it's faster than any other
 regex engine on the market.


And how would you do that for SQL. Build a CTFE SQL database?

-- 
/Jacob Carlborg



 Jan 04 2016



prev sibling next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Mon, 2016-01-04 at 01:44 -0800, Walter Bright via Digitalmars-d
wrote:

 On 1/1/2016 3:33 AM, Russel Winder via Digitalmars-d wrote:

 Or alternative 4, fix D so that proper operator definition can be
 achieved.


=20
 The way D's operator overloading is designed is not a mistake. It's
 limitations=C2=A0
 are a feature, not a bug. It was deliberately set up to:
=20
 1. Make doing C++ style iostreams hard.


What is the problem with having the << and >> operators do input
output. Very object-oriented.=C2=A0


 2. Prevent clever use of operator overloading and expression
 templates to create=C2=A0
 languages that look like D, but are NOT.


But it would be D. Boost.Sprint code may look like EBNF, but it is C++.


 3. Work well when using operator overloading to implement arithmetic
 types.


It is important that this works. But it should be possible to create an
operator algebra for any type: arithmetic types are a very small subset
of types used in computing.


 For example, I've seen operator overloading used in C++ to turn it
 into a=C2=A0
 sort-of regex language. The failures of it are:
=20
 1. Sort-of because C++ operator precedence and prefix/postfix grammar
 is=C2=A0
 different than that of regex, so it can't be emulated correctly.
=20
 2. It is visually indistinguishable from C++ code. You simply cannot
 look at a=C2=A0
 piece of code and tell it is regex with utterly different meaning,
 rather than=C2=A0
 the usual meanings.
=20
 3. Any error messages from misuse are utterly and totally
 incomprehensible,=C2=A0
 because the compiler is designed to compile C++ and gives C++
 messages, not=C2=A0
 regex messages.
=20
 4. C++ ETs are legendary in their tendency to consume all the memory
 in the=C2=A0
 computer and take incredibly long to compile. Most fail when the
 expressions=C2=A0
 exceed a rather small level of complexity because of this.
=20
 My not-so-humble opinion is these sorts of DSLs are technical
 demonstrations,=C2=A0
 but not useful nor desirable tools.
=20
 ***************************************
 So, what does D do?
=20
 =C2=A0=C2=A0=C2=A0=C2=A0http://dlang.org/phobos/std_regex.html
=20
 D enables CTFE to write a PROPER compile time regex language, with
 correct regex=C2=A0
 grammar, correct regex tokens, regex-specific error messages, etc.
 And it works=C2=A0
 great! It's not a hack, compromise, or workaround. It's a real,
 embedded DSL.=C2=A0
 And it's faster than any other regex engine on the market.


I avoid regular expressions except when editing using ed, sed, or
emacs, so I cannot properly comment on the above. However I find
Boost.Sprint a very sensible way of using compile-time meta-object
protocols. You write a grammar in EBNF in the source code, and the
compiler writes the parser. Excellent use of internal DSL. And type
safe.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 04 2016



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 1/4/2016 10:25 AM, Russel Winder via Digitalmars-d wrote:

 On Mon, 2016-01-04 at 01:44 -0800, Walter Bright via Digitalmars-d
 wrote:

 1. Make doing C++ style iostreams hard.


 What is the problem with having the << and >> operators do input
 output. Very object-oriented.


I'm surprised you'd ask. I thought this was well known.

1. ugly as hell
2. not exception safe
3. not thread safe
4. interleaved output in the middle of lines when writing to both stdout and
stderr

2..4 are caused by inability to encapsulate a sequence of these operations.



 2. Prevent clever use of operator overloading and expression
 templates to create
 languages that look like D, but are NOT.


 But it would be D. Boost.Sprint code may look like EBNF, but it is C++.


  #define BEGIN {
  #define END }

is still C++, too.



 3. Work well when using operator overloading to implement arithmetic
 types.


 It is important that this works. But it should be possible to create an
 operator algebra for any type: arithmetic types are a very small subset
 of types used in computing.


What do you suggest when the operators and precedence of the desired algebraic 
type simply do not map cleanly onto C++ operators and precedence grammar? Allow 
users to define their own operators and redefine the precedence? Where is the 
line that shouldn't be crossed?



 I avoid regular expressions except when editing using ed, sed, or
 emacs, so I cannot properly comment on the above. However I find
 Boost.Sprint a very sensible way of using compile-time meta-object
 protocols. You write a grammar in EBNF in the source code, and the
 compiler writes the parser. Excellent use of internal DSL. And type
 safe.


EBNF doesn't map onto C++ operators and precedence, either. It's still a hack:

   https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_Form

With CFTE, the user can actually write actual EBNF.



 Jan 04 2016



 next sibling parent  Wyatt <wyatt.epp gmail.com>  writes:


On Monday, 4 January 2016 at 20:07:55 UTC, Walter Bright wrote:

 On 1/4/2016 10:25 AM, Russel Winder via Digitalmars-d wrote:

 It is important that this works. But it should be possible to 
 create an
 operator algebra for any type: arithmetic types are a very 
 small subset
 of types used in computing.


 What do you suggest when the operators and precedence of the 
 desired algebraic type simply do not map cleanly onto C++ 
 operators and precedence grammar? Allow users to define their 
 own operators and redefine the precedence? Where is the line 
 that shouldn't be crossed?


On this point specifically, I'm still considering attempting to 
write a DIP because it would be very useful to have some 

where they provide a set of acceptable characters for operators 
and define precedence automatically (depending on first character 
of the name).  (And definitely don't do the Haskell thing where 
any binary function can be turned into an infix op.)

-Wyatt



 Jan 04 2016



prev sibling  parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Mon, 2016-01-04 at 12:07 -0800, Walter Bright via Digitalmars-d
wrote:

=20


[=E2=80=A6]

 I'm surprised you'd ask. I thought this was well known.
=20
 1. ugly as hell


For you perhaps, others have different opinions. I like it.


 2. not exception safe


Indeed. but then most code is not exception safe which is part of the
problem with computing. Fixing C++ I/O is only about 0.001% of the
problem.


 3. not thread safe


Indeed. But then a properly architected system only has one point of
I/O per channel, which could then be single threaded, so less of a
problem.


 4. interleaved output in the middle of lines when writing to both
 stdout and stderr


This is a shell problem not a C++ problem.


 2..4 are caused by inability to encapsulate a sequence of these
 operations.


And also generally bad architecting of I/O in applications.

[=E2=80=A6]


=20


+.

=20
 =C2=A0 #define BEGIN {
 =C2=A0 #define END }
=20
 is still C++, too.


I thought Stephen Bourne was the only person who ever thought this was
sensible.

There is a difference always between what you can do and what you
should do. The language should not be fascist in stopping all things
not deemed the one true way by the language designers. To do so stops
serendipitous evolution.

[=E2=80=A6]


=20
 What do you suggest when the operators and precedence of the desired
 algebraic=C2=A0
 type simply do not map cleanly onto C++ operators and precedence
 grammar? Allow=C2=A0
 users to define their own operators and redefine the precedence?
 Where is the=C2=A0
 line that shouldn't be crossed?


There is no line. cf. Algol68.=C2=A0

Python only offers compiler understood operators and preserves
precedence, ditto C++ (the relationship is fairly obvious). Scala
allows arbitrary operators, cf. Algol68. Groovy follows the Python/C++
approach over Java.

So currently most people stick with know operators and allow
redefinition within a fixed precedence structure. Easy for the compiler
writer, constraining for the user programmers, but workable usually.

Scala has shown the way that Algol68 set out, compiler writers should
give up on imposing fixed expression structures in their languages.
Haskell has gone a little this way using `` for infix operator words.
Sort of works can be ugly. Of course Scala code can be made
incomprehensible using the operator definitions.

In the end it is up to programmers to do the right thing, and for
compiler writers not to tell them what they may or may not thing in a
given language. =C2=A0Restrictive languages, lead to restrictive
applications, and demise in the market.

[=E2=80=A6]

=20
 EBNF doesn't map onto C++ operators and precedence, either. It's
 still a hack:
=20
 =C2=A0=C2=A0=C2=A0https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93N=


aur_Form

=20
 With CFTE, the user can actually write actual EBNF.


Assuming EBNF is actually a good way of writing grammers. ;-)

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 09 2016



 next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 1/9/2016 3:02 AM, Russel Winder via Digitalmars-d wrote:

 [...]
 And also generally bad architecting of I/O in applications.


These problems have persisted since the genesis of iostreams. The problems do 
not exist in the way other modern languages do I/O. Iostreams was a great idea 
in 1985, but 30 years of experience has shown there are better ways.



 There is a difference always between what you can do and what you
 should do. The language should not be fascist in stopping all things
 not deemed the one true way by the language designers. To do so stops
 serendipitous evolution.


On the other hand, consider the C preprocessor. It has well known problems. But 
this never seems to discourage the most execrable edifices built on top of it. 
Their designers inevitably regard them as shining examples of their creativity. 
And maybe they are. But do they belong in professional code? No. But the only 
way to stop them is to not have a preprocessor in the language. (Of course, you 
can always use m4 as a front end to any language, but thankfully nobody seems
to 
do that.)

I believe there is a notion of "best practices", that some practices are simply 
better than others. A good language design makes it easy to follow best 
practices, and hard to follow known inferior ones. D is meant for professional 
use, and people who want professional code tend to want some mechanical 
assurance that best practices are followed (consider the increasing smorgasbord 
of third party tools designed to ferret out bad practices in C/C++ code).

There is plenty of innovation in programming language practice, I don't think
it 
is stifled.



 Jan 09 2016



prev sibling next sibling parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Saturday, 9 January 2016 at 11:02:46 UTC, Russel Winder wrote:

 There is a difference always between what you can do and what 
 you should do. The language should not be fascist in stopping 
 all things not deemed the one true way by the language 
 designers. To do so stops serendipitous evolution.


Indeed, although I think languages should have a library-mode and 
an application-mode.

That way you can constrain certain types of meta-programming to 
libraries and keep application implementation syntax simple and 
readable.

This is what happens in well structured programs anyway.



 Jan 09 2016



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 1/9/16 6:02 AM, Russel Winder via Digitalmars-d wrote:

 Assuming EBNF is actually a good way of writing grammers.;-)


What would be a better way? -- Andrei



 Jan 09 2016



  parent reply w0rp <devw0rp gmail.com>  writes:


I've played with the idea of using operator overloading for some 
kind of ORM before, but I don't think it's strictly necessary to 
use operator overloading for an ORM at all. Maybe in some cases 
it might make sense.

I don't think the answer for building such a thing is to think of 
one idea, find out it won't work out well, and then give up. You 
have to be more creative than that.



 Jan 12 2016



  parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Tuesday, 12 January 2016 at 12:53:29 UTC, w0rp wrote:

 I've played with the idea of using operator overloading for 
 some kind of ORM before, but I don't think it's strictly 
 necessary to use operator overloading for an ORM at all. Maybe 
 in some cases it might make sense.


The question is whether we want to provide a comparable 
experience to other languages or not.  What is desired depends on 
what other languages provide for a particular application area?

The ability to create abstractions is important for a language 
like D since that is supposed to be a primary feature.



 Jan 12 2016



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 01/04/2016 01:25 PM, Russel Winder via Digitalmars-d wrote:

 What is the problem with having the << and >> operators do input
 output. Very object-oriented.


What is the connection between using shift operators for I/O and object 
orientation?


 But it would be D. Boost.Sprint code may look like EBNF, but it is C++.


[snip]

 I avoid regular expressions except when editing using ed, sed, or
 emacs, so I cannot properly comment on the above. However I find
 Boost.Sprint a very sensible way of using compile-time meta-object
 protocols. You write a grammar in EBNF in the source code, and the
 compiler writes the parser. Excellent use of internal DSL. And type
 safe.


I don't think regexen were the point as much as an example. What is 
Boost.Sprint? I haven't heard of it, and can't find it. Thanks! -- Andrei



 Jan 04 2016



 next sibling parent  Zz <zz nospam.com>  writes:


 What is Boost.Sprint? I haven't heard of it, and can't find it. 
 Thanks! -- Andrei


I believe he meant Boost Spirit

http://www.boost.org/doc/libs/1_60_0/libs/spirit/doc/html/spirit/introduction.html

Zz



 Jan 04 2016



prev sibling next sibling parent  Ettienne Gilbert <ettienne.gilbert gmail.com>  writes:


On Monday, 4 January 2016 at 20:38:46 UTC, Andrei Alexandrescu 
wrote:

 On 01/04/2016 01:25 PM, Russel Winder via Digitalmars-d wrote:


[snip]

 What is Boost.Sprint? I haven't heard of it, and can't find it. 
 Thanks! -- Andrei


I think he miss-typed or miss-remembered. It is Boost.Spirit.

http://boost-spirit.com/home/



 Jan 05 2016



prev sibling  parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Mon, 2016-01-04 at 15:38 -0500, Andrei Alexandrescu via Digitalmars-
d wrote:

=20


[=E2=80=A6]

 I don't think regexen were the point as much as an example. What is=C2=A0
 Boost.Sprint? I haven't heard of it, and can't find it. Thanks!=C2=A0


As others mentioned, it was a mis-type, it is Boost.Spirit.

http://boost-spirit.com/home/


=20


--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 07 2016



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 01/07/2016 02:02 PM, Russel Winder via Digitalmars-d wrote:

 On Mon, 2016-01-04 at 15:38 -0500, Andrei Alexandrescu via Digitalmars-
 d wrote:


 […]

 I don't think regexen were the point as much as an example. What is
 Boost.Sprint? I haven't heard of it, and can't find it. Thanks!


 As others mentioned, it was a mis-type, it is Boost.Spirit.

 http://boost-spirit.com/home/


Got it, thanks. I took a look at a familiar topic - json grammars.

I just googled `boost spirit json` and followed through the top result. 
For C++, the grammar definition would be 
https://github.com/cierelabs/json_spirit/blob/master/ciere/json/parser/grammar_def.hpp.

Then I googled `pegged json` and the top result led to 
https://github.com/PhilippeSigaud/Pegged/blob/master/pegged/examples/json.d.

Which do you like more?


Andrei



 Jan 07 2016



  parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Thu, 2016-01-07 at 15:32 -0500, Andrei Alexandrescu via Digitalmars-
d wrote:

 On 01/07/2016 02:02 PM, Russel Winder via Digitalmars-d wrote:


[=E2=80=A6]

=20
 Got it, thanks. I took a look at a familiar topic - json grammars.
=20
 I just googled `boost spirit json` and followed through the top
 result.=C2=A0
 For C++, the grammar definition would be=C2=A0
 https://github.com/cierelabs/json_spirit/blob/master/ciere/json/parse
 r/grammar_def.hpp.
=20
 Then I googled `pegged json` and the top result led to=C2=A0
 https://github.com/PhilippeSigaud/Pegged/blob/master/pegged/examples/
 json.d.
=20
 Which do you like more?


Just a quick look I'm afraid, so very superficial at this stage:

The D code as presented is clearly simpler and easy to read. However
the D grammar is a string and not code so the anticipation is that
errors in the C++ code would be easier to find and correct. I guess the
question here is how good the compile time evaluation of strings is in
D and does it do good error reporting.

I still prefer the concept of Internal DSL explicitly in the code
rather than in strings just now, but this may still just be prejudice.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 09 2016



 next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 1/9/2016 2:49 AM, Russel Winder via Digitalmars-d wrote:

 The D code as presented is clearly simpler and easy to read. However
 the D grammar is a string and not code so the anticipation is that
 errors in the C++ code would be easier to find and correct. I guess the
 question here is how good the compile time evaluation of strings is in
 D and does it do good error reporting.


The compiler does not report the errors. The parsing is done by code supplied
by 
the module, and the error messages are as good or as bad as that code is 
written. The point being that one is not subjected to compiler generated error 
messages based on the compiler thinking it is C++/D code, but a custom written 
parser for the DSL that can give messages specific to the DSL.



 I still prefer the concept of Internal DSL explicitly in the code
 rather than in strings just now, but this may still just be prejudice.


I can see only one advantage to using ETs for DSLs:

* 3rd party tools like syntax directed editors, formatters and highlighters
will 
operate on it. But they'll still operate on it as if it were C++ code.

CTFE based DSLs win on every other point I can think of, including error
messages.



 Jan 09 2016



prev sibling  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 1/9/16 5:49 AM, Russel Winder via Digitalmars-d wrote:

 On Thu, 2016-01-07 at 15:32 -0500, Andrei Alexandrescu via Digitalmars-
 d wrote:

 On 01/07/2016 02:02 PM, Russel Winder via Digitalmars-d wrote:


 […]

 Got it, thanks. I took a look at a familiar topic - json grammars.

 I just googled `boost spirit json` and followed through the top
 result.
 For C++, the grammar definition would be
 https://github.com/cierelabs/json_spirit/blob/master/ciere/json/parse
 r/grammar_def.hpp.

 Then I googled `pegged json` and the top result led to
 https://github.com/PhilippeSigaud/Pegged/blob/master/pegged/examples/
 json.d.

 Which do you like more?


 Just a quick look I'm afraid, so very superficial at this stage:

 The D code as presented is clearly simpler and easy to read. However
 the D grammar is a string and not code so the anticipation is that
 errors in the C++ code would be easier to find and correct. I guess the
 question here is how good the compile time evaluation of strings is in
 D and does it do good error reporting.


The string parser is custom code and issues errors as defined by the 
library author. (I don't have experience with Pegged in particular.) I 
think one difficulty with ETs is reporting line where errors occur in 
multi-line strings. Language facilities could help here. This is an area 
of strategic importance for D.

I do have extensive experience with error messages in C++ ETs, which 
were and are inscrutable.


 I still prefer the concept of Internal DSL explicitly in the code
 rather than in strings just now, but this may still just be prejudice.


Prejudice can last only so long in the face of evidence.


Andrei



 Jan 09 2016



prev sibling  parent  rsw0x <anonymous anonymous.com>  writes:


On Monday, 4 January 2016 at 09:44:35 UTC, Walter Bright wrote:

 On 1/1/2016 3:33 AM, Russel Winder via Digitalmars-d wrote:

 Or alternative 4, fix D so that proper operator definition can 
 be
 achieved.


 The way D's operator overloading is designed is not a mistake. 
 It's limitations are a feature, not a bug. It was deliberately 
 set up to:

 1. Make doing C++ style iostreams hard.


struct _cout {

   auto opBinary(string s : "<<", T)(auto ref T rhs) {
     import std.stdio : write;

     write(rhs);
     return this;
   }
}

_cout cout;
void main() {
   cout << "Walter " << "hates " << "this." << endl;
}


and yet, I can't overload the address-of operator because Walter 
is afraid I might shoot my foot.



 Jan 07 2016



prev sibling  parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Fri, 2016-01-01 at 10:40 +0000, Kapps via Digitalmars-d wrote:

 [=E2=80=A6]
=20
 Someone else can explain better / more correctly than me, but I=C2=A0
 believe the issue lies with opCmp and opEquals. You can make=C2=A0
 expressions like p.Name.equals("James") work (I believe using=C2=A0
 opDispatch), but because all you have is opEquals, you can't know=C2=A0
 if the user put in 'p.Name =3D=3D "James"` or `p.Name !=3D "James"`, as=


=C2=A0

 they both simply call opEquals. In order to do that, you would=C2=A0
 need things like opLessThan, opEquals, opNotEquals,=C2=A0
 opGreaterThan, etc, which would (with improper use or bugs) cause=C2=A0
 other issues, like a < b && a > b and a =3D=3D b && a !=3D b to be=C2=A0
 true, or a =3D=3D b || a !=3D b to be false.
=20
 I'm also not certain how you could implement `p =3D> p.Name =3D=3D=C2=A0
 "James" || p.Name =3D=3D "Bob"`, but there might be a way? I think=C2=A0
 this is the gist of it, but I'm likely wrong on some aspects of=C2=A0
 this, so it would be good if someone else clarified..


Hummm=E2=80=A6 so to put it another way, the D meta-object protocol is even
more broken than that of Java: at least in Java there isn't even a
pretence that you can create an internal DSL. This is very, very sad, I
had not realized D was this broken.
=C2=A0
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 01 2016



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 1/1/16 6:30 AM, Russel Winder via Digitalmars-d wrote:

 On Fri, 2016-01-01 at 10:40 +0000, Kapps via Digitalmars-d wrote:

 […]

 Someone else can explain better / more correctly than me, but I
 believe the issue lies with opCmp and opEquals. You can make
 expressions like p.Name.equals("James") work (I believe using
 opDispatch), but because all you have is opEquals, you can't know
 if the user put in 'p.Name == "James"` or `p.Name != "James"`, as
 they both simply call opEquals. In order to do that, you would
 need things like opLessThan, opEquals, opNotEquals,
 opGreaterThan, etc, which would (with improper use or bugs) cause
 other issues, like a < b && a > b and a == b && a != b to be
 true, or a == b || a != b to be false.

 I'm also not certain how you could implement `p => p.Name ==
 "James" || p.Name == "Bob"`, but there might be a way? I think
 this is the gist of it, but I'm likely wrong on some aspects of
 this, so it would be good if someone else clarified..


 Hummm… so to put it another way, the D meta-object protocol is even
 more broken than that of Java: at least in Java there isn't even a
 pretence that you can create an internal DSL. This is very, very sad, I
 had not realized D was this broken.


Well I guess it's time to cut losses and cancel that D tutorial at ACCU 
2016, eh :o).

I understand how the shortcomings of D's expression templates (ETs) for 
SQL (as compared to C++ or LINQ) can be construed as an indication of a 
language breakage, but I don't quite agree for a few reasons.

One, D offers another mechanism aside from ETs for embedded 
domain-specific languages. So in a way there's less pressure on offering 
lush ET than would be in other languages.

Second, ET as a mechanism for SQL interface has other inherent 
limitations. Consider the "LIKE" operator in SQL, which has no ET 
equivalent in C++ with similar syntax, and no direct equivalent in LINQ. 
That doesn't mean the respective languages are broken.

Third, evaluating the merits and demerits of a language choice should be 
done within several appropriate use cases, not just one. For example, 
D's use of opEquals/opCmp saves a lot of boilerplate and potential bugs 
in many cases compared to C++. On another example, C++'s overloading of 
&&, ||, and the comma operator are considered downright disastrous and 
are unrecommended by virtually all coding standards and guidelines, yet 
do work with ETs.


Andrei



 Jan 03 2016



 next sibling parent reply rumbu <rumbu rumbu.ro>  writes:


On Monday, 4 January 2016 at 00:49:29 UTC, Andrei Alexandrescu 
wrote:

 Second, ET as a mechanism for SQL interface has other inherent 
 limitations. Consider the "LIKE" operator in SQL, which has no 
 ET equivalent in C++ with similar syntax, and no direct 
 equivalent in LINQ. That doesn't mean the respective languages 
 are broken.


LINQ transforms StartsWith, EndsWith and Contains methods in LIKE 
queries under the hood since Entity Framework 4.0 (8 years ago).
Also, when in doubt, there is an entire namespace dedicated to 
LINQ queries with direct SQL equivalents (including LIKE) - 
https://msdn.microsoft.com/en-us/library/system.data.linq.sqlclient.sqlmethods(v=vs.110).aspx



 Jan 03 2016



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 01/04/2016 01:49 AM, rumbu wrote:

 On Monday, 4 January 2016 at 00:49:29 UTC, Andrei Alexandrescu wrote:

 Second, ET as a mechanism for SQL interface has other inherent
 limitations. Consider the "LIKE" operator in SQL, which has no ET
 equivalent in C++ with similar syntax, and no direct equivalent in
 LINQ. That doesn't mean the respective languages are broken.


 LINQ transforms StartsWith, EndsWith and Contains methods in LIKE
 queries under the hood since Entity Framework 4.0 (8 years ago).
 Also, when in doubt, there is an entire namespace dedicated to LINQ
 queries with direct SQL equivalents (including LIKE) -
 https://msdn.microsoft.com/en-us/library/system.data.linq.sqlclient.sqlmethods(v=vs.110).aspx


Misunderstanding. Please re-read what I wrote. -- Andrei



 Jan 04 2016



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-04 01:49, Andrei Alexandrescu wrote:


 Second, ET as a mechanism for SQL interface has other inherent
 limitations. Consider the "LIKE" operator in SQL, which has no ET
 equivalent in C++ with similar syntax, and no direct equivalent in LINQ.
 That doesn't mean the respective languages are broken.


It's one operator that doesn't have a direct corresponding in the host 
language. With D, it's not possible to handle most of the operators.


 Third, evaluating the merits and demerits of a language choice should be
 done within several appropriate use cases, not just one. For example,
 D's use of opEquals/opCmp saves a lot of boilerplate and potential bugs
 in many cases compared to C++.


If D allowed to separately overload the comparison operators the 
standard library should of course provide a mixin of some kind to reduce 
the boilerplate.


 On another example, C++'s overloading of
 &&, ||, and the comma operator are considered downright disastrous and
 are unrecommended by virtually all coding standards and guidelines, yet
 do work with ETs.


I think for generating SQL, overloading && and || would be a valid use case.

There's of course AST macros as well, which have many other good use 
cases. Unfortunately you don't like those either :(

-- 
/Jacob Carlborg



 Jan 03 2016



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 1/3/2016 11:40 PM, Jacob Carlborg wrote:

 There's of course AST macros as well, which have many other good use cases.
 Unfortunately you don't like those either :(


Neither Andrei nor I have changed our minds on that one.



 Jan 04 2016



  parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Mon, 2016-01-04 at 01:45 -0800, Walter Bright via Digitalmars-d
wrote:

 On 1/3/2016 11:40 PM, Jacob Carlborg wrote:

 There's of course AST macros as well, which have many other good
 use cases.
 Unfortunately you don't like those either :(


=20
 Neither Andrei nor I have changed our minds on that one.


Pity I like good AST transforms: Groovy, Scala, Lisp, Clojure, Rust=E2=80=
=A6
C++ template meta-programming isn't really my cup of tea.

Rusts macros show it can be done well.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 04 2016



 next sibling parent reply rsw0x <anonymous anonymous.com>  writes:


On Monday, 4 January 2016 at 18:28:15 UTC, Russel Winder wrote:

 On Mon, 2016-01-04 at 01:45 -0800, Walter Bright via 
 Digitalmars-d wrote:

 On 1/3/2016 11:40 PM, Jacob Carlborg wrote:

 There's of course AST macros as well, which have many other 
 good
 use cases.
 Unfortunately you don't like those either :(


 
 Neither Andrei nor I have changed our minds on that one.


 Pity I like good AST transforms: Groovy, Scala, Lisp, Clojure, 
 Rust… C++ template meta-programming isn't really my cup of tea.

 Rusts macros show it can be done well.


I agree heavily with this, it feels like an arbitrary limitation.
D did wonders in making template metaprogramming usable compared 
to C++, but sometimes it feels like trying to pound nails in with 
a screwdriver.

I think just looking at the kind of stuff being done in languages 
that enable AST macros might change minds.



 Jan 04 2016



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 01/04/2016 02:50 PM, rsw0x wrote:

 D did wonders in making template metaprogramming usable compared to C++,
 but sometimes it feels like trying to pound nails in with a screwdriver.


Do you have examples of that awkwardness happening? Is it suitable to 
supplement those with CTFE? -- Andrei



 Jan 04 2016



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 01/04/2016 01:28 PM, Russel Winder via Digitalmars-d wrote:

 Rusts macros show it can be done well.


Do you have a few examples handy? Thanks. -- Andrei



 Jan 04 2016



 next sibling parent  Charles Hixson via Digitalmars-d <digitalmars-d puremagic.com>  writes:


FWIW, were I proposing a "Database Engine for D" I'd be proposing a 
B+Tree that was restricted to storing explicit data (no pointers or 
other indirection...including strings, you'd need to specify fixed size 
arrays of char, wchar, or dchar).  There would be one "type"/file, and 
the "type" would be given by a struct that contained no pointers either 
obvious or hidden (i.e., dynamic arrays, strings, etc.).  This would be 
for simplicity of implementation and basic functionality. Later I would 
consider implementing secondary keys which would be implemented as 
databases storing only the key value and the record number.  Since 
records are of fixed size, sorting them would be trivial, but only the 
secondary keys would provide this option because you want the record 
number of the main file to be reliable and invariant.

SQL seems like a poor fit to D.  At some point one might consider a 
fancier front end to the database that would allow indirections, but 
that's a whole can of worms, with threats of multiple copies that have 
different values for items that started out pointing to the same variable.

On 01/04/2016 12:39 PM, Andrei Alexandrescu via Digitalmars-d wrote:

 On 01/04/2016 01:28 PM, Russel Winder via Digitalmars-d wrote:

 Rusts macros show it can be done well.


 Do you have a few examples handy? Thanks. -- Andrei





 Jan 04 2016



prev sibling  parent  Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Mon, 2016-01-04 at 15:39 -0500, Andrei Alexandrescu via Digitalmars-
d wrote:

 On 01/04/2016 01:28 PM, Russel Winder via Digitalmars-d wrote:

 Rusts macros show it can be done well.


=20
 Do you have a few examples handy? Thanks. -- Andrei


Will do but it may be next week. Sorry, but medical stuff=E2=80=A6
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 07 2016



prev sibling  parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Monday, 4 January 2016 at 00:49:29 UTC, Andrei Alexandrescu 
wrote:

 Second, ET as a mechanism for SQL interface has other inherent 
 limitations. Consider the "LIKE" operator in SQL, which has no 
 ET equivalent in C++ with similar syntax, and no direct


like(Person.Name,"peter%")

Person.Name == pattern("peter%")


 On another example, C++'s overloading of &&, ||, and the comma 
 operator are considered downright disastrous and are 
 unrecommended by virtually all coding standards and guidelines,


Guideline means that you should think hard about it before you do 
break it. A guidelines does not mean that you never should do it. 
It means you should only break the guideline when you have good 
reasons to do so.

If you can overload "and" and "or", you also can implement 
ternary logic, fuzzy logic and other logics. In some cases that 
is desirable.

In C++ features were abused when they were new because people 
were eager to stretch new features to the limits. This is not a 
problem in regular C++ programs. And in the case of C++ 
iostreams, it has become an idiom that isn't causing any 
problems. C++ programmers are never confused by shift and output 
operators.

But if this is a big deal the easy solution is to add unicode 
operators without default behaviour.

String processing is not an acceptable alternative, it ruins 
tooling. You should essentially never be able to turn a symbol 
into a string and vice versa. D unfortunately allows you to do 
it. That is a mal-feature aka a textual macro feature.

You can't really say that D has gotten rid of macros when you 
actually encourage using macro-like textual substitution features 
in place of processing  proper symbols. That is an outdated 70s 
paradigm.



 Jan 04 2016



prev sibling  parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Friday, 1 January 2016 at 10:26:14 UTC, Russel Winder wrote:

 Why does it need language changes?

 Having the ability to have an internal DSL instead of SQL 
 string fiddling is one of the major wins for SQLAlchemy. If it 
 can be done in Python why can't it be done in D?


In D1 Walter made a point about restricting operator overloading 
to discourage reuse of operators. In D2 there are many ways to 
create your own weird syntax, but Walter is locked on his D1 
position, even though it makes little sense in D2.

This is an overarching theme in D: the design rationale, that 
goes a decade back and is based on making a restricted easy to 
use version of C++, does not change, even though the context is 
very different in 2015 and the premises has changed.

These things are unlikely to change without a fork. Which is a 
pitty as D needs a more coherent design to get out of stagnation.



 Jan 01 2016



 next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Fri, 2016-01-01 at 10:45 +0000, Ola Fosheim Gr=C3=B8stad via Digitalmars=
-
d wrote:

=20


[=E2=80=A6]

 In D1 Walter made a point about restricting operator overloading=C2=A0
 to discourage reuse of operators. In D2 there are many ways to=C2=A0
 [=E2=80=A6]


Java also went the route of "operator definition is too hard for
programmers to deal with so we will not allow it". Every language on
the JVM other than Java has made a point of allowing, indeed
encouraging, sensible operator definition (*). =C2=A0Sadly too few people
working on the JVM are allowed to use languages other than Java.

(*) Arguably Scala takes this too far.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 01 2016



 next sibling parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Friday, 1 January 2016 at 11:37:40 UTC, Russel Winder wrote:

 Java also went the route of "operator definition is too hard 
 for programmers to deal with so we will not allow it". Every 
 language on the JVM other than Java has made a point of 
 allowing, indeed encouraging, sensible operator definition (*).


Unfortunately language design is often not as principled, 
scientific or ergonomic as it ought to be and is often heavily 
tainted by the moral opinions and personal practices of their 
designers.

Unfortunately infrastructure matters more than ergonomics. I 
guess Python is a rare exception.



 Jan 01 2016



prev sibling next sibling parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Friday, 1 January 2016 at 11:37:40 UTC, Russel Winder wrote:

 (*) Arguably Scala takes this too far.



Btw, regarding operator overloading:

Your comment about Scala made me think about the proposals for 
Algol which allowed the language to be heavily extended with even 
custom operator precedence IIRC. Interestingly, the Pony authors 
have decided that most operators should be explicitly grouped 
using parentheses, they argue that most programmers fail to 
remember precedence. I think that is mostly the case, especially 
with the somewhat flawed C precedence rules.

Related note, I find this tongue-in-cheek comparison of Go and 
Algol68 is quite entertaining:

http://cowlark.com/2009-11-15-go/

I think older languages such as Simula67, Algol68, Beta, ML etc 
still are quite interesting objects for study. It is scary how 
little progress have been made in the past 30 years... 
Fortunately some people are preserving books in PDF format.

http://www.softwarepreservation.org/projects/ALGOL/book/Lindsey_van_der_Meulen-IItA68-Revised.pdf



 Jan 02 2016



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 1/1/2016 3:37 AM, Russel Winder via Digitalmars-d wrote:

 On Fri, 2016-01-01 at 10:45 +0000, Ola Fosheim Grøstad via Digitalmars-
 d wrote:

 In D1 Walter made a point about restricting operator overloading
 to discourage reuse of operators. In D2 there are many ways to


 Java also went the route of "operator definition is too hard for
 programmers to deal with so we will not allow it".



"Also" is the wrong word. See my other post in this thread for why that was 
done, and no, it is not Java's reason. It has nothing to do with "too hard for 
programmers".



 Jan 04 2016



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-01 11:45, Ola Fosheim Grøstad wrote:


 In D1 Walter made a point about restricting operator overloading to
 discourage reuse of operators. In D2 there are many ways to create your
 own weird syntax, but Walter is locked on his D1 position, even though
 it makes little sense in D2.

 This is an overarching theme in D: the design rationale, that goes a
 decade back and is based on making a restricted easy to use version of
 C++, does not change, even though the context is very different in 2015
 and the premises has changed.

 These things are unlikely to change without a fork. Which is a pitty as
 D needs a more coherent design to get out of stagnation.


The core developers are making a big deal out of being able to have 
DSL's as string literals and process them at compile time. Although 
that's kind of pointless with SQL, since one still needs to send to 
string to the database to perform the query. The only thing that's 
possible is to validate the syntax at compile time.

-- 
/Jacob Carlborg



 Jan 02 2016



 next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Saturday, 2 January 2016 at 12:23:30 UTC, Jacob Carlborg wrote:

 The core developers are making a big deal out of being able to 
 have DSL's as string literals and process them at compile time.


They have probably never done professional work with an ORM...

Nobody wants that.



 Jan 02 2016



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-02 13:55, Ola Fosheim Grøstad wrote:


 They have probably never done professional work with an ORM...

 Nobody wants that.


Exactly, but tell them ;)

-- 
/Jacob Carlborg



 Jan 02 2016



  parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Saturday, 2 January 2016 at 19:46:48 UTC, Jacob Carlborg wrote:

 On 2016-01-02 13:55, Ola Fosheim Grøstad wrote:


 They have probably never done professional work with an ORM...

 Nobody wants that.


 Exactly, but tell them ;)


I did:

http://forum.dlang.org/post/mfdjcgykaizbzuuhoupv forum.dlang.org

But Walter thinks that enforcing object-value comparisons in a 
rather limited way is a feature. It is a feature... but it is the 
wrong kind of feature... ;) It is an oversimplified and 
ineffective solution.

A more sensible approach would be to use type-classes/concepts 
for defining what kind of comparison relation a class is required 
to provide and then deduce comparison operators based on what the 
programmer provides (e.g. if the programmer only provides a 
definition for "≤" then the compiler can deduce "<", ">", "≥", 
"==" etc.).

D moved into templates and generics without cleaning up  the 
language and is essentially ducktyped and macroish. Like C++. And 
Go (just in a different way).

But C++ might start on a change process with C++17/20. We'll see.



 Jan 02 2016



prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu>  writes:


On Saturday, 2 January 2016 at 12:23:30 UTC, Jacob Carlborg wrote:

 The core developers are making a big deal out of being able to 
 have DSL's as string literals and process them at compile time. 
 Although that's kind of pointless with SQL, since one still 
 needs to send to string to the database to perform the query. 
 The only thing that's possible is to validate the syntax at 
 compile time.


Well, you can also generate the structs and specific 
serialization code. And depending on how advanced your dsl you 
can also auto generate database migration code. There are 
propably tons of other stuff you can do with it.

All in all much better than extending the language.



 Jan 02 2016



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-02 19:37, Sebastiaan Koppe wrote:


 Well, you can also generate the structs and specific serialization code.
 And depending on how advanced your dsl you can also auto generate
 database migration code. There are propably tons of other stuff you can
 do with it.

 All in all much better than extending the language.


I would rather to the opposite. Generate the necessary SQL for a 
migration based on a struct, not the other way around.

-- 
/Jacob Carlborg



 Jan 02 2016



  parent reply Sebastiaan Koppe <mail skoppe.eu>  writes:


On Saturday, 2 January 2016 at 19:48:26 UTC, Jacob Carlborg wrote:

 I would rather to the opposite. Generate the necessary SQL for 
 a migration based on a struct, not the other way around.


I meant that you generate the struct from the DSL, and then 
migration code from that struct.



 Jan 02 2016



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-02 21:48, Sebastiaan Koppe wrote:


 I meant that you generate the struct from the DSL, and then migration
 code from that struct.


I don't think I understand, it seems complicated.

-- 
/Jacob Carlborg



 Jan 03 2016



  parent reply Sebastiaan Koppe <mail skoppe.eu>  writes:


On Sunday, 3 January 2016 at 14:32:48 UTC, Jacob Carlborg wrote:

 On 2016-01-02 21:48, Sebastiaan Koppe wrote:


 I meant that you generate the struct from the DSL, and then 
 migration
 code from that struct.


 I don't think I understand, it seems complicated.


Suppose you have this:

mixin(db(`
Entity Person
   Fields
     name -> string
     age -> integer
   Query
     byAge(a -> integer) -> age == a
`));

which generates something like this:

struct Person
{
   string name;
   int age
}
auto getPersonByAge(DB db, int a)
{
   return db.prepare!Person("SELECT name,age FROM Person WHERE age 
= ?").query(a);
}

and then later in time:

mixin(db(`
Entity Person
   Fields
     name -> string
     age -> integer
     phone -> string
   Query
     byAge(a -> integer) -> age == a
`));

Given that you have access to both version it is easy to generate 
migration code for the phone field.

Maybe it is contrived, but I think it shows you can do more with 
the DSL than just validating queries.



 Jan 03 2016



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-03 17:45, Sebastiaan Koppe wrote:


 Suppose you have this:

 mixin(db(`
 Entity Person
    Fields
      name -> string
      age -> integer
    Query
      byAge(a -> integer) -> age == a
 `));

 which generates something like this:

 struct Person
 {
    string name;
    int age
 }
 auto getPersonByAge(DB db, int a)
 {
    return db.prepare!Person("SELECT name,age FROM Person WHERE age =
 ?").query(a);
 }

 and then later in time:

 mixin(db(`
 Entity Person
    Fields
      name -> string
      age -> integer
      phone -> string
    Query
      byAge(a -> integer) -> age == a
 `));

 Given that you have access to both version it is easy to generate
 migration code for the phone field.

 Maybe it is contrived, but I think it shows you can do more with the DSL
 than just validating queries.


Perhaps I'm missing something obvious but there are several problems 
with this:

1. What happens when you use more than one query for the same table at 
the same scope? In the above case, "Person" is already defined the 
second time "db" is invoked. It's not possible to add fields for already 
declared structs. Unless you use some form of opDispatch backed by an 
associative array of variants

2. What happens if I want to execute a custom query in a function, i.e. 
a query that is only used once. Will it generate the sturct inside the 
function or am I forced to always use this mixin at module level?

I still think it's a lot easier to declare the struct with standard D 
code. I don't think the DSL adds any value in this case. Just do 
something like:

 db struct Person
{
     string name;
     int age;
}

The  db attribute would allow to create the migrations.

-- 
/Jacob Carlborg



 Jan 03 2016



  parent reply Sebastiaan Koppe <mail skoppe.eu>  writes:


On Monday, 4 January 2016 at 07:48:14 UTC, Jacob Carlborg wrote:

 Perhaps I'm missing something obvious but there are several 
 problems with this:

 1. What happens when you use more than one query for the same 
 table at the same scope? In the above case, "Person" is already 
 defined the second time "db" is invoked. It's not possible to 
 add fields for already declared structs. Unless you use some 
 form of opDispatch backed by an associative array of variants


While it wasn't apparent, when I said "later" I meant "later" in 
git's history. The second version would be in a future commit, 
i.e. when changes are made. Then you can diff it and generate 
migration code. (Obviously this requires the DSL to be in a 
separate file without other code. We used to have such a 
reflective system in place in our continuous deployment. It would 
look at the changes in git since the last deployment and when 
certain files changed it knew it had to do certain stuff.)


 2. What happens if I want to execute a custom query in a 
 function, i.e. a query that is only used once. Will it generate 
 the sturct inside the function or am I forced to always use 
 this mixin at module level?


I suppose you need to declare each and every query in the DSL. 
But you might be on to something ugly in what I am proposing.

Having said that, facebook seems to be going the DSL route. 
https://facebook.github.io/graphql and 
https://facebook.github.io/react/blog/2015/05/01/graphql-introduction.html

Another benefit of a DSL is that you can generate code for other 
languages. For instance you could generate a JS, Swift or Android 
client.


 I still think it's a lot easier to declare the struct with 
 standard D code. I don't think the DSL adds any value in this 
 case. Just do something like:

  db struct Person
 {
     string name;
     int age;
 }

 The  db attribute would allow to create the migrations.


But then you are back to square one with regards to queries.



 Jan 04 2016



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-04 12:14, Sebastiaan Koppe wrote:


 But then you are back to square one with regards to queries.


If you can't use the host language, i.e. Person.where(e => e.name == 
"John");, what's the point of inventing a new DSL? Why not just use SQL 
if the DSL needs to be in a string anyway.

-- 
/Jacob Carlborg



 Jan 04 2016



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 01/04/2016 07:39 AM, Jacob Carlborg wrote:

 If you can't use the host language, i.e. Person.where(e => e.name ==
 "John");, what's the point of inventing a new DSL? Why not just use SQL
 if the DSL needs to be in a string anyway.


I agree. A CTFE SQL parser with automatic variable binding would be 
pretty rad. Not to mention it would by design take care of things like 
string injection attacks etc. Also: finding SQL syntax errors during 
compilation of D code... priceless. -- Andrei



 Jan 04 2016



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 1/2/16 7:23 AM, Jacob Carlborg wrote:

 On 2016-01-01 11:45, Ola Fosheim Grøstad wrote:


 In D1 Walter made a point about restricting operator overloading to
 discourage reuse of operators. In D2 there are many ways to create your
 own weird syntax, but Walter is locked on his D1 position, even though
 it makes little sense in D2.

 This is an overarching theme in D: the design rationale, that goes a
 decade back and is based on making a restricted easy to use version of
 C++, does not change, even though the context is very different in 2015
 and the premises has changed.

 These things are unlikely to change without a fork. Which is a pitty as
 D needs a more coherent design to get out of stagnation.


 The core developers are making a big deal out of being able to have
 DSL's as string literals and process them at compile time. Although
 that's kind of pointless with SQL, since one still needs to send to
 string to the database to perform the query. The only thing that's
 possible is to validate the syntax at compile time.


Binding D variables to SQL expressions also comes to mind. -- Andrei



 Jan 03 2016



  parent  Jacob Carlborg <doob me.com>  writes:


On 2016-01-04 01:50, Andrei Alexandrescu wrote:


 Binding D variables to SQL expressions also comes to mind. -- Andrei


You have also been pushing a lot for ranges, which I think is good. I 
would much rather like to view a table as a range, but the algorithms 
would be preform in the database instead of in the application code. 
Something like:

Person.filter!(e => e.name == "John")

-- 
/Jacob Carlborg



 Jan 03 2016



prev sibling  parent reply Piotrek <nodata nodata.pl>  writes:


On Friday, 1 January 2016 at 10:00:43 UTC, Kapps wrote:


 This example shows the difficulty of doing this in D. You can't 
 really have something like `p.Name == "James"`, or `p.Age < 21` 
 translate to SQL properly without language changes, which I 
 believe Walter or Andrei were against. This has been the key 
 problem when things like Linq to Sql for D have been brought up 
 before.


Not really. There is no translation stage to SQL or any other DSL 
in the proposal. So this problem doesn't exist and no language 
changes are needed.

However there is another issue which must be taken into account. 
One should decide if the object is retrieved directly or via via 
proxy. Especially for big objects with lot of aggregated types.

Piotrek



 Jan 02 2016



  parent reply Chris Wright <dhasenan gmail.com>  writes:


On Sat, 02 Jan 2016 16:40:16 +0000, Piotrek wrote:


 On Friday, 1 January 2016 at 10:00:43 UTC, Kapps wrote:
 

 This example shows the difficulty of doing this in D. You can't really
 have something like `p.Name == "James"`, or `p.Age < 21`
 translate to SQL properly without language changes, which I believe
 Walter or Andrei were against. This has been the key problem when
 things like Linq to Sql for D have been brought up before.


 
 Not really. There is no translation stage to SQL or any other DSL in the
 proposal. So this problem doesn't exist and no language changes are
 needed.


So you want to create the following query:

  people.filter!(x => x.surname == "Slughorn");

And you've got ten million people in the collection, and you want this 
query to finish soonish. So you need to use an index. But a full index 
scan isn't so great; you want to do an index lookup if possible.

That's simple enough; we generate proxy types to record what properties 
you're using and what operations you're performing. PersonProxy records 
that you're accessing a field 'surname', gives a StringFieldProxy, and 
that records that you're checking for equality with the string "Slughorn". 
The lambda returns true when opEquals returns true.

But people write queries that are more complex than that, like:

  people.filter!(x => x.surname == "Slughorn" || x.age <= 17);

First time you run this, x.surname.opEquals("Slughorn") returns true and 
the expression as a whole returns true. You missed the second part of the 
expression. That's bad.

So we need to evaluate this lambda twice per parameter. (Actually, thanks 
to opCmp, sometimes you'll have to evaluate it three times.) We use that 
to build up a giant truth table, and then we can execute your query.

And that "twice per parameter" thing is exponential, and we build up a 
truth table that's exponentially large with respect to the complexity of 
the query. Some queries I've written for production systems would take a 
week for this system to prepare to execute and require a petabyte of 
storage space.

This is, shall we say, less than ideal.

You might be able to support queries as large as ten comparisons in a 
reasonable timeframe. But for all but the most trivial queries, it'll be 
faster to use SQL.

Fortunately, trivial queries are common. You could write this system and 
have it work up to, say, five comparisons, and when your query exceeds 
that limit, it will throw an exception asking you to rewrite the query in 
SQL.

It wouldn't be able to tell you what the equivalent query is, however. 
It's using a truth table and doesn't have access to what you wrote. I 
mean, sure, it could try to work backwards from the truth table, but 
that's rather expensive.



 Jan 02 2016



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 1/2/16 3:47 PM, Chris Wright wrote:

 On Sat, 02 Jan 2016 16:40:16 +0000, Piotrek wrote:


 On Friday, 1 January 2016 at 10:00:43 UTC, Kapps wrote:


 This example shows the difficulty of doing this in D. You can't really
 have something like `p.Name == "James"`, or `p.Age < 21`
 translate to SQL properly without language changes, which I believe
 Walter or Andrei were against. This has been the key problem when
 things like Linq to Sql for D have been brought up before.


 Not really. There is no translation stage to SQL or any other DSL in the
 proposal. So this problem doesn't exist and no language changes are
 needed.


 So you want to create the following query:

    people.filter!(x => x.surname == "Slughorn");

 And you've got ten million people in the collection, and you want this
 query to finish soonish. So you need to use an index. But a full index
 scan isn't so great; you want to do an index lookup if possible.

 That's simple enough; we generate proxy types to record what properties
 you're using and what operations you're performing. PersonProxy records
 that you're accessing a field 'surname', gives a StringFieldProxy, and
 that records that you're checking for equality with the string "Slughorn".
 The lambda returns true when opEquals returns true.

 But people write queries that are more complex than that, like:

    people.filter!(x => x.surname == "Slughorn" || x.age <= 17);

 First time you run this, x.surname.opEquals("Slughorn") returns true and
 the expression as a whole returns true. You missed the second part of the
 expression. That's bad.

 So we need to evaluate this lambda twice per parameter. (Actually, thanks
 to opCmp, sometimes you'll have to evaluate it three times.) We use that
 to build up a giant truth table, and then we can execute your query.

 And that "twice per parameter" thing is exponential, and we build up a
 truth table that's exponentially large with respect to the complexity of
 the query. Some queries I've written for production systems would take a
 week for this system to prepare to execute and require a petabyte of
 storage space.

 This is, shall we say, less than ideal.


This may in fact be good signal that an approach based on expression 
templates is not the most appropriate for D. -- Andrei



 Jan 03 2016



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-04 00:50, Andrei Alexandrescu wrote:


 This may in fact be good signal that an approach based on expression
 templates is not the most appropriate for D. -- Andrei


This whole thread has already discussed and showed that D operator 
overloading is lacking in terms of expression templates. The post by 
Chris assumes no language changes. If D supported overloading all 
operators the opEquals method would not return true, it would return a 
proxy that overloaded the || operator. The whole lambda expression would 
only generate a single query. The rest of the post falls apart from this 
mistake.

-- 
/Jacob Carlborg



 Jan 03 2016



  parent  Piotrek <nodata nodata.pl>  writes:


On Monday, 4 January 2016 at 07:59:40 UTC, Jacob Carlborg wrote:

 On 2016-01-04 00:50, Andrei Alexandrescu wrote:


 This may in fact be good signal that an approach based on 
 expression
 templates is not the most appropriate for D. -- Andrei


 This whole thread has already discussed and showed that D 
 operator overloading is lacking in terms of expression 
 templates. The post by Chris assumes no language changes. If D 
 supported overloading all operators the opEquals method would 
 not return true, it would return a proxy that overloaded the || 
 operator. The whole lambda expression would only generate a 
 single query. The rest of the post falls apart from this 
 mistake.


But most of the posts are related to DSL (->SQL). What do you 
think about solution where no DSL, VM or other intermediate layer 
is used?

Also I have a theory that SQL appeared because of limiting 
expressiveness of other programming languages (like C) ;)

Cheers
Piotrek



 Jan 04 2016



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-02 21:47, Chris Wright wrote:


 So you want to create the following query:

    people.filter!(x => x.surname == "Slughorn");

 And you've got ten million people in the collection, and you want this
 query to finish soonish. So you need to use an index. But a full index
 scan isn't so great; you want to do an index lookup if possible.

 That's simple enough; we generate proxy types to record what properties
 you're using and what operations you're performing. PersonProxy records
 that you're accessing a field 'surname', gives a StringFieldProxy, and
 that records that you're checking for equality with the string "Slughorn".
 The lambda returns true when opEquals returns true.

 But people write queries that are more complex than that, like:

    people.filter!(x => x.surname == "Slughorn" || x.age <= 17);

 First time you run this, x.surname.opEquals("Slughorn") returns true and
 the expression as a whole returns true. You missed the second part of the
 expression. That's bad.


If D had better operator overloading or supported AST macros, opEquals 
would not return true. It would return a new proxy that overloads the || 
operator. The rest of the post falls apart from this mistake.


-- 
/Jacob Carlborg



 Jan 03 2016



 next sibling parent reply rsw0x <anonymous anonymous.com>  writes:


On Monday, 4 January 2016 at 07:55:53 UTC, Jacob Carlborg wrote:

 On 2016-01-02 21:47, Chris Wright wrote:


 So you want to create the following query:

    people.filter!(x => x.surname == "Slughorn");

 And you've got ten million people in the collection, and you 
 want this
 query to finish soonish. So you need to use an index. But a 
 full index
 scan isn't so great; you want to do an index lookup if 
 possible.

 That's simple enough; we generate proxy types to record what 
 properties
 you're using and what operations you're performing. 
 PersonProxy records
 that you're accessing a field 'surname', gives a 
 StringFieldProxy, and
 that records that you're checking for equality with the string 
 "Slughorn".
 The lambda returns true when opEquals returns true.

 But people write queries that are more complex than that, like:

    people.filter!(x => x.surname == "Slughorn" || x.age <= 17);

 First time you run this, x.surname.opEquals("Slughorn") 
 returns true and
 the expression as a whole returns true. You missed the second 
 part of the
 expression. That's bad.


 If D had better operator overloading or supported AST macros, 
 opEquals would not return true. It would return a new proxy 
 that overloads the || operator. The rest of the post falls 
 apart from this mistake.


compiler plugins like Rust to enable AST macros could fix this 
but walter and andrei seem extremely opposed to any form of 
macros so we end up with ugly heavily templated ugly band-aids

shame, because I think D will quickly lose their lead in the 
metaprogramming area if Rust keeps it up - e.g, someone already 
made a GC Plugin for rust's compiler



 Jan 04 2016



  parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Mon, 2016-01-04 at 08:29 +0000, rsw0x via Digitalmars-d wrote:

=20


[=E2=80=A6]

 compiler plugins like Rust to enable AST macros could fix this=C2=A0
 but walter and andrei seem extremely opposed to any form of=C2=A0
 macros so we end up with ugly heavily templated ugly band-aids


Rust may have macros, in fact it has two sorts, but they are not as
useful as you might at first think.

The macro system as in Rust stable is not up to the task of doing code
generation such that you can replicate PyTest's pytest.mark.parametrize
decorator in Rust. The compiler plugins that would allow this are only
allowed in Rust nightly and are forbidden in =C2=A0Rust beta and stable.
Rust nightly is effectively a totally different platform to Rust beta
or stable, which is undermining the Rust platform. =C2=A0

I must now try creating a D version of the pytest.mark.parametrize
decorator =E2=80=93 unless someone already has and I have just missed it.


 shame, because I think D will quickly lose their lead in the=C2=A0
 metaprogramming area if Rust keeps it up - e.g, someone already=C2=A0
 made a GC Plugin for rust's compiler


I think Rust and not C++17 is the biggest threat to D just now.

I am still intending to rewrite Me TV in D.
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 04 2016



  parent reply John Colvin <john.loughran.colvin gmail.com>  writes:


On Monday, 4 January 2016 at 12:28:47 UTC, Russel Winder wrote:

 I must now try creating a D version of the 
 pytest.mark.parametrize decorator – unless someone already has 
 and I have just missed it.


I quick look at pytest.mark.parametrize suggests it could be 
implemented with UDAs and a test-runner that finds all 
declarations a module (recursively) and does all the relevant 
logic (e.g. that's got more than one instance of parametrize, so 
do some sort of cartesian product of the inputs) and actually 
runs the test.

The main thing that python has here over D is that D's UDAs can't 
directly modify the function they're attached to, but I don't 
think that's necessary for parametrize.

Interestingly, functions can query their own attributes:

 (3) auto attr()
{
	return __traits(getAttributes, attr)[0];
}

unittest
{
	assert(attr() == 3);
}

not sure when I'd use that though...



 Jan 04 2016



 next sibling parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 01/04/2016 08:09 AM, John Colvin wrote:

 On Monday, 4 January 2016 at 12:28:47 UTC, Russel Winder wrote:

 I must now try creating a D version of the pytest.mark.parametrize
 decorator – unless someone already has and I have just missed it.


 I quick look at pytest.mark.parametrize suggests it could be implemented
 with UDAs and a test-runner that finds all declarations a module
 (recursively) and does all the relevant logic (e.g. that's got more than
 one instance of parametrize, so do some sort of cartesian product of the
 inputs) and actually runs the test.

 The main thing that python has here over D is that D's UDAs can't
 directly modify the function they're attached to, but I don't think
 that's necessary for parametrize.

 Interestingly, functions can query their own attributes:

  (3) auto attr()
 {
      return __traits(getAttributes, attr)[0];
 }

 unittest
 {
      assert(attr() == 3);
 }

 not sure when I'd use that though...


For computed attributes in template functions. -- Andrei



 Jan 04 2016



prev sibling  parent  Russel Winder via Digitalmars-d <digitalmars-d puremagic.com>  writes:


On Mon, 2016-01-04 at 13:09 +0000, John Colvin via Digitalmars-d wrote:

=20


[=E2=80=A6]

 not sure when I'd use that though...


At a trivial level that nonetheless leads to functionality that can be
used more sensibly, think of testing factorial. You want property-based=20
testing, but also you want a (well usually two) table of examples.

Python via pytest.mark.parametrize does the latter very well.
Hypothesis does the former. Spock enables both very easily.

It is know that the tests for the Java driver to MongoDB relies on
table-driven and property-based testing, mostly using Spock.

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder



 Jan 04 2016



prev sibling  parent reply Chris Wright <dhasenan gmail.com>  writes:


On Mon, 04 Jan 2016 08:55:53 +0100, Jacob Carlborg wrote:


 If D had better operator overloading or supported AST macros, opEquals
 would not return true. It would return a new proxy that overloads the ||
 operator. The rest of the post falls apart from this mistake.


Not proposing language changes was an intentional feature, not a mistake.



 Jan 04 2016



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-01-05 05:16, Chris Wright wrote:


 Not proposing language changes was an intentional feature, not a mistake.


Then you obviously can't use the operators. You would have to fall back 
to methods:

Person.where!(e => e.name.eq("John"));

-- 
/Jacob Carlborg



 Jan 05 2016



 next sibling parent reply Chris Wright <dhasenan gmail.com>  writes:


On Tue, 05 Jan 2016 09:26:16 +0100, Jacob Carlborg wrote:


 On 2016-01-05 05:16, Chris Wright wrote:
 

 Not proposing language changes was an intentional feature, not a
 mistake.


 
 Then you obviously can't use the operators. You would have to fall back
 to methods:
 
 Person.where!(e => e.name.eq("John"));


Which some people call a DSL, and DSLs were, I understand, something we 
were trying to avoid. You get far less readability than you do with plain 
D or a SQL string.

Plus side, you spend no time parsing, but it took less than a millisecond 
to parse your SQL query in the first place, and most SQL databases allow 
you to cache prepared statements.



 Jan 05 2016



  parent  Jacob Carlborg <doob me.com>  writes:


On 2016-01-05 18:32, Chris Wright wrote:


 Which some people call a DSL, and DSLs were, I understand, something we
 were trying to avoid. You get far less readability than you do with plain
 D or a SQL string.


The above is plain D. Not sure what you're referring to. And a DSL is 
usually used to increase readability.

Andrei and Walter want to avoid expression templates, not a DSL, as far 
as I understand it.

-- 
/Jacob Carlborg



 Jan 06 2016



prev sibling  parent reply Mengu <mengukagan gmail.com>  writes:


On Tuesday, 5 January 2016 at 08:26:16 UTC, Jacob Carlborg wrote:

 On 2016-01-05 05:16, Chris Wright wrote:


 Not proposing language changes was an intentional feature, not 
 a mistake.


 Then you obviously can't use the operators. You would have to 
 fall back to methods:

 Person.where!(e => e.name.eq("John"));


i love how things can become so complex in this community. i am a 
web developer so i am going to talk in terms of simplicity.

as russel said, with SQLAlchemy (python library) operators are 
overloaded and we can do 
DBSession.query(User).filter(User.first_name == "Jacob", User.age 

 27). this will generate the following query:


"SELECT * FROM users u WHERE u.first_name = 'Jacob' AND u.age > 
27" and will let the DB handle the rest.

but as a web developer, i don't mind if i have to type 
User.where("first_name = ? AND age > 27", 
request.POST.get('name')). this will generate the same query 
above. i added the question mark to say the parameters should be 
escaped properly.

guys, what we need is a DB abstraction supporting PostgreSQL, 
MySQL and other major database systems and we need it yesterday. 
let's not make things more complex than they are and build up 
this thing.

p.s. this is not a reply to jacob, just his post was the one i 
clicked reply for. :-)



 Jan 05 2016



 next sibling parent  Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Tuesday, 5 January 2016 at 22:12:05 UTC, Mengu wrote:

 guys, what we need is a DB abstraction supporting PostgreSQL, 
 MySQL and other major database systems and we need it 
 yesterday. let's not make things more complex than they are and 
 build up this thing.


What is needed isn't a DB abstraction, but a language that makes 
elegant programming abstractions easy to implement for a wide 
variety of problem areas.

You can do DB abstractions in any language... No point in having 
a thread about it if anything goes. Just do 2-3 different 
prototypes and select the best ones. No discussion needed...



 Jan 05 2016



prev sibling next sibling parent  Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net>  writes:


On Tuesday, 5 January 2016 at 22:12:05 UTC, Mengu wrote:

 but as a web developer, i don't mind if i have to type 
 User.where("first_name = ? AND age > 27", 
 request.POST.get('name')). this will generate the same query 
 above. i added the question mark to say the parameters should 
 be escaped properly.


As a web developer, I _do_ care, though. For one, I shouldn't 
have to know how the fields are called in the database. But more 
importantly, it isn't composable: in the general case, you can 
not write e.g.

     User.joins("other_table ON ...").where("first_name = ? ...");

and expect it to work, because `first_name` could be ambiguous. 
This in turn means that you can't create abstractions, for 
example:

     auto users = User
                  .onlyMale
                  .where(["birthday": Date.today])
                  .orderBy!(User.firstName);

Here your query would have to know how male and female users are 
represented in your database, e.g. by an integer, a lowercase 
'm', an uppercase 'M'... Such lack of abstraction can quickly 
make your codebase unmaintainable.



 Jan 06 2016



prev sibling next sibling parent  jmh530 <john.michael.hall gmail.com>  writes:


On Tuesday, 5 January 2016 at 22:12:05 UTC, Mengu wrote:

 guys, what we need is a DB abstraction supporting PostgreSQL, 
 MySQL and other major database systems and we need it 
 yesterday. let's not make things more complex than they are and 
 build up this thing.


I'm personally a fan of dplyr using R. bachmeier is apparently 
working on a way to embed R in D. Might be an option if you need 
it yesterday.

An example I saw above of

people.filter!(x => x.surname == "Slughorn");

looked a lot like a D version of dplyr's filter function.



 Jan 07 2016



prev sibling  parent reply Mengu <mengukagan gmail.com>  writes:


On Tuesday, 5 January 2016 at 22:12:05 UTC, Mengu wrote:

 On Tuesday, 5 January 2016 at 08:26:16 UTC, Jacob Carlborg 
 wrote:

 [...]


 i love how things can become so complex in this community. i am 
 a web developer so i am going to talk in terms of simplicity.

 as russel said, with SQLAlchemy (python library) operators are 
 overloaded and we can do 
 DBSession.query(User).filter(User.first_name == "Jacob", 
 User.age > 27). this will generate the following query:

 "SELECT * FROM users u WHERE u.first_name = 'Jacob' AND u.age > 
 27" and will let the DB handle the rest.

 but as a web developer, i don't mind if i have to type 
 User.where("first_name = ? AND age > 27", 
 request.POST.get('name')). this will generate the same query 
 above. i added the question mark to say the parameters should 
 be escaped properly.

 guys, what we need is a DB abstraction supporting PostgreSQL, 
 MySQL and other major database systems and we need it 
 yesterday. let's not make things more complex than they are and 
 build up this thing.

 p.s. this is not a reply to jacob, just his post was the one i 
 clicked reply for. :-)


and while we were talking the talk, rust community rolled out 
something good called diesel. check it out at http://diesel.rs/.

we need tools that get things done. we do not need tools that 
makes things more complex than they already are.



 Feb 05 2016



 next sibling parent  Sebastiaan Koppe <mail skoppe.eu>  writes:


On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:

 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.


I saw the examples on the site. In every case I prefer reading 
SQL instead of a complicated function chain with semantics I have 
yet to learn; SQL I already know.

A nice thing is the `less boilerplate` example. But I think they 
took a wrong approach by deriving the class from Queryable. I 
like D's approach better.



 Feb 06 2016



prev sibling  parent reply Piotrek <nodata nodata.pl>  writes:


On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:

 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.


Almost no one (including me) is interested in ORM for SQL. The 
point is ORM+SQL is limiting and sooner or later you fallback to 
SQL.

Additionally there is no critical mass for this kind of big 
project (combining all the SQL engines - good luck).

Andrei suggested a CTFE sql parser, so people who like SQL (not 
me) can benefit from the D metaprogramming power.

For the rest there is my proposal ;) : a language embedded DB. As 
far as I can tell none of the known PLes has this "killer" 
feature.

Piotrek



 Feb 06 2016



 next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?=  writes:


On Saturday, 6 February 2016 at 13:41:03 UTC, Piotrek wrote:

 On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:

 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at 
 http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.


 Almost no one (including me) is interested in ORM for SQL. The 
 point is ORM+SQL is limiting and sooner or later you fallback 
 to SQL.


A good ORM-like interface is mandatory for working with NoSQL 
databases...



 Feb 06 2016



  parent  Piotrek <nodata nodata.pl>  writes:


On Saturday, 6 February 2016 at 14:04:42 UTC, Ola Fosheim Grøstad 
wrote:

 A good ORM-like interface is mandatory for working with NoSQL 
 databases...


Fortunately, I don't plan to work with so called NoSQL 
databases...

BTW. Take a look at the example from the PoC code and check what 
works currently (i.e. passing unit tests)

https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#example

Piotrek



 Feb 06 2016



prev sibling next sibling parent  Charles <csmith.ku2013 gmail.com>  writes:


On Saturday, 6 February 2016 at 13:41:03 UTC, Piotrek wrote:

 For the rest there is my proposal ;) : a language embedded DB. 
 As far as I can tell none of the known PLes has this "killer" 
 feature.

 Piotrek


SAS does, and has for quite a few decades now. Its a pretty big 
corporate language used for statistics and analysis. It does work 
exceptionally well, easily handling well over 100M records within 
a single table. They implement a table per file (sas7bdat). That 
said, the binary table format is proprietary, and while there 
have been attempts to reverse engineer it none have been 
completely successful.



 Feb 06 2016



prev sibling next sibling parent  Chris Wright <dhasenan gmail.com>  writes:


On Sat, 06 Feb 2016 13:41:03 +0000, Piotrek wrote:


 On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:

 and while we were talking the talk, rust community rolled out something
 good called diesel. check it out at http://diesel.rs/.

 we need tools that get things done. we do not need tools that makes
 things more complex than they already are.


 
 Almost no one (including me) is interested in ORM for SQL.


You mean, a high-level query language abstracting across multiple 
databases, expressed in terms of my object model rather than physical 
tables. You're talking about HQL, not Hibernate as a whole.

Nobody wants to write manual serialization and deserialization code. Not 
when there's a reliable, low-cost alternative. And if I could just write 
my object model and have an automated system create my database tables 
and upgrade scripts for me, that would be cool too.


 The point is
 ORM+SQL is limiting and sooner or later you fallback to SQL.


The HQL I recall would be bad for reporting and aggregation, but since I 
never needed that, perhaps I just didn't find the relevant options and 
syntax.

I find HQL slightly nicer to work with than SQL, but since the databases 
themselves aren't speaking it, it's not so handy. I found one query that 
I needed where HQL could do it in two minutes and hand-written SQL could 
do it in fifty milliseconds. That's rare, but it does happen.



 Feb 06 2016



prev sibling next sibling parent  David DeWitt <dkdewitt gmail.com>  writes:


On Saturday, 6 February 2016 at 13:41:03 UTC, Piotrek wrote:

 On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:

 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at 
 http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.


 Almost no one (including me) is interested in ORM for SQL. The 
 point is ORM+SQL is limiting and sooner or later you fallback 
 to SQL.


I wouldn't say almost no one.  I know Sql Alchemy is very popular 
and used by a lot of ppl. Ive used and not used ORMs on numerous 
projects and sometimes they make sense and sometimes they do not. 
  I agree its not needed but alot of these tools are used by 
average developers and not having them isn't going to help D.   
Not that we don't need them but when ppl say "Well I like D but 
find there are missing libraries I want"  A decent ORM would 
prolly be one of them.  Basically if Rust continues to get these 
libraries and they are decent then they will def attract more ppl 
to the language instead of them possibly coming to D if that is 
important which I know has been brought up before and I think 
projects and tools like these with further attract ppl.



 Feb 06 2016



prev sibling  parent reply Mengu <mengukagan gmail.com>  writes:


On Saturday, 6 February 2016 at 13:41:03 UTC, Piotrek wrote:

 On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:

 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at 
 http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.


 Almost no one (including me) is interested in ORM for SQL. The 
 point is ORM+SQL is limiting and sooner or later you fallback 
 to SQL.

 Additionally there is no critical mass for this kind of big 
 project (combining all the SQL engines - good luck).

 Andrei suggested a CTFE sql parser, so people who like SQL (not 
 me) can benefit from the D metaprogramming power.

 For the rest there is my proposal ;) : a language embedded DB. 
 As far as I can tell none of the known PLes has this "killer" 
 feature.

 Piotrek


i don't mind if it's an ORM or something else. my point was that 
instead of complaining about stuff, we need a safe, stable and 
extendable database library supporting sqlite, mysql, postgresql, 
mssql and oracle dbs and we need it like yesterday. nothing 
fancy. people can get creative and fancy over that standard api 
and users get to choose.



 Feb 08 2016



  parent  Zardoz <luis.panadero gmail.com>  writes:


On Monday, 8 February 2016 at 13:36:09 UTC, Mengu wrote:

 i don't mind if it's an ORM or something else. my point was 
 that instead of complaining about stuff, we need a safe, stable 
 and extendable database library supporting sqlite, mysql, 
 postgresql, mssql and oracle dbs and we need it like yesterday. 
 nothing fancy. people can get creative and fancy over that 
 standard api and users get to choose.


Something like JDBC on D ... --> 
https://code.dlang.org/packages/ddbc



 Feb 09 2016



prev sibling  parent reply Piotrek <nodata nodata.pl>  writes:


On Saturday, 2 January 2016 at 20:47:37 UTC, Chris Wright wrote:

 So you want to create the following query:

   people.filter!(x => x.surname == "Slughorn");

 And you've got ten million people in the collection, and you 
 want this query to finish soonish. So you need to use an index. 
 But a full index scan isn't so great; you want to do an index 
 lookup if possible.



Correct.


 That's simple enough; we generate proxy types to record what 
 properties you're using and what operations you're performing. 
 PersonProxy records that you're accessing a field 'surname', 
 gives a StringFieldProxy, and that records that you're checking 
 for equality with the string "Slughorn". The lambda returns 
 true when opEquals returns true.

 But people write queries that are more complex than that, like:

   people.filter!(x => x.surname == "Slughorn" || x.age <= 17);

 First time you run this, x.surname.opEquals("Slughorn") returns 
 true and the expression as a whole returns true. You missed the 
 second part of the expression. That's bad.
 So we need to evaluate this lambda twice per parameter.


Hmm. Probably we don't think about the "proxy" term in the same 
way.
When I mentioned proxy I thought about a skeleton object to 
access parts of the original heavy object. So in fact, filter 
works on proxy and one evaluation is performed no matter how 
complex the condition is. The proxy is responsible to retrieve 
needed data. Additionally (I haven't mentioned it yet) we can 
provide mechanism for data integrity when joining separate 
objects.


 You might be able to support queries as large as ten 
 comparisons in a reasonable timeframe. But for all but the most 
 trivial queries, it'll be faster to use SQL.


SQL is no magic. You can write a code to beat any SQL planner. 
Also I think you had "joins" in mind which I guess are the 
biggest problem for performance.

But "What if I tell there is no SQL" !!!! Not even under the hood.

Piotrek



 Jan 04 2016



  parent reply Chris Wright <dhasenan gmail.com>  writes:


On Mon, 04 Jan 2016 16:59:47 +0000, Piotrek wrote:

 SQL is no magic.


True. You could equivalently have a string containing valid D code, 
accompanied by CTFE parsers that will determine which indices to use. 
This has typically been considered an antipattern. It tends to work 
poorly with refactoring tools, for instance, and there's an assumption 
that string literals are data and not code.


keeps the benefits of writing everything in the same language.



 Jan 04 2016



  parent  Piotrek <nodata nodata.pl>  writes:


On Tuesday, 5 January 2016 at 04:19:01 UTC, Chris Wright wrote:

 You could equivalently have a string containing valid D code, 
 accompanied by CTFE parsers that will determine which indices 
 to use. This has typically been considered an antipattern. It 
 tends to work poorly with refactoring tools, for instance, and 
 there's an assumption that string literals are data and not 
 code.


Would the D string tokens do?

BTW. Thanks for all your valuable comments. I really like that 
kind of critique.



 expression trees keeps the benefits of writing everything in 
 the same language.


I think I'm still not convinced to that approach (seems hacky). I 
agree with Walter in that matter (language changes, AST, 
overloading etc).

Especially, D is number one in meta-programming capabilities 
(among C++ and Rust) and nothing seems to change it in 
foreseeable future 
(https://users.rust-lang.org/t/rust-vs-dlang-i-want-more-experienced/4472/11)

Piotrek



 Feb 06 2016



prev sibling next sibling parent reply Abdulhaq <alynch4047 gmail.com>  writes:


On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:

 The goal of this post is to measure the craziness of an idea to 
 embed a database engine into the D language ;)

 I think about a database engine which would meet my three main 
 requirements:
   - integrated with D (ranges)
   - ACID
   - fast

 Since the days when I was working on financing data SW I become 
 allergic to SQL. I though that NoSQL databases would fill the 
 bill. Unfortunately they didn't. And I want to have an ability 
 to write a code like this without too much effort:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);

 And other things like updating and deleting from db. I think 
 you get my point.

 So I started a PoC project based on SQLite design:
 https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

 The PoC code: 
 https://github.com/PiotrekDlang/AirLock/tree/master/src/database

 Can you please share your thoughts and experience on the topic? 
 Has anyone tried similar things?

 Piotrek


My two pence, if you want it to be fast then it must have a good 
implementation of indices. Your filter functions should not 
actually start collecting real records, but instead should simply 
change the way that the cursor traverses the underlying data 
store. You will need good query 'compilation' like the big boys 
do, which work out which tables and indices to use and in which 
order, based on stats of the data / indices.

If you want ACID then SQL seems like a good approach to me, 
certainly I wouldn't want anything ORM-like for updating / 
inserting data.

There a number of good libraries out there already, SQLite 
obviously springs to mind.

It would be a fun project but perhaps a lot more work than you 
realised if you really want isolation levels, speed etc.



 Jan 03 2016



  parent  Piotrek <nodata nodata.pl>  writes:


On Sunday, 3 January 2016 at 19:48:42 UTC, Abdulhaq wrote:

 My two pence, if you want it to be fast then it must have a 
 good implementation of indices. Your filter functions should 
 not actually start collecting real records, but instead should 
 simply change the way that the cursor traverses the underlying 
 data store. You will need good query 'compilation' like the big 
 boys do, which work out which tables and indices to use and in 
 which order, based on stats of the data / indices.


Agree (I mentioned about proxies before). But I'm not sure about 
"good query compilation" thing. Good storage management should 
do. Rest can be done in user code.


 If you want ACID then SQL seems like a good approach to me, 
 certainly I wouldn't want anything ORM-like for updating / 
 inserting data.


Well. Some people really like SQL. I'm not one of them. I'm 
neither a fan of ORM+SQL. But I think ACID is not only reserved 
for SQL.


 There a number of good libraries out there already, SQLite 
 obviously springs to mind.


The SQLite project is great.


 It would be a fun project but perhaps a lot more work than you 
 realised if you really want isolation levels, speed etc.


Yes, I know. But one step at a time. It's a some kind of fun for 
me as well ;)

Cheers
Piotrek



 Jan 04 2016



prev sibling next sibling parent reply Jakob Jenkov <jakob jenkov.com>  writes:


You could just target your database at data analysis. Then you 
don't need to care about ACID, transactions etc. Just load all 
the data into memory, and start analyzing it.

Also, you'd typically be scanning over large parts of the data 
set for each query, so you may not need to support a full query 
language. Just what is needed for data analysis.

Later you can modify your engine to support ACID, more expressive 
query language etc.

On one of the projects I am working on right now, we will also 
implement our own database engine. We need it to integrate 
tightly with the rest our architecture, and the only way to do 
that is to roll our own. We will also not be using SQL because 
SQL is so limiting.

So, I'd say "go ahead" - you can only learn something from the 
project. I've "reinvented a lot of wheels" over the years, and 
each time I came out smarter than before. Not every reinvention 
was a success, but I always learned something from the process.



 Jan 03 2016



  parent reply Piotrek <nodata nodata.pl>  writes:


On Sunday, 3 January 2016 at 23:22:17 UTC, Jakob Jenkov wrote:

 You could just target your database at data analysis. Then you 
 don't need to care about ACID, transactions etc. Just load all 
 the data into memory, and start analyzing it.

 Also, you'd typically be scanning over large parts of the data 
 set for each query, so you may not need to support a full query 
 language. Just what is needed for data analysis.

 Later you can modify your engine to support ACID, more 
 expressive query language etc.


That's the plan:) Except no dedicated query language is planned. 
At least that's my vision based on what I know about D and 
databases currently.


 On one of the projects I am working on right now, we will also 
 implement our own database engine. We need it to integrate 
 tightly with the rest our architecture, and the only way to do 
 that is to roll our own. We will also not be using SQL because 
 SQL is so limiting.

 So, I'd say "go ahead" - you can only learn something from the 
 project. I've "reinvented a lot of wheels" over the years, and 
 each time I came out smarter than before. Not every reinvention 
 was a success, but I always learned something from the process.


Thanks! So at least one more soul believing that D can approach 
the SQL expressiveness in db domain.

Cheers
Piotrek



 Jan 04 2016



  parent  Gianni Pisetta <pisetta.gianni alice.it>  writes:


On Monday, 4 January 2016 at 17:36:20 UTC, Piotrek wrote:

 Thanks! So at least one more soul believing that D can approach 
 the SQL expressiveness in db domain.

 Cheers
 Piotrek


Some time ago I had the same idea as yours. IMHO, no D to SQL is 
really needed for this, and the same goes for relational 
databases. My idea was to initially build a DB as library, using 
graph databases as starting point, where the metadata is defined 
only one time in the code. The query methods must allow easy 
access not only to the data, but also to the relations defined 
from object to object. As a starting point you must be able to do 
something like this:

// Stores are where the data is saved
auto localStore = FileStore!Endian.littleEndian( "some_file_name" 
); // Local file

// load() and save() on objects of type User uses this store.
User.setDefaultStore( localStore );

User user = new User();

user.name = "John";
user.password = sha1( "password" );

// Uses the default store
auto localId = user.save();
// Uses the store specified as argument
auto otherId = user.save( someOtherStore );

// load the object from the default store
User localUser = User.load( localId );
// load the object from the network store
User otherUser = User.load( someOtherStore, otherId );

// Ideally users where the last login is older than one year 
using the networkStore
auto oldUsers = User.all( someOtherStore )
   .filterUsing!(User.lastLogin, (lastLogin) => lastLogin < ( now 
- 1.year ) );

// Select the first user that match userName and hashedPassword 
in the default store, with a challenge-response login
auto loginUser = User.all.
   .filterUsing!(User.name, (name) => name == userName )
   .filterUsing!(User.password, (pass) => hmac( pass, key ) == 
hashedPassword )
   .onlyOne();

Gianni Pisetta



 Jan 06 2016



prev sibling  parent reply Craig Dillabaugh <craig.dillabaugh gmail.com>  writes:


On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:

 The goal of this post is to measure the craziness of an idea to 
 embed a database engine into the D language ;)

 I think about a database engine which would meet my three main 
 requirements:
   - integrated with D (ranges)
   - ACID
   - fast

 Since the days when I was working on financing data SW I become 
 allergic to SQL. I though that NoSQL databases would fill the 
 bill. Unfortunately they didn't. And I want to have an ability 
 to write a code like this without too much effort:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);

 And other things like updating and deleting from db. I think 
 you get my point.

 So I started a PoC project based on SQLite design:
 https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

 The PoC code: 
 https://github.com/PiotrekDlang/AirLock/tree/master/src/database

 Can you please share your thoughts and experience on the topic? 
 Has anyone tried similar things?

 Piotrek


I've scanned this thread, but haven't seen if any 'decisions' 
have been more, or if it is just more of the usual back-and-forth 
with nothing being decided.  However, I did have one (Ok two) 
questions.

1. Is there a GSOC project in here somewhere?
2. Who would want to mentor such a thing?



 Feb 06 2016



  parent  Craig Dillabaugh <craig.dillabaugh gmail.com>  writes:


On Saturday, 6 February 2016 at 13:33:34 UTC, Craig Dillabaugh 
wrote:
clip

 I've scanned this thread, but haven't seen if any 'decisions' 
 have been more, or if it is just more of the usual


have been more => have been made



 Feb 06 2016