www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Operator overloading or alternatives to expression templates

reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
I find the reasons for turining down my ER a bit moot.

[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

AFAIK expression templates are the primary choice tom implement SIMD and
matrix libraries.
And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
implementing a nice query language for ORMs.

D currently doesn't allow to override some operators like < <= > >= &&
|| !=.

At least the comparison operators are really limiting, e.g. it's not
possible to efficiently implement logical indexing.

vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

Also opCmp is less efficient to implement than opBinary!"<" for many
types. Generally any good implementation of an algorithm should only
require a less operator, not a full ordering opCmp.

The short circuit operators && and || have a special semantic and can't
be easily, but there is & and | so it's not really required.

Now expression templates make an awful DSL when being used to create a
HTML formatter, but when the original semantic of the operators is
preserved they are really powerful b/c the compiler already handles
typechecking, operator precedence, and captures variables.

Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
Sep 11 2015
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:

 Does anyone have a different idea how to make a nice query 
 language? db.get!Person.where!(p => p.age > 21 && p.name == 
 "Peter")


Don't the suggestions for DSLs generally revolve around using 
string mixins to use whatever syntax you want?

- Jonathan M Davis
Sep 11 2015
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 09/11/2015 09:40 PM, Martin Nowak wrote:

 I find the reasons for turining down my ER a bit moot.
 ...


+1.


 [Issue 14593 – operator overloading can't be used with expression
 templates](https://issues.dlang.org/show_bug.cgi?id=14593)

 AFAIK expression templates are the primary choice tom implement SIMD and
 matrix libraries.


OTOH, they are a hack.


 And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
 implementing a nice query language for ORMs.

 D currently doesn't allow to override some operators like < <= > >= &&
 || !=.

 At least the comparison operators are really limiting, e.g. it's not
 possible to efficiently implement logical indexing.

 vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

 Also opCmp is less efficient to implement than opBinary!"<" for many
 types. Generally any good implementation of an algorithm should only
 require a less operator, not a full ordering opCmp.
 ...


The rewrite a < b => a.opCmp(b)<0 is usually wasteful, but both methods 
can be the most efficient choice depending on the application. Two calls 
to opBinary!"<" will usually not be more efficient than one call to 
"opCmp".

I.e. the precedence should be the other way: try a.opBinary!"<"(b) first 
and then fall back to a.opCmp(b)<0. (Maybe it would also make sense to 
automatically provide opCmp if missing when opBinary!"<" has been 
implemented, such that generic code can use opCmp for all comparable 
types. Built-in comparable types should also provide opCmp.)


 The short circuit operators && and || have a special semantic and can't
 be easily, but there is & and | so it's not really required.
 ...


There's the 'lazy' keyword.


 ...

 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


You could give up on operator syntax.
Sep 11 2015
parent "Martin Nowak" <code dawg.eu> writes:
On Friday, 11 September 2015 at 23:19:54 UTC, Timon Gehr wrote:

 Does anyone have a different idea how to make a nice query 
 language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


 You could give up on operator syntax.


That's what I did in the prototype, p.age.gt(21), but it's 
somewhat ugly.
Sep 12 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/11/2015 03:40 PM, Martin Nowak wrote:

 I find the reasons for turining down my ER a bit moot.

 [Issue 14593 – operator overloading can't be used with expression
 templates](https://issues.dlang.org/show_bug.cgi?id=14593)

 AFAIK expression templates are the primary choice tom implement SIMD and
 matrix libraries.
 And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
 implementing a nice query language for ORMs.


Expression templates are interesting, but from experience with them in 
C++ they're more trouble than they're worth. They haven't made much 
inroads in C++ outside exotic libraries because (a) they have odd and 
random limitations and corner cases and (b) they have really byzantine 
failure modes. Just look at any non-toy-example C++ use of ETs - it's 
completely bizarre.

I'd say if a language wants to support expression templates properly, 
there's a lot of careful design to get into it, way beyond intercepting 



 D currently doesn't allow to override some operators like < <= > >= &&
 || !=.

 At least the comparison operators are really limiting, e.g. it's not
 possible to efficiently implement logical indexing.

 vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

 Also opCmp is less efficient to implement than opBinary!"<" for many
 types. Generally any good implementation of an algorithm should only
 require a less operator, not a full ordering opCmp.

 The short circuit operators && and || have a special semantic and can't
 be easily, but there is & and | so it's not really required.

 Now expression templates make an awful DSL when being used to create a
 HTML formatter, but when the original semantic of the operators is
 preserved they are really powerful b/c the compiler already handles
 typechecking, operator precedence, and captures variables.

 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


There's two canonical ways to do that.

1. Use lambdas, which seem to already do what you want:

db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

The way this'd go, the db.get!Person() call returns an input range of 
Person. Presumably introspection is being used to bind fields in the 
query such as "age" and "name" to static field names in struct Person. 
Then good old std.algorithm.filter takes care of the rest.

2. If you want to embed real SQL into D, use string-based DSLs.


Andrei
Sep 11 2015
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Sep 11, 2015 at 07:47:42PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 09/11/2015 03:40 PM, Martin Nowak wrote:

I find the reasons for turining down my ER a bit moot.

[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

AFAIK expression templates are the primary choice tom implement SIMD
and matrix libraries.
And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
implementing a nice query language for ORMs.


 
 Expression templates are interesting, but from experience with them in
 C++ they're more trouble than they're worth. They haven't made much
 inroads in C++ outside exotic libraries because (a) they have odd and
 random limitations and corner cases and (b) they have really byzantine
 failure modes. Just look at any non-toy-example C++ use of ETs - it's
 completely bizarre.


A number of years ago I wrote a computational application using Blitz++,
an ET-based matrix / multidimensional array C++ library, and I have to
say that the experience was mostly pleasant.  However, it did push the
limits of operator overloading abuse, such as:

	Array<2,int> matrix = Array<2,int>(4, 4);
	matrix = 1, 0, 0, 0,
	         0, 1, 0, 0,
		 0, 0, 1, 0,
		 0, 0, 0, 1;

It also had unexpected reference/copying semantics (operator=() copies
by reference, but `=` in a variable declarationi copies by value), as
well as the occasional wrinkle when assigning the result of an
expression template to a variable (sometimes you have to explicitly call
a function to turn it into something storable in a variable).

The error messages however, were completely inscrutable, just as you
said. They generally begin anywhere from 15-16 lines of compiler output
per error, and only grows from there. Needless to say, I did not have
the patience (nor persistence!) to decipher those error messages; most
of my efforts lay in copying textbook examples from the documentation
and modifying them piece by piece, checking their compilability at every
step, until they matched what I ultimately wanted. Writing anything
complex directly was an invitation to be faced with an incomprehensible
screen-filling error message (often more than one), and endless hours of
randomly modifying random bits of syntax in hopes that the error will
somehow, magically, go away.



 I'd say if a language wants to support expression templates properly,
 there's a lot of careful design to get into it, way beyond

 for that.


Pluggable syntax modules seem like an attractive idea, though. But
probably outside the scope of D2 at present.


[...]

Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


 
 There's two canonical ways to do that.
 
 1. Use lambdas, which seem to already do what you want:
 
 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")
 
 The way this'd go, the db.get!Person() call returns an input range of
 Person. Presumably introspection is being used to bind fields in the
 query such as "age" and "name" to static field names in struct Person.
 Then good old std.algorithm.filter takes care of the rest.
 
 2. If you want to embed real SQL into D, use string-based DSLs.


[...]

Yeah, string-based DSLs make more sense in D: avoid complicating the
core language, thorny issues surrounding operator overloading (and the
abuse thereof), and also allow fully-free syntax of your choice.  With
CTFE, you can parse just about any DSL with any syntax at compile-time,
thus incur none of the performance issues that runtime string-based DSLs
would have.


T

-- 
The only difference between male factor and malefactor is just a little
emptiness inside.
Sep 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/11/2015 08:03 PM, H. S. Teoh via Digitalmars-d wrote:

 Needless to say, I did not have
 the patience (nor persistence!) to decipher those error messages; most
 of my efforts lay in copying textbook examples from the documentation
 and modifying them piece by piece, checking their compilability at every
 step, until they matched what I ultimately wanted. Writing anything
 complex directly was an invitation to be faced with an incomprehensible
 screen-filling error message (often more than one), and endless hours of
 randomly modifying random bits of syntax in hopes that the error will
 somehow, magically, go away.


All the more ironic considering Blitz++ was intended for scientists who 
were not supposed to be programmer experts. -- Andrei
Sep 11 2015
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 12 September 2015 at 00:27:45 UTC, Andrei 
Alexandrescu wrote:

 On 09/11/2015 08:03 PM, H. S. Teoh via Digitalmars-d wrote:

 Needless to say, I did not have
 the patience (nor persistence!) to decipher those error 
 messages; most
 of my efforts lay in copying textbook examples from the 
 documentation
 and modifying them piece by piece, checking their 
 compilability at every
 step, until they matched what I ultimately wanted. Writing 
 anything
 complex directly was an invitation to be faced with an 
 incomprehensible
 screen-filling error message (often more than one), and 
 endless hours of
 randomly modifying random bits of syntax in hopes that the 
 error will
 somehow, magically, go away.


 All the more ironic considering Blitz++ was intended for 
 scientists who were not supposed to be programmer experts. -- 
 Andrei


I do think it says more about C++ than it does about expression 
templates.
Sep 11 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-12 01:47, Andrei Alexandrescu wrote:


 There's two canonical ways to do that.

 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input range of
 Person. Presumably introspection is being used to bind fields in the
 query such as "age" and "name" to static field names in struct Person.
 Then good old std.algorithm.filter takes care of the rest.


A database is really good at what it does. Therefore one wants the 
filter/where statement to be executed by the database, not the 
application code.

It would also be quite wasteful to have to create an instance of Person 
to make the comparison then realizing it doesn't fulfill the condition 
and throw it away.

I'm not sure how database protocols work but one definitely don't want 
to query the database for each row or return all the rows from the database.


 2. If you want to embed real SQL into D, use string-based DSLs.


The point is to _not_ use raw SQL.

-- 
/Jacob Carlborg
Sep 12 2015
prev sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu 
wrote:

 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input 
 range of Person. Presumably introspection is being used to bind 
 fields in the query such as "age" and "name" to static field 
 names in struct Person. Then good old std.algorithm.filter 
 takes care of the rest.


I'm instantiating the lambda with a fake p to capture the 
expression so I can translate it to whatever SQL, MongoDB, 
columnar db is used.


 2. If you want to embed real SQL into D, use string-based DSLs.


Strings don't capture context, aren't typechecked, and require a 
complex CTFE parser.

db.get!Person.where!"age > 21 & name = ?"(name);
Sep 12 2015
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 12-Sep-2015 23:08, Martin Nowak wrote:

 On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu wrote:

 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input range of
 Person. Presumably introspection is being used to bind fields in the
 query such as "age" and "name" to static field names in struct Person.
 Then good old std.algorithm.filter takes care of the rest.


 I'm instantiating the lambda with a fake p to capture the expression so
 I can translate it to whatever SQL, MongoDB, columnar db is used.


 2. If you want to embed real SQL into D, use string-based DSLs.


 Strings don't capture context, aren't typechecked, and require a complex
 CTFE parser.

 db.get!Person.where!"age > 21 & name = ?"(name);


What if we add generic string interpolation a-la:

s"$age > 21 && name = ${someobj.field}"

would translate to:
AliasSeq!("",age," > 21 && name = ", someobj.field, "");

Thoughts? I think it was proposed before by Timothy Cour.

-- 
Dmitry Olshansky
Sep 12 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/12/2015 04:08 PM, Martin Nowak wrote:

 On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu wrote:

 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input range of
 Person. Presumably introspection is being used to bind fields in the
 query such as "age" and "name" to static field names in struct Person.
 Then good old std.algorithm.filter takes care of the rest.


 I'm instantiating the lambda with a fake p to capture the expression so
 I can translate it to whatever SQL, MongoDB, columnar db is used.


Yah, understood. Problem here is the approach is bound to run into walls 
at every few steps. Say you fix the comparisons to work. Then you have 
operators such as LIKE that are necessary yet not representable in D. So 
more tricks are in order. This is what I've seen with ET in C++ - an 
endless collection of tricks to achieve modest outcomes at enormous costs.


 2. If you want to embed real SQL into D, use string-based DSLs.


 Strings don't capture context, aren't typechecked, and require a complex
 CTFE parser.

 db.get!Person.where!"age > 21 & name = ?"(name);


Understood as well. I figure from the example you have binding in mind, 
which is good. At some point you need to bite the bullet and write a 
parser for the relational query language you want there.


Andrei
Sep 12 2015
next sibling parent deadalnix <deadalnix gmail.com> writes:
On Sunday, 13 September 2015 at 03:03:44 UTC, Andrei Alexandrescu 
wrote:

 On 09/12/2015 04:08 PM, Martin Nowak wrote:

 On Friday, 11 September 2015 at 23:47:42 UTC, Andrei 
 Alexandrescu wrote:

 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input 
 range of
 Person. Presumably introspection is being used to bind fields 
 in the
 query such as "age" and "name" to static field names in 
 struct Person.
 Then good old std.algorithm.filter takes care of the rest.


 I'm instantiating the lambda with a fake p to capture the 
 expression so
 I can translate it to whatever SQL, MongoDB, columnar db is 
 used.


 Yah, understood. Problem here is the approach is bound to run 
 into walls at every few steps. Say you fix the comparisons to 
 work. Then you have operators such as LIKE that are necessary 
 yet not representable in D. So more tricks are in order. This 
 is what I've seen with ET in C++ - an endless collection of 
 tricks to achieve modest outcomes at enormous costs.



such mechnism and users seems very happy with it.


 2. If you want to embed real SQL into D, use string-based 
 DSLs.


 Strings don't capture context, aren't typechecked, and require 
 a complex
 CTFE parser.

 db.get!Person.where!"age > 21 & name = ?"(name);


 Understood as well. I figure from the example you have binding 
 in mind, which is good. At some point you need to bite the 
 bullet and write a parser for the relational query language you 
 want there.


That sounds like a bigger problem than whatever you mentioned 
before.
Sep 12 2015
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 13 September 2015 at 03:03:44 UTC, Andrei Alexandrescu 
wrote:

 Yah, understood. Problem here is the approach is bound to run 
 into walls at every few steps. Say you fix the comparisons to 
 work. Then you have operators such as LIKE that are necessary


like(Person.Name,"peter%")

Person.Name == pattern("peter%")


 yet not representable in D. So more tricks are in order. This 
 is what I've seen with ET in C++ - an endless collection of 
 tricks to achieve modest outcomes at enormous costs.


FWIW, Guido wrote a similar query language to what Martin 
proposes for Google App Engine called NDB:

https://cloud.google.com/appengine/docs/python/ndb/queries

They already had a string query language called GQL:

https://cloud.google.com/appengine/docs/python/datastore/gqlreference

I never use GQL, because the NDB solution is superior in every 
way.

GQL is only for the REPL.


 2. If you want to embed real SQL into D, use string-based 
 DSLs.


 Strings don't capture context, aren't typechecked, and require 
 a complex
 CTFE parser.

 db.get!Person.where!"age > 21 & name = ?"(name);


 Understood as well. I figure from the example you have binding 
 in mind, which is good. At some point you need to bite the 
 bullet and write a parser for the relational query language you 
 want there.


No. D should stop abusing strings. Strings should never contain 
references to type names. It makes it impossible to transpile D 
in the general case.

Walter is unlikely to change his mind on opCmp, because he has 
put a lot of emphasis on that restriction since D1, but I think 
he overreacted on some C++ libraries being less than user 
friendly.

The only thing you achieve is that people will create even uglier 
hacks to get what they want, like using a thread local global AST 
root pointer and build the AST as hidden side effects at runtime 
instead.
Sep 12 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/13/2015 05:03 AM, Andrei Alexandrescu wrote:

 Yah, understood. Problem here is the approach is bound to run into walls
 at every few steps. Say you fix the comparisons to work. Then you have
 operators such as LIKE that are necessary yet not representable in D. So
 more tricks are in order. This is what I've seen with ET in C++ - an
 endless collection of tricks to achieve modest outcomes at enormous costs.


But this is not an argument to rule out `opBinary!"<"`.

To summarize the arguments.

- logical indexing x[x < 20]

  e.g. opBinary!"<" returns a bit mask to select entries of a large vector

- faster comparison

  struct Foo
  {
      size_t id;
      int opCmp(Foo rhs)
      {
          if (id < rhs.id) return -1;
          if (id == rhs.id) return 0;
          else return 1;
      }
      bool opBinary(string s:"<")(Foo rhs)
      {
          return id < rhs.id;
      }
  }

  Sorting a million Foos w/ random ids is 37.5% slower with opCmp.

  foos.sort!((a, b) => a.opBinary!"<"(b))(); // 104ms
  foos.sort!((a, b) => a < b)(); // 143ms

- expression templates

  I'm well aware of the limitations, but still think it will work out
nicely for an ORM b/c there is precedence in other language, e.g. Rails'
(ActiveRecord) query syntax.

- language regularization

  It's surprising to find these "arbitrary" language limitations.
  The non-predictability of what's possible has always been a huge issue
for me with C++, i.e. you come up with an idea, spend 4 hours
implementing it only to find out that the small detail x isn't feasible.
Sep 13 2015
next sibling parent Jack Stouffer <jack jackstouffer.com> writes:
On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:

 - logical indexing x[x < 20]

   e.g. opBinary!"<" returns a bit mask to select entries of a 
 large vector


Oh yes please! This is one of the more powerful features of numpy.
Sep 13 2015
prev sibling next sibling parent reply Daniel N <ufo orbiting.us> writes:
On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:

   struct Foo
   {
       size_t id;
       int opCmp(Foo rhs)
       {
           if (id < rhs.id) return -1;
           if (id == rhs.id) return 0;
           else return 1;
       }
       bool opBinary(string s:"<")(Foo rhs)
       {
           return id < rhs.id;
       }
   }

   Sorting a million Foos w/ random ids is 37.5% slower with 
 opCmp.


Could you try this?

int opCmp(Foo rhs)
{
   return (id > rhs.id) - (id < rhs.id);
}
Sep 13 2015
next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/13/2015 07:16 PM, Daniel N wrote:

 
 Could you try this?
 
 int opCmp(Foo rhs)
 {
   return (id > rhs.id) - (id < rhs.id);
 }


That's not the point, opCmp requires twice as many comparisons as needed
for <. If they are more expansive, e.g. string comparison, your trick
won't work.
Sep 13 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/13/2015 03:06 PM, Martin Nowak wrote:

 On 09/13/2015 07:16 PM, Daniel N wrote:

 Could you try this?

 int opCmp(Foo rhs)
 {
    return (id > rhs.id) - (id < rhs.id);
 }


 That's not the point, opCmp requires twice as many comparisons as needed
 for <. If they are more expansive, e.g. string comparison, your trick
 won't work.


Well for string comparisons, the added cost (if not optimized away) is 
constant. Overall it seems to me that D is to be appreciated for 
requiring only one operator for all ordering comparisons (it causes a 
lot of noise in C++). Reducing comparisons is the job of the optimizer. 
Expression templates are a possible loss, but of a questionable trick. 
-- Andrei
Sep 14 2015
prev sibling next sibling parent reply burjui <bytefu gmail.com> writes:
On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:

 int opCmp(Foo rhs)
 {
   return (id > rhs.id) - (id < rhs.id);
 }


IMO, subtracting boolean values is bad code style, it's better to 
be explicit about your intention:

 (id > rhs.id ? 1 : 0) - (id < rhs.id ? 1 : 0)
Sep 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2015 07:32 AM, burjui wrote:

 On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:

 int opCmp(Foo rhs)
 {
   return (id > rhs.id) - (id < rhs.id);
 }


 IMO, subtracting boolean values is bad code style


I love it! -- Andrei
Sep 14 2015
parent deadalnix <deadalnix gmail.com> writes:
On Monday, 14 September 2015 at 18:24:00 UTC, Andrei Alexandrescu 
wrote:

 On 09/14/2015 07:32 AM, burjui wrote:

 On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:

 int opCmp(Foo rhs)
 {
   return (id > rhs.id) - (id < rhs.id);
 }


 IMO, subtracting boolean values is bad code style


 I love it! -- Andrei


Beside loving, it, this has good ILP, no branches, and have nice 
codegen on most CPUs.

I was surprised to find that style in jemalloc, and compared to 
various alternatives. The only way I was able to get better is by 
ignoring (and by extension getting invalid results) overflow.
Sep 14 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/13/2015 01:16 PM, Daniel N wrote:

 On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:

   struct Foo
   {
       size_t id;
       int opCmp(Foo rhs)
       {
           if (id < rhs.id) return -1;
           if (id == rhs.id) return 0;
           else return 1;
       }
       bool opBinary(string s:"<")(Foo rhs)
       {
           return id < rhs.id;
       }
   }

   Sorting a million Foos w/ random ids is 37.5% slower with opCmp.


 Could you try this?

 int opCmp(Foo rhs)
 {
    return (id > rhs.id) - (id < rhs.id);
 }


Apparently that does well with gcc: 
http://stackoverflow.com/questions/10996418/efficient-integer-compare-function 
-- Andrei
Sep 14 2015
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09/14/2015 08:18 PM, Andrei Alexandrescu wrote:

 On 09/13/2015 01:16 PM, Daniel N wrote:

 On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:

   struct Foo
   {
       size_t id;
       int opCmp(Foo rhs)
       {
           if (id < rhs.id) return -1;
           if (id == rhs.id) return 0;
           else return 1;
       }
       bool opBinary(string s:"<")(Foo rhs)
       {
           return id < rhs.id;
       }
   }

   Sorting a million Foos w/ random ids is 37.5% slower with opCmp.


 Could you try this?

 int opCmp(Foo rhs)
 {
    return (id > rhs.id) - (id < rhs.id);
 }


 Apparently that does well with gcc:
 http://stackoverflow.com/questions/10996418/efficient-integer-compare-function
 -- Andrei


The first version was actually slightly faster when I tested it with 
std::sort. (i.e. if only "<" is used). I guess it is optimized to a 
simple integer comparison after inlining. In any case, any performance 
differences for such simple equivalent functions are random quirks of 
the respective back-ends.
Sep 14 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/13/2015 10:06 AM, Martin Nowak wrote:

 On 09/13/2015 05:03 AM, Andrei Alexandrescu wrote:

 Yah, understood. Problem here is the approach is bound to run into walls
 at every few steps. Say you fix the comparisons to work. Then you have
 operators such as LIKE that are necessary yet not representable in D. So
 more tricks are in order. This is what I've seen with ET in C++ - an
 endless collection of tricks to achieve modest outcomes at enormous costs.


 But this is not an argument to rule out `opBinary!"<"`.


Agreed.


 To summarize the arguments.

 - logical indexing x[x < 20]

    e.g. opBinary!"<" returns a bit mask to select entries of a large vector


Good one. But niche, can be solved with a function, etc.


 - faster comparison

    struct Foo
    {
        size_t id;
        int opCmp(Foo rhs)
        {
            if (id < rhs.id) return -1;
            if (id == rhs.id) return 0;
            else return 1;
        }
        bool opBinary(string s:"<")(Foo rhs)
        {
            return id < rhs.id;
        }
    }

    Sorting a million Foos w/ random ids is 37.5% slower with opCmp.

    foos.sort!((a, b) => a.opBinary!"<"(b))(); // 104ms
    foos.sort!((a, b) => a < b)(); // 143ms


I think this is a good candidate for a peephole optimization.


 - expression templates

    I'm well aware of the limitations, but still think it will work out
 nicely for an ORM b/c there is precedence in other language, e.g. Rails'
 (ActiveRecord) query syntax.


Fine, but let's not forget the ET experience in D will be a lot closer 
to that in C++ than that in Rails. And that doesn't bode well.


 - language regularization

    It's surprising to find these "arbitrary" language limitations.
    The non-predictability of what's possible has always been a huge issue
 for me with C++, i.e. you come up with an idea, spend 4 hours
 implementing it only to find out that the small detail x isn't feasible.


This is the case in all powerful languages. Martin Odersky told me 
there's a constant problem with Scala having such a powerful type 
systems, the envelope of what can and cannot be done with it is really 
fuzzy and frustrates its power users.


Andrei
Sep 14 2015
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:

 On 09/13/2015 10:06 AM, Martin Nowak wrote:

 ...
 - language regularization

    It's surprising to find these "arbitrary" language limitations.
    The non-predictability of what's possible has always been a huge issue
 for me with C++, i.e. you come up with an idea, spend 4 hours
 implementing it only to find out that the small detail x isn't feasible.


 This is the case in all powerful languages.


That's an overgeneralization.

Furthermore, having arbitrary designed-in irregularities is not 
comparable to implementing a system whose emergent behavior is not 
understood sufficiently well. (D is guilty of both, but the former is 
much easier to fix than the latter.)


 Martin Odersky told me there's a constant problem with Scala ...


Both C++ and Scala have accidentally Turing-complete type systems.
Sep 14 2015
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 14 September 2015 at 19:35:39 UTC, Timon Gehr wrote:

 Furthermore, having arbitrary designed-in irregularities is not 
 comparable to implementing a system whose emergent behavior is 
 not understood sufficiently well. (D is guilty of both, but the 
 former is much easier to fix than the latter.)


Well, D1 was a restrictive design that made C++ style programming 
simpler and less error prone (sans compiler bugs). D2 is a 
flexible and complicated package and thereby those restrictions 
cause disharmony. It is basically too incompatible design 
philosophies that are brought together.

Like in D1 it would make sense to say "the language provide the 
essentials, don't design your own pointer types", whereas in D2 
it makes sense to say "all features should be library based". I'm 
a bit miffed that it is difficult to built proper smart pointer 
types in D2.


 Both C++ and Scala have accidentally Turing-complete type 
 systems.


Yes, not sure how smart that is in terms of maintainable code. 
There is a reaction against overdone type systems that sometimes 
can be more of burden as programs age/grow (e.g. C++/Boost), so 
now we see gradual typing, deliberately unsound type-systems etc.
Sep 14 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2015 03:35 PM, Timon Gehr wrote:

 On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:

 On 09/13/2015 10:06 AM, Martin Nowak wrote:

 ...
 - language regularization

    It's surprising to find these "arbitrary" language limitations.
    The non-predictability of what's possible has always been a huge
 issue
 for me with C++, i.e. you come up with an idea, spend 4 hours
 implementing it only to find out that the small detail x isn't feasible.


 This is the case in all powerful languages.


 That's an overgeneralization.


Aren't they all :o).


 Furthermore, having arbitrary designed-in irregularities is not
 comparable to implementing a system whose emergent behavior is not
 understood sufficiently well. (D is guilty of both, but the former is
 much easier to fix than the latter.)


 Martin Odersky told me there's a constant problem with Scala ...


 Both C++ and Scala have accidentally Turing-complete type systems.


I think what I'm trying to say is "I'm trying to do this very advanced 
thing and the language support is insufficient" is the kind of argument 
that needs to be made and taken with caution.


Andrei
Sep 15 2015
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09/15/2015 06:53 PM, Andrei Alexandrescu wrote:

 On 09/14/2015 03:35 PM, Timon Gehr wrote:

 On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:

 On 09/13/2015 10:06 AM, Martin Nowak wrote:

 ...
 - language regularization

    It's surprising to find these "arbitrary" language limitations.
    The non-predictability of what's possible has always been a huge
 issue
 for me with C++, i.e. you come up with an idea, spend 4 hours
 implementing it only to find out that the small detail x isn't
 feasible.


 This is the case in all powerful languages.


 That's an overgeneralization.


 Aren't they all :o).
 ...


I guess we have now established that there are at least two of them. :P


 Furthermore, having arbitrary designed-in irregularities is not
 comparable to implementing a system whose emergent behavior is not
 understood sufficiently well. (D is guilty of both, but the former is
 much easier to fix than the latter.)


 Martin Odersky told me there's a constant problem with Scala ...


 Both C++ and Scala have accidentally Turing-complete type systems.


 I think what I'm trying to say is "I'm trying to do this very advanced
 thing and the language support is insufficient" is the kind of argument
 that needs to be made and taken with caution.
 ...


This is certainly the case, and I believe the necessary care has been 
applied. The language support required isn't actually very advanced, nor 
are all usage scenarios. Arbitrary restrictions are usually design mistakes.
Sep 17 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-11 21:40, Martin Nowak wrote:

 I find the reasons for turining down my ER a bit moot.

 [Issue 14593 – operator overloading can't be used with expression
 templates](https://issues.dlang.org/show_bug.cgi?id=14593)

 AFAIK expression templates are the primary choice tom implement SIMD and
 matrix libraries.
 And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
 implementing a nice query language for ORMs.


I would really like this.


 The short circuit operators && and || have a special semantic and can't
 be easily, but there is & and | so it's not really required.


Not sure if this would be a problem in D but there's a library in Ruby 
that does exactly this. Due to the operator precedence you end up 
needing to wrap everything in parentheses:

Person.where{ |p| (p.age > 21) & (p.name == "Peter") | (p.name == "Foo") }


 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


AST macros ;)

-- 
/Jacob Carlborg
Sep 12 2015
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:

 Does anyone have a different idea how to make a nice query 
 language? db.get!Person.where!(p => p.age > 21 && p.name == 
 "Peter")


Do it at runtime?
Sep 12 2015
prev sibling next sibling parent reply Bahman Movaqar <Bahman BahmanM.com> writes:
On Fri, 11 Sep 2015 21:40:58 +0200, Martin Nowak 
<code+news.digitalmars dawg.eu> wrote:

 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


Django's approach is, IMO,  the cleverest and least magical one while 
keeping it expressive and efficient:

     Person.objects.filter(age__gt=21, name__eq='peter')

The input of `filter` is keyword argument style. So, filter can 
access the dynamic argument names, in addition to their values. This 
makes it easy to understand user's intent without the need to 
overloaded operators or black magic (like repeated reflections 
Hibernate uses) and the syntax stays clean from user's perspective as 
well.

-- 
--
Bahman Movaqar
Sep 12 2015
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 12 September 2015 at 22:38:53 UTC, Bahman Movaqar 
wrote:

 Django's approach is, IMO,  the cleverest and least magical one 
 while keeping it expressive and efficient:

     Person.objects.filter(age__gt=21, name__eq='peter')


As the main examples in this thread are for ORMs, I think the 
best case for more powerful operator overloading is the very 
popular Python library SQLAlchmey. SQLAlchmey is the reason that 
any project of mine that requires a database is going to be in 
Python.

Here is a real code sample from one of my projects from my day 
job:

shipped = session.query(
         func.sum(
             InvoiceHistoryDetail.QuantityShipped * 
InvoiceHistoryDetail.UnitPrice
         ).label("sum")
).join(
         InvoiceHistoryHeader,
         InvoiceHistoryHeader.InvoiceNo == 
InvoiceHistoryDetail.InvoiceNo
).join(
         Customer,
         and_(
             InvoiceHistoryHeader.ARDivisionNo == 
Customer.ARDivisionNo,
             InvoiceHistoryHeader.CustomerNo == Customer.CustomerNo
         )
).filter(
         InvoiceHistoryHeader.ShipDate >= 
fiscal_month.PeriodStartingDate,
         InvoiceHistoryHeader.ShipDate == datetime.date.today() - 
datetime.timedelta(days=1),
         InvoiceHistoryHeader.SalesOrderNo != None,
         Customer.UDF_PARTICIPATION.in_(participation)
).one()

Note how complex operator overloading is used to make very 
readable JOIN and WHERE clauses. One other cool thing to note is 
the func function, which is a generator function that allows 
expressing any SQL function in Python by translating the name 
after the dot to the function name is the final SQL. So 
func.whatever(model.field) would become WHATEVER(`model.field`) 
in the SQL.

I know that this effect is much harder to create in a explicitly 
and strongly typed language, but I just wanted to show a real 
world example of how these could be used to great effect.
Sep 13 2015
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/13/2015 03:15 PM, Jack Stouffer wrote:

 
 I know that this effect is much harder to create in a explicitly and
 strongly typed language, but I just wanted to show a real world example
 of how these could be used to great effect.


But it is doable in D, and even better it's possible to optimize the
queries b/c we can know in advance what fields are used.
http://dpaste.dzfl.pl/cd375ac594cf
Sep 13 2015
prev sibling next sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 11.09.2015 um 21:40 schrieb Martin Nowak:

 I find the reasons for turining down my ER a bit moot.

 [Issue 14593 – operator overloading can't be used with expression
 templates](https://issues.dlang.org/show_bug.cgi?id=14593)

 (...)


The missing support for overloading the individual relational operators 
is something that has often turned out very limiting to me in the past, 
too. The opCmp approach is good as a default, but explicitly disallowing 
implementation of the individual operators seems arbitrary.


 Now expression templates make an awful DSL when being used to create a
 HTML formatter, but when the original semantic of the operators is
 preserved they are really powerful b/c the compiler already handles
 typechecking, operator precedence, and captures variables.

 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")


I had played around with some ideas for a similar project, but didn't 
find a really satisfying solution:
https://github.com/rejectedsoftware/dotter/blob/11ec72325e76c3329a58545526940c1df5328a2d/source/dotter/orm.d#L320
Sep 13 2015
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/13/2015 11:00 AM, Sönke Ludwig wrote:

 I had played around with some ideas for a similar project, but didn't
 find a really satisfying solution:
 https://github.com/rejectedsoftware/dotter/blob/11ec72325e76c3329a58545526940c1df5328a2d/source/dotter/orm.d#L320


Yeah, that doesn't look too nice.

I think db.get!Author.where!(a => a.books.count > 10) is very powerful
b/c you can easily use relations for querying.

struct Author
{
   hasMany // relation in SQL or nested in MongoDB
  Book[] books;
}
Sep 13 2015
prev sibling next sibling parent Dicebot <public dicebot.lv> writes:
I am pretty sure I'd prefer hygienic string DSL (with explicit 
alias list for used context symbols, like in Diet) over LINQ-like 
magic.
Sep 13 2015
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:

 Does anyone have a different idea how to make a nice query 
 language? db.get!Person.where!(p => p.age > 21 && p.name == 
 "Peter")


In our last project we took the following approach:

`auto q = query.builder!Person.age!">"(20).name("Peter");`

In the query-builders there was a lot of ugly concatenating sql 
stuff. But there was just one guy writing that, everybody else 
was using the high-level functions.

It allows for a lot of checking, not only on types but also on 
logic (e.g. age!">"(20).age!"<"(10) was caught at runtime). Plus, 
very unittestable.

It worked so well that once the query-builders were written, all 
the constraints, sorting and ordering parameters could be passed 
straight from the http api down to the query-builders.

Still, you would have to write the query-builders for each Object 
you want query, although that is something you could do in a DSL.
Sep 14 2015
next sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/14/2015 03:47 PM, Sebastiaan Koppe wrote:

 
 In our last project we took the following approach:
 
 `auto q = query.builder!Person.age!">"(20).name("Peter");`


Interesting idea.
Sep 14 2015
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 14 September 2015 at 13:47:10 UTC, Sebastiaan Koppe 
wrote:

 `auto q = query.builder!Person.age!">"(20).name("Peter");`


I confess that I'm not really paying attention to this thread, 
but I can't help but think plain old literal: `Person.age > 20 && 
Person.name = 'Peter'` is nicer. You can still CT check that by 
parsing the string if you want.
Sep 14 2015
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 14 September 2015 at 18:17:05 UTC, Adam D. Ruppe wrote:

 On Monday, 14 September 2015 at 13:47:10 UTC, Sebastiaan Koppe 
 wrote:

 `auto q = query.builder!Person.age!">"(20).name("Peter");`


 I confess that I'm not really paying attention to this thread, 
 but I can't help but think plain old literal: `Person.age > 20 
 && Person.name = 'Peter'` is nicer. You can still CT check that 
 by parsing the string if you want.


It is definitely nicer, but this is also a contrived use-case. 
What happens when you have a couple of joins, how would you write 
that? Or inner queries?

Suppose we wanted to fetch all users who didn't place an order in 
the last year, it would be as simple as calling 
`.lastOrderDate!"<"(lastYear)`. It would do the join with the 
order table on the appropriate column.

Granted, someone has to write `lastOrderDate()`.
Sep 14 2015
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:

 AFAIK expression templates are the primary choice tom implement 
 SIMD and
 matrix libraries.
 And I still have [this 
 idea](http://dpaste.dzfl.pl/cd375ac594cf) of
 implementing a nice query language for ORMs.


While expression templates are used in many matrix libraries, 
there has also been some move away from them. Blaze uses 
so-called smart expression templates because of what they view as 
performance limitations of expression templates. I think the 
general idea is to reduce the creation of temporaries and 
ordering the calculations so as to minimize the size of the 
problem (like in the multiplication A*B*c, doing A*(B*c) instead 
of (A*B)*c)
http://arxiv.org/pdf/1104.1729.pdf


 At least the comparison operators are really limiting, e.g. 
 it's not possible to efficiently implement logical indexing.

 vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]


I loves the logical indexing.
Sep 14 2015