www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - What's going on with std.experimental.lexer?

reply "Brian Schott" <briancschott gmail.com> writes:
I've been looking at ways to optimize the D lexer's operation 
using SIMD instructions. I'm not yet sure if I'll need to change 
the lexer generator's API to do this. I'm going to wait until I 
have my proof-of-concept code and some benchmark results before 
asking for a voting thread or creating a pull request.
Jun 04 2014
next sibling parent "Philpax" <phillip philliplarkson.com> writes:
I've been meaning to mention this, but I use 
std.experimental.lexer in a code generation tool for our project; 
it's worked well so far, and I'd happily recommend its use. 
Looking forward to further updates.
Jun 05 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:

 I've been looking at ways to optimize the D lexer's operation 
 using SIMD instructions. I'm not yet sure if I'll need to 
 change the lexer generator's API to do this. I'm going to wait 
 until I have my proof-of-concept code and some benchmark 
 results before asking for a voting thread or creating a pull 
 request.


I still thing we should use it more like `std.staging` - once 
your updates are ready, go through review/voting and keep module 
in `std.experimental` for at least one DMD release before adding 
to Phobos core.

This also means relaxing API requirements a lot for initial 
inclusion.
Jun 05 2014
next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 05 Jun 2014 10:57:35 +0000
Dicebot via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:

 I've been looking at ways to optimize the D lexer's operation
 using SIMD instructions. I'm not yet sure if I'll need to
 change the lexer generator's API to do this. I'm going to wait
 until I have my proof-of-concept code and some benchmark
 results before asking for a voting thread or creating a pull
 request.


 I still thing we should use it more like `std.staging` - once
 your updates are ready, go through review/voting and keep module
 in `std.experimental` for at least one DMD release before adding
 to Phobos core.

 This also means relaxing API requirements a lot for initial
 inclusion.


Yeah. std.experimental really should be for stuff that's gone through the
review process and is essentially ready for inclusion in Phobos but needs more
real world usage to make sure that it really is fully ready to be merged into
std and then have its API frozen. It really doesn't make any sense to put
anything that's in more flux than that in there, because the release cycle is
just way too long for it.

Now, Brian's lexer might effectively be at that point already, I don't know.
It's been reviewed before, but I generally haven't had much time to look it
over for those reviews and don't know what state it's currently in. So, if its
API is close enough to final at this point, then it could probably be put in
std.experimental, but if it's still in a lot of flux, then that probably isn't
appropriate.

We do need to figure out what the official process is on this though so that
it's clear when it's appropriate for something to go in std.experimental,
otherwise we'll have folks who keep trying to put stuff in there which really
isn't ready yet, and we'll keep end up arguing over what can and can't go in
there.

- Jonathan M Davis
Jun 06 2014
prev sibling parent reply "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Thursday, 5 June 2014 at 10:57:37 UTC, Dicebot wrote:

 On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:

 I've been looking at ways to optimize the D lexer's operation 
 using SIMD instructions. I'm not yet sure if I'll need to 
 change the lexer generator's API to do this. I'm going to wait 
 until I have my proof-of-concept code and some benchmark 
 results before asking for a voting thread or creating a pull 
 request.


 I still thing we should use it more like `std.staging` - once 
 your updates are ready, go through review/voting and keep 
 module in `std.experimental` for at least one DMD release 
 before adding to Phobos core.

 This also means relaxing API requirements a lot for initial 
 inclusion.


I am more for "stdx", which is what some developers already use 
as package name for experimental stuff.
Jun 09 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/14, 2:15 PM, Dejan Lekic wrote:

 On Thursday, 5 June 2014 at 10:57:37 UTC, Dicebot wrote:

 On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:

 I've been looking at ways to optimize the D lexer's operation using
 SIMD instructions. I'm not yet sure if I'll need to change the lexer
 generator's API to do this. I'm going to wait until I have my
 proof-of-concept code and some benchmark results before asking for a
 voting thread or creating a pull request.


 I still thing we should use it more like `std.staging` - once your
 updates are ready, go through review/voting and keep module in
 `std.experimental` for at least one DMD release before adding to
 Phobos core.

 This also means relaxing API requirements a lot for initial inclusion.


 I am more for "stdx", which is what some developers already use as
 package name for experimental stuff.


The way I see it is instead of explaining "stdx stands for the 
experimental part of std" I'd rather use std.experimental which explains 
itself.

We added "immutable" after having explained to a million people 
"invariant in this context means immutable".


Andrei
Jun 09 2014
next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 09 Jun 2014 14:23:59 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:


 On 6/9/14, 2:15 PM, Dejan Lekic wrote:



 I am more for "stdx", which is what some developers already use as
 package name for experimental stuff.


 The way I see it is instead of explaining "stdx stands for the
 experimental part of std" I'd rather use std.experimental which
 explains itself.

 We added "immutable" after having explained to a million people
 "invariant in this context means immutable".


That and the fact that we'd like to avoid the javax disaster. std.experimental
_should_ be annoyingly long and obvious. We want people to use it, but it
needs to be clear that it's temporary and _experimental_ until it actually
ends up in std.

- Jonathan M Davis
Jun 09 2014
prev sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 June 2014 07:26, Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Mon, 09 Jun 2014 14:23:59 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:


 On 6/9/14, 2:15 PM, Dejan Lekic wrote:



 I am more for "stdx", which is what some developers already use as
 package name for experimental stuff.


 The way I see it is instead of explaining "stdx stands for the
 experimental part of std" I'd rather use std.experimental which
 explains itself.

 We added "immutable" after having explained to a million people
 "invariant in this context means immutable".


 That and the fact that we'd like to avoid the javax disaster. std.experimental
 _should_ be annoyingly long and obvious. We want people to use it, but it
 needs to be clear that it's temporary and _experimental_ until it actually
 ends up in std.


I agree all the way with std.experimental as the package name.  Though
I might throw in an alternative argument to stdx and instead promote
unsafe.* or std.unsafe.  ;-)
Jun 10 2014
parent reply David Gileadi <gileadis NSPMgmail.com> writes:
On 6/10/14, 3:57 AM, Iain Buclaw via Digitalmars-d wrote:

 I agree all the way with std.experimental as the package name.  Though
 I might throw in an alternative argument to stdx and instead promote
 unsafe.* or std.unsafe.  ;-)


The only issue I see with *.unsafe.* is that it sounds related to 
 safety. So far I haven't seen a better name than std.experimental.
Jun 10 2014
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 10 Jun 2014 10:31:59 -0400, David Gileadi <gileadis nspmgmail.com>  
wrote:


 On 6/10/14, 3:57 AM, Iain Buclaw via Digitalmars-d wrote:

 I agree all the way with std.experimental as the package name.  Though
 I might throw in an alternative argument to stdx and instead promote
 unsafe.* or std.unsafe.  ;-)


 The only issue I see with *.unsafe.* is that it sounds related to  
  safety. So far I haven't seen a better name than std.experimental.


If it weren't a keyword, volatile would be a very good package name for  
this.

-Steve
Jun 10 2014
prev sibling parent reply Jeremy Powers via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, Jun 9, 2014 at 2:15 PM, Dejan Lekic via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 I am more for "stdx", which is what some developers already use as package
 name for experimental stuff.


Please no.  See: javax

Spelling out 'experimental' is probably the best, for all those reasons
already stated.
Jun 09 2014
parent reply "Dejan Lekic" <dejan.lekic gmail.com> writes:
 Please no.  See: javax

 Spelling out 'experimental' is probably the best, for all those 
 reasons
 already stated.


What's wrong with javax?
Jun 13 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 13 June 2014 at 14:59:55 UTC, Dejan Lekic wrote:

 Please no.  See: javax

 Spelling out 'experimental' is probably the best, for all 
 those reasons
 already stated.


 What's wrong with javax?


The fact that it started as same experimental package but stuff 
there never got moved to main namespace or actually changed in 
breaking fashion. Because many people missed "experimental" part 
and started using it as normal standard library package.
Jun 13 2014
prev sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 13.06.2014 16:59, schrieb Dejan Lekic:

 Please no.  See: javax

 Spelling out 'experimental' is probably the best, for all those
 reasons
 already stated.


 What's wrong with javax?


experimental is 100% clear and simple to understand beeing evil

javax was interpreted as eXtendet or eXtra or whatever, so people have
no problem in using experimental stuff all the while
Jun 13 2014
prev sibling next sibling parent reply "Brian Schott" <briancschott gmail.com> writes:
On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:

 I've been looking at ways to optimize the D lexer's operation 
 using SIMD instructions. I'm not yet sure if I'll need to 
 change the lexer generator's API to do this. I'm going to wait 
 until I have my proof-of-concept code and some benchmark 
 results before asking for a voting thread or creating a pull 
 request.


Implementing some SIMD code just in the lexWhitespace function 
causes a drop in total lexing time of roughly 3.7%. This looks 
promising so far, so I'm going to implement similar code in 
lexStringLiteral, lexSlashStarComment, lexSlashSlashComment, and 
lexSlashPlusComment.

So far this only uses SSE2 instructions, so all 64-bit x86 
processors can benefit from this. (SSE4.2 would allow us to do 
more interesting things, but it's not as common)
Jun 05 2014
parent reply "Brian Schott" <briancschott gmail.com> writes:
On Friday, 6 June 2014 at 00:33:23 UTC, Brian Schott wrote:

 Implementing some SIMD code just in the lexWhitespace function 
 causes a drop in total lexing time of roughly 3.7%. This looks 
 promising so far, so I'm going to implement similar code in 
 lexStringLiteral, lexSlashStarComment, lexSlashSlashComment, 
 and lexSlashPlusComment.


Some moe numbers:

SIMD reduces execution time by 5.15% with DMD.
Compiling the non-SIMD code with GDC reduces execution time by 
42.39%.

So... There's that.
Jun 06 2014
next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 07.06.2014 01:50, schrieb Brian Schott:

 On Friday, 6 June 2014 at 00:33:23 UTC, Brian Schott wrote:

 Implementing some SIMD code just in the lexWhitespace function
 causes a drop in total lexing time of roughly 3.7%. This looks
 promising so far, so I'm going to implement similar code in
 lexStringLiteral, lexSlashStarComment, lexSlashSlashComment,
 and lexSlashPlusComment.


 Some moe numbers:

 SIMD reduces execution time by 5.15% with DMD.
 Compiling the non-SIMD code with GDC reduces execution time by
 42.39%.

 So... There's that.


thats why im always puzzled when people start to optimze algorithms 
based on DMD results - currently one should always compare any results 
before optimization with GDC/LDC
Jun 08 2014
parent Martin Nowak <code dawg.eu> writes:
On 06/08/2014 08:21 PM, dennis luehring wrote:

 thats why im always puzzled when people start to optimze algorithms
 based on DMD results - currently one should always compare any results
 before optimization with GDC/LDC


I second that, dmd results are easily misleading as you often optimize 
around backend deficiencies.
Jun 08 2014
prev sibling parent reply "Brian Schott" <briancschott gmail.com> writes:
On Friday, 6 June 2014 at 23:50:40 UTC, Brian Schott wrote:

 SIMD reduces execution time by 5.15% with DMD.
 Compiling the non-SIMD code with GDC reduces execution time by 
 42.39%.

 So... There's that.


Changing the code generator to output a set of if statements that 
implements a binary search did more-or-less nothing with the DMD 
timings, but brought GDC's lead up to 49%. (i.e. the GDC-compiled 
version executes in 51% of the time that the DMD-compiled version 
does)
Jun 09 2014
parent dennis luehring <dl.soluz gmx.net> writes:
Am 09.06.2014 22:21, schrieb Brian Schott:

 On Friday, 6 June 2014 at 23:50:40 UTC, Brian Schott wrote:

 SIMD reduces execution time by 5.15% with DMD.
 Compiling the non-SIMD code with GDC reduces execution time by
 42.39%.

 So... There's that.


 Changing the code generator to output a set of if statements that
 implements a binary search did more-or-less nothing with the DMD
 timings, but brought GDC's lead up to 49%. (i.e. the GDC-compiled
 version executes in 51% of the time that the DMD-compiled version
 does)


the LLVM optimizer is also very good (sometimes far better then the gcc 
one) - what are your LDC timings?
Jun 09 2014
prev sibling parent reply Tom Browder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, Jun 4, 2014 at 4:12 PM, Brian Schott via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 I've been looking at ways to optimize the D lexer's operation using SIMD


Brian, i just found the lexer code repo (and fixed the broken code
link on the wiki), but the review thread on the wiki looks very old.

The wiki page:

  http://wiki.dlang.org/Review_Queue

The review thread link:

  http://forum.dlang.org/post/aiipgaqyddjijpjiulfu forum.dlang.org

Thanks.

Best regards,

-Tom
Jun 13 2014
parent "Dicebot" <public dicebot.lv> writes:
On Friday, 13 June 2014 at 12:01:00 UTC, Tom Browder via 
Digitalmars-d wrote:

 On Wed, Jun 4, 2014 at 4:12 PM, Brian Schott via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 I've been looking at ways to optimize the D lexer's operation 
 using SIMD


 Brian, i just found the lexer code repo (and fixed the broken 
 code
 link on the wiki), but the review thread on the wiki looks very 
 old.

 The wiki page:

   http://wiki.dlang.org/Review_Queue

 The review thread link:

   
 http://forum.dlang.org/post/aiipgaqyddjijpjiulfu forum.dlang.org

 Thanks.

 Best regards,

 -Tom


Last formal review has happened quite some time ago, this is 
correct. Much has been done since then. Anyway, I am ready to 
proceed with it as soon as Brian is.
Jun 13 2014