www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - parseJSON bug

reply "khurshid" <khurshid.normuradov gmail.com> writes:
std.json  parseJSON has a bug,

//                         leading whitespaces
auto json = parseJSON("1                                  .000");

-- should throws, but it's success parsed as `1.000` .


Khurshid...
Aug 08 2013
next sibling parent reply "khurshid" <khurshid.normuradov gmail.com> writes:
On Thursday, 8 August 2013 at 12:08:00 UTC, khurshid wrote:

 std.json  parseJSON has a bug,

 //                         leading whitespaces
 auto json = parseJSON("1                                  
 .000");

 -- should throws, but it's success parsed as `1.000` .


 Khurshid...


Or more...

auto json = parseJSON(`{ "one": 1        .24E          +1    
}`);//should throws.
writeln(toJSON(&json)); // printed `{"one":12.4}`
Aug 08 2013
parent David <d dav1d.de> writes:
Am 08.08.2013 14:15, schrieb khurshid:

 On Thursday, 8 August 2013 at 12:08:00 UTC, khurshid wrote:

 std.json  parseJSON has a bug,

 //                         leading whitespaces
 auto json = parseJSON("1                                  .000");

 -- should throws, but it's success parsed as `1.000` .


 Khurshid...


 
 Or more...
 
 auto json = parseJSON(`{ "one": 1        .24E          +1   
 }`);//should throws.
 writeln(toJSON(&json)); // printed `{"one":12.4}`


Why is this a bug? JSON ignores whitespaces outside strings.
Aug 08 2013
prev sibling next sibling parent reply "SteveGuo" <steveguo outlook.com> writes:
On Thursday, 8 August 2013 at 12:08:00 UTC, khurshid wrote:

 std.json  parseJSON has a bug,

 //                         leading whitespaces
 auto json = parseJSON("1                                  
 .000");

 -- should throws, but it's success parsed as `1.000` .


 Khurshid...


I think bug reports should be here 
http://forum.dlang.org/group/digitalmars.D.bugs
Aug 08 2013
next sibling parent "anonymous" <anonymous example.com> writes:
On Thursday, 8 August 2013 at 12:38:39 UTC, SteveGuo wrote:

 I think bug reports should be here 
 http://forum.dlang.org/group/digitalmars.D.bugs


Actually, that only shows the activity on the bug tracker. New 
issues should be reported here: http://d.puremagic.com/issues
Aug 08 2013
prev sibling next sibling parent "khurshid" <khurshid.normuradov gmail.com> writes:
On Thursday, 8 August 2013 at 12:38:39 UTC, SteveGuo wrote:

 On Thursday, 8 August 2013 at 12:08:00 UTC, khurshid wrote:

 std.json  parseJSON has a bug,

 //                         leading whitespaces
 auto json = parseJSON("1                                  
 .000");

 -- should throws, but it's success parsed as `1.000` .


 Khurshid...


 I think bug reports should be here 
 http://forum.dlang.org/group/digitalmars.D.bugs


I reported.
http://d.puremagic.com/issues/show_bug.cgi?id=10776
Aug 08 2013
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, August 08, 2013 14:38:34 SteveGuo wrote:

 I think bug reports should be here
 http://forum.dlang.org/group/digitalmars.D.bugs


Please, never, never post to that list. It shouldn't even accept posts to it 
(and it may not anymore, since I haven't seen anyone post to it for a while). 
The whole purpose of that list is to sign up for notifications from bugzilla, 
and bugzilla will post to the list whenever a bug report is created or 
changed. All bug reports should go to

http://d.puremagic.com/issues

- Jonathan M Davis
Aug 08 2013
prev sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 8 August 2013 at 12:08:00 UTC, khurshid wrote:

 std.json  parseJSON has a bug,

 //                         leading whitespaces
 auto json = parseJSON("1                                  
 .000");

 -- should throws, but it's success parsed as `1.000` .


 Khurshid...


I don't think this is a bug, the json spec seems to indicate that 
this is valid.
Aug 08 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Tofu Ninja:


 I don't think this is a bug, the json spec seems to indicate 
 that this is valid.



The JSON decode of the Python2.6.6 standard library seems to 
refuse it:


 import json
 json.loads("1.000")






1.0

 json.loads("1                                  .000")






Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "...\Python26\lib\json\__init__.py", line 307, in loads
     return _default_decoder.decode(s)
   File "...\Python26\lib\json\decoder.py", line 322, in decode
     raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 35 - line 1 column 39 (char 
35 - 39)


Bye,
bearophile
Aug 08 2013
parent reply David <d dav1d.de> writes:
Am 08.08.2013 14:54, schrieb bearophile:

 Tofu Ninja:
 

 I don't think this is a bug, the json spec seems to indicate that this
 is valid.


 
 
 The JSON decode of the Python2.6.6 standard library seems to refuse it:
 

 import json
 json.loads("1.000")






 1.0

 json.loads("1                                  .000")






 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "...\Python26\lib\json\__init__.py", line 307, in loads
     return _default_decoder.decode(s)
   File "...\Python26\lib\json\decoder.py", line 322, in decode
     raise ValueError(errmsg("Extra data", s, end, len(s)))
 ValueError: Extra data: line 1 column 35 - line 1 column 39 (char 35 - 39)
 
 
 Bye,
 bearophile


JSON.parse("1                                  .000")
SyntaxError: Unexpected token .

If we follow an existing implementation, we should follow JavaScript
Aug 08 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
David:


 If we follow an existing implementation, we should follow 
 JavaScript


In my opinion we should follow the formal JSON grammar.

Bye,
bearophile
Aug 08 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 8 August 2013 at 13:49:22 UTC, bearophile wrote:

 In my opinion we should follow the formal JSON grammar.


This. Anyone who wants JavaScript behavior can use own 
third-party library bust standard library must behave according 
to published standards and specifications.
Aug 08 2013
next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Thursday, 8 August 2013 at 13:56:15 UTC, Dicebot wrote:

 On Thursday, 8 August 2013 at 13:49:22 UTC, bearophile wrote:

 In my opinion we should follow the formal JSON grammar.


 This. Anyone who wants JavaScript behavior can use own 
 third-party library bust standard library must behave according 
 to published standards and specifications.


Then my reading says this really _is_ a bug. At least, according 
to http://www.ietf.org/rfc/rfc4627.txt, which states and shows 
only that:
"Insignificant whitespace is allowed before or after any of the 
six structural characters."

..unless there's a different version that supersedes that one?

-Wyatt
Aug 08 2013
prev sibling next sibling parent reply "Tyler Jameson Little" <beatgammit gmail.com> writes:
On Thursday, 8 August 2013 at 13:56:15 UTC, Dicebot wrote:

 On Thursday, 8 August 2013 at 13:49:22 UTC, bearophile wrote:

 In my opinion we should follow the formal JSON grammar.


 This. Anyone who wants JavaScript behavior can use own 
 third-party library bust standard library must behave according 
 to published standards and specifications.


Exactly. Here's the official web page complete with nice graphics 
detailing the grammar: http://json.org/. I've read the JSON RFC 
before, but I can't remember what it says about whitespace within 
a basic value, but the graphics here make it very clear that 
whitespace does *not* belong inside a value.

The real question is, is this worth fixing before std.serialize 
makes it in? There will likely be other bugs once/if that's 
accepted. I tried extending std.json in the past with static 
reflection, but that didn't make it in for this very reason.
Aug 08 2013
parent reply David <d dav1d.de> writes:
 The real question is, is this worth fixing before std.serialize makes it
 in? There will likely be other bugs once/if that's accepted. I tried
 extending std.json in the past with static reflection, but that didn't
 make it in for this very reason.


Yep no one seems to care about std.json.
Aug 08 2013
parent reply "Lemonfiend" <lemon fie.nd> writes:
On Thursday, 8 August 2013 at 15:15:44 UTC, David wrote:

 The real question is, is this worth fixing before 
 std.serialize makes it
 in? There will likely be other bugs once/if that's accepted. I 
 tried
 extending std.json in the past with static reflection, but 
 that didn't
 make it in for this very reason.


 Yep no one seems to care about std.json.


I heard about std.xml being crap and was going to use std.json, 
then I ran into this thread.

So what should one use to parse json, or xml for that matter?
Aug 08 2013
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 8 August 2013 at 16:26:40 UTC, Lemonfiend wrote:

 On Thursday, 8 August 2013 at 15:15:44 UTC, David wrote:

 The real question is, is this worth fixing before 
 std.serialize makes it
 in? There will likely be other bugs once/if that's accepted. 
 I tried
 extending std.json in the past with static reflection, but 
 that didn't
 make it in for this very reason.


 Yep no one seems to care about std.json.


 I heard about std.xml being crap and was going to use std.json, 
 then I ran into this thread.

 So what should one use to parse json, or xml for that matter?


There are several JSON implementations floating around, I 
personally prefer vibe.d one - 
https://github.com/Dicebot/vibe.d/blob/master/source/vibe/data/json.d

For XML Tango module is often mentioned as most awesome one - 
https://github.com/SiegeLord/Tango-D2/tree/d2port/tango/text/xml
Aug 08 2013
prev sibling parent reply David <d dav1d.de> writes:
Am 08.08.2013 18:26, schrieb Lemonfiend:

 On Thursday, 8 August 2013 at 15:15:44 UTC, David wrote:

 The real question is, is this worth fixing before std.serialize makes it
 in? There will likely be other bugs once/if that's accepted. I tried
 extending std.json in the past with static reflection, but that didn't
 make it in for this very reason.


 Yep no one seems to care about std.json.


 
 I heard about std.xml being crap and was going to use std.json, then I
 ran into this thread.
 
 So what should one use to parse json, or xml for that matter?


std.json is Ok, the API is a real pain. I made a pull request improving
the API a few weeks ago, no one seems to really care. Better use a 3rd
party implementation or roll your own (what seems to be Ds mentallity)
Aug 08 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

 I made a pull request improving the API a few weeks ago,
 no one seems to really care.


Phobos needs a new dictator.
Aug 08 2013
next sibling parent reply David <d dav1d.de> writes:
Am 08.08.2013 20:24, schrieb Adam D. Ruppe:

 On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

 I made a pull request improving the API a few weeks ago,
 no one seems to really care.


 
 Phobos needs a new dictator.


Either that, or I will soon start my own standard lib and stop caring
about phobos. I think, I even know a few who would help me...
Aug 08 2013
next sibling parent "Craig Dillabaugh" <cdillaba cg.scs.carleton.ca> writes:
On Thursday, 8 August 2013 at 18:31:52 UTC, David wrote:

 Am 08.08.2013 20:24, schrieb Adam D. Ruppe:

 On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

 I made a pull request improving the API a few weeks ago,
 no one seems to really care.


 
 Phobos needs a new dictator.


 Either that, or I will soon start my own standard lib and stop 
 caring
 about phobos. I think, I even know a few who would help me...


Maybe the Tango developers would want to join in :o)
Aug 08 2013
prev sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 8 August 2013 at 18:31:52 UTC, David wrote:

 Am 08.08.2013 20:24, schrieb Adam D. Ruppe:

 On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

 I made a pull request improving the API a few weeks ago,
 no one seems to really care.


 
 Phobos needs a new dictator.


 Either that, or I will soon start my own standard lib and stop 
 caring
 about phobos. I think, I even know a few who would help me...


It is really bad that people are actually talking about starting 
there own standard lib, I wasn't around for the whole phobos vs 
tango thing but from what I hear, it wasn't pretty. If there is 
problems with phobos or the way its managed, I feel like we 
should try and fix them and not try to replace it.
Aug 08 2013
next sibling parent reply David <d dav1d.de> writes:
Am 08.08.2013 22:15, schrieb Tofu Ninja:

 On Thursday, 8 August 2013 at 18:31:52 UTC, David wrote:

 Am 08.08.2013 20:24, schrieb Adam D. Ruppe:

 On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

 I made a pull request improving the API a few weeks ago,
 no one seems to really care.


 Phobos needs a new dictator.


 Either that, or I will soon start my own standard lib and stop caring
 about phobos. I think, I even know a few who would help me...


 
 It is really bad that people are actually talking about starting there
 own standard lib, I wasn't around for the whole phobos vs tango thing
 but from what I hear, it wasn't pretty. If there is problems with phobos
 or the way its managed, I feel like we should try and fix them and not
 try to replace it.


phobos is actually useable, still I don't like many of its design
choices. I feel the same with D. D itself is better than most languages,
I still think there is a long way to go, but "official" D people seem
like it's enough and block progress (e.g.  property/-property the whole
situation is a mess, but instead of properly reimplementing it, they try
to fix an already broken system and break code with it, why not break
code once, but replace it with a proper system)
Aug 08 2013
next sibling parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 8 August 2013 at 21:40:46 UTC, David wrote:

 Am 08.08.2013 22:15, schrieb Tofu Ninja:

 On Thursday, 8 August 2013 at 18:31:52 UTC, David wrote:

 Am 08.08.2013 20:24, schrieb Adam D. Ruppe:

 On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

 I made a pull request improving the API a few weeks ago,
 no one seems to really care.


 Phobos needs a new dictator.


 Either that, or I will soon start my own standard lib and 
 stop caring
 about phobos. I think, I even know a few who would help me...


 
 It is really bad that people are actually talking about 
 starting there
 own standard lib, I wasn't around for the whole phobos vs 
 tango thing
 but from what I hear, it wasn't pretty. If there is problems 
 with phobos
 or the way its managed, I feel like we should try and fix them 
 and not
 try to replace it.


 phobos is actually useable, still I don't like many of its 
 design
 choices. I feel the same with D. D itself is better than most 
 languages,
 I still think there is a long way to go, but "official" D 
 people seem
 like it's enough and block progress (e.g.  property/-property 
 the whole
 situation is a mess, but instead of properly reimplementing it, 
 they try
 to fix an already broken system and break code with it, why not 
 break
 code once, but replace it with a proper system)


I can understand how you feel, I do sort of feel like people are 
way to scared of breaking changes and it causing things that 
badly need to be fix to not be...  but still, I don't think a 
whole new standard lib would do much good
Aug 08 2013
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 08, 2013 at 11:40:45PM +0200, David wrote:

 Am 08.08.2013 22:15, schrieb Tofu Ninja:


[...]

 It is really bad that people are actually talking about starting
 there own standard lib, I wasn't around for the whole phobos vs
 tango thing but from what I hear, it wasn't pretty. If there is
 problems with phobos or the way its managed, I feel like we should
 try and fix them and not try to replace it.


 
 phobos is actually useable, still I don't like many of its design
 choices. I feel the same with D. D itself is better than most
 languages, I still think there is a long way to go, but "official" D
 people seem like it's enough and block progress


Well, I don't speak for them, whoever they are, but in my understanding
one of the main reasons there's such a high bar to changes is because
we're trying to stabilize the language so that adopters can rely on it
not having drastic changes every other release, that will require
massive code rewrites. No language will ever be perfect, and if we keep
on breaking people's existing code in the endless quest to improve the
language, D will have no users left.



 (e.g.  property/-property the whole situation is a mess, but instead
 of properly reimplementing it, they try to fix an already broken
 system and break code with it, why not break code once, but replace it
 with a proper system)


The problem is that nobody can agree on what the "proper" system should
be. We've had that  property discussion before. Nobody is happy with the
current state of affairs, but nobody can agree with what the solution
should be, either. We just have long neverending debates about it but no
conclusion is ever reached.

Perhaps what's needed, is somebody who's headstrong enough, and
persistent enough, to just pick one of the solutions -- any one of them
-- take the time to implement it in DMD, and show it to the rest of us
to prove how superior it is. Then continue to pester Walter ceaselessly
until he agrees to merge it. In the current state of stalemate, an
actual, working implementation of a solution (even if it's not quite the
one that one may have in mind) should be pretty convincing.

Complaining about it yet being unwilling to do anything about it, OTOH,
is very likely to just fall on deaf ears, judging from what I've seen
around these parts. That's just the way things work, unfortunately.
*shrug*


T

-- 
Some days you win; most days you lose.
Aug 08 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
On Saturday, 10 August 2013 at 18:28:32 UTC, H. S. Teoh wrote:

 ...
 T


Perhaps there's a forum bug here. I was not in this broken piece 
of therad.

Bye,
bearophile
Aug 10 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Aug 10, 2013 at 09:09:17PM +0200, bearophile wrote:

 On Saturday, 10 August 2013 at 18:28:32 UTC, H. S. Teoh wrote:

...
T


 
 Perhaps there's a forum bug here. I was not in this broken piece of
 therad.


[...]

You snipped too much of the context, so I've no idea what you're talking
about. But this is a known problem with the mailman/NNTP interface:
somewhere along the line message IDs get rewritten when they shouldn't
be, causing threads to break. Somebody seriously needs to look into
this, as it's an annoying problem that keeps cropping up.


T

-- 
He who laughs last thinks slowest.
Aug 10 2013
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 08 Aug 2013 22:15:28 +0200
schrieb "Tofu Ninja" <emmons0 purdue.edu>:


 On Thursday, 8 August 2013 at 18:31:52 UTC, David wrote:

 Am 08.08.2013 20:24, schrieb Adam D. Ruppe:

 On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

 I made a pull request improving the API a few weeks ago,
 no one seems to really care.


=20
 Phobos needs a new dictator.


 Either that, or I will soon start my own standard lib and stop=20
 caring
 about phobos. I think, I even know a few who would help me...


=20
 It is really bad that people are actually talking about starting=20
 there own standard lib, I wasn't around for the whole phobos vs=20
 tango thing but from what I hear, it wasn't pretty. If there is=20
 problems with phobos or the way its managed, I feel like we=20
 should try and fix them and not try to replace it.


I'm sorry that you've had such bad experiences when contributing to
phobos. I think this is less of a phobos than a std.json problem. I'd
like to explain the special issue we have with std.json - as far as I
understand it:

std.json is a very old module:=20
* It's API is not up to current phobos standards
* there are some - probably serious - bugs
* no interaction with ranges
* doesn't use  safe const, nothrow IIRC
* inherently unsafe API (accessing the union part)
* could be (should be) much faster
* the orignal author is no longer around. AFAICS there's nobody feeling
  responsible for this module.

The last point is probably the biggest problem. Most people here would
probably like to see a complete replacement - std.json2. Some of us
have argued that we should remove std.json (and std.xml) from phobos
ASAP even if there's no replacement as this code is really not what we
want in the standard lib.

Others are strongly against any API breakage (especially without
replacement) and therefore std.json is still there. But this makes
things even more difficult: Improving std.json means we'd have both API
breakage and a sub-optimal std.json. I guess nobody wants to merge
changes to std.json because 1) they feel std.json has to be replaced
anyway and improving the old design is wasting time 2) they don't want
to be blamed for any possible API breakage.

There were some discussions how a std.json replacement should look
like, I'll try to reiterate the main points:

=3D=3D=3D Input API: =3D=3D=3D
1) A pull-parser/tokenizer/lexer
   * Should implement a InputRange API with ElementType JSONToken.
   * Should be as fast as possible
   * shouldn't allocate memory
   * shouldn't even convert strings to numbers. Instead JSONToken
     should have a 'type' field (ObjectStart, ArrayStart, String,
     Number, ...) and a value field which is always a (raw!) w/d/string.
     A templated T get!(JSONType) helper which could also verify the
     type would be useful. Any decoding of JSON strings should be done
     in the get function. Get may optionally use some caching.
   * It should be specialized for input arrays
     (const/immutable/mutable) of char/wchar/dchar which are completely
     in-memory and use slicing in these cases.
   * It should work with arbitrary ranges of w/d/char. To avoid memory
     allocation a fixed-length internal buffer should be used and token
     values should be slices to that buffer. (It would be good if the
     user can query whether values are slices of the original input
     which can be reused, or sliced of the iternal buffer which have to
     be .dup ed to keep them)

2) A DOM-API
   * Should be based on 1)
   * Same principle as current std.json, but in modern D
   * API similar to std.variant, DYAML

optional, nice-to have:

3) A Sax-style API (based on 1)

4) A simple deserialization API
   * basically T fromJSON(T, CHAR)(CHAR input) (supporting all
     inputs supported by 1)
   * Shouldn't allocate any memory
   * Optionally skipping fields which are in the JSON data but not part
     of T and the other way round.
   * Usage: auto artist =3D fromJSON(Artist, `{"name": "", "songs":42}=C2=
=B4);


=3D=3D=3D Output API: =3D=3D=3D
1) A OutputRange of JSONTokens
   * Shouldn't allocate
   * Should work on top of other OutputRanges
   * should accept the tokens from the pull-parser, so we can do
     copy(JSONTokenizer(json), JSONWriter(stdout));
   * Additionally provide an easier user API: JSONWriter.startObject(),
     JSONWriter.startArray(), JSONWriter.writeField(name, T),...

2) DOM API
   * It should of course be possible to manipulate the DOM, then write
     it back
   * Should be the same as the input DOM API
   * Should be based on 2)

optional, nice-to have:
3) A simple serialization API
   * counterpart to the deserialization API
   * Shouldn't allocate any memory
   * Based on output-ranges and 1)
   * Usage: myvariable.toJSON(stdout);
   * Helper function using Appender: string json =3D myvariable.toJSON();

Another questions is if and how this should interact with streams, if
we ever get them.

This list is quite long. As you see the demands are quite high and
therefore it will be a difficult and time-consuming task to write a
std.json2 which pleases everyone. Of course it's possible we're
overthinking this.

I don't want to make any particular point here whether std.json should
be removed, partially updated & fixed or simply made deprecated and
left alone. I'm just trying to give some background information on this
issue as far as I perceive it.

BTW: Because of these issues there already are alternative libraries
implementing JSON stuff. Apart from vibe.d there's also ae.util.json
and some more. Fortunately as long as nobody decides to fork druntime
there's no real problem. Other languages (C/C++) don't have JSON in the
standard library either. But still please don't let this turn you away
from phobos in general - improvements to other modules are usually
handled in a much better way.
Aug 08 2013
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 8 August 2013 at 22:19:28 UTC, Johannes Pfau wrote:

 I'd like to explain the special issue we have with std.json -
 as far as I understand it:
 *snip*


This actually brings up the main beefs I have with the phobos dev 
process:

1) your requirements list should be prominently documented, so 
people considering writing something for phobos know what is 
needed up front

2) what's considered for phobos and what is just outside its 
scope?

3) these processes should be more authoritative than "as far as I 
understand it" - then new people could get involved with 
reviewing too, since they have an objective list of stuff to be 
on the lookout for and don't have to wait for someone else to 
come along and say something

4) phobos lets the perfect be the enemy of the good. std.json 
really isn't that bad, and ~50 lines of prettier add-on API could 
make it nicer, but instead of doing that we wait years for 
something that doesn't seem to be happening at all.



 * the orignal author is no longer around. AFAICS there's nobody 
 feeling responsible for this module.


This is a problem too: a module shouldn't be in the hands of one 
person. If any random contributor follows the documented rules, 
they should get their code pulled in. Since phobos is a community 
project, I think we should all be equally responsible for every 
part of it. One person might take the lead and do most the 
work.... but it shouldn't be *exclusive*, so if that one person 
can't or won't do something, someone else can just do it.
Aug 08 2013
parent Johannes Pfau <nospam example.com> writes:
Am Fri, 09 Aug 2013 00:39:50 +0200
schrieb "Adam D. Ruppe" <destructionator gmail.com>:


 On Thursday, 8 August 2013 at 22:19:28 UTC, Johannes Pfau wrote:

 I'd like to explain the special issue we have with std.json -
 as far as I understand it:
 *snip*


 
 This actually brings up the main beefs I have with the phobos dev 
 process:
 
 1) your requirements list should be prominently documented, so 
 people considering writing something for phobos know what is 
 needed up front


I posted this to the wiki:
http://wiki.dlang.org/Wish_list/std.json
and linked it from http://wiki.dlang.org/Wish_list
If you know some better place for that page, feel free to move it.


 
 2) what's considered for phobos and what is just outside its 
 scope?


Sometimes I wonder about that as well. We usually follow the 'batteries
included' approach and all kind of modules are accepted into phobos.
However Andrei often dislikes 'trivial to implement' functions and
simple helper functions or helper aliases.


 
 3) these processes should be more authoritative than "as far as I 
 understand it" - then new people could get involved with 
 reviewing too, since they have an objective list of stuff to be 
 on the lookout for and don't have to wait for someone else to 
 come along and say something


There was a discussion about std.json some time ago and the list I
posted is what I remember from that discussion. But we should really
document such 'requirements'. Though std.json is an exception here, we
usually don't have a list of 'requirements' before the module is
written.


 
 4) phobos lets the perfect be the enemy of the good. std.json 
 really isn't that bad, and ~50 lines of prettier add-on API could 
 make it nicer, but instead of doing that we wait years for 
 something that doesn't seem to be happening at all.
 


I totally agree. I think it's a problem with exaggerating API stability
when phobos is not ready for that yet. Although if it's really add-on
only this shouldn't be a problem.


 

 * the orignal author is no longer around. AFAICS there's nobody 
 feeling responsible for this module.


 
 This is a problem too: a module shouldn't be in the hands of one 
 person. If any random contributor follows the documented rules, 
 they should get their code pulled in. Since phobos is a community 
 project, I think we should all be equally responsible for every 
 part of it. One person might take the lead and do most the 
 work.... but it shouldn't be *exclusive*, so if that one person 
 can't or won't do something, someone else can just do it.


This is working for some modules(std.algorithm, std.range). I don't
know why it's not working for modules like std.json or std.xml.
Aug 09 2013
prev sibling parent David <d dav1d.de> writes:
Am 09.08.2013 00:19, schrieb Johannes Pfau:

 Am Thu, 08 Aug 2013 22:15:28 +0200
 schrieb "Tofu Ninja" <emmons0 purdue.edu>:
 

 On Thursday, 8 August 2013 at 18:31:52 UTC, David wrote:

 Am 08.08.2013 20:24, schrieb Adam D. Ruppe:

 On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

 I made a pull request improving the API a few weeks ago,
 no one seems to really care.


 Phobos needs a new dictator.


 Either that, or I will soon start my own standard lib and stop 
 caring
 about phobos. I think, I even know a few who would help me...


 It is really bad that people are actually talking about starting 
 there own standard lib, I wasn't around for the whole phobos vs 
 tango thing but from what I hear, it wasn't pretty. If there is 
 problems with phobos or the way its managed, I feel like we 
 should try and fix them and not try to replace it.


 
 I'm sorry that you've had such bad experiences when contributing to
 phobos. I think this is less of a phobos than a std.json problem. I'd
 like to explain the special issue we have with std.json - as far as I
 understand it:
 


I think the problem is, people are afraid that changes will hurt D. The
mentallity to improve D and make it awsome is lost (at least if feels
like it), now all what is left "WE NEED TO MAKE D PRODUCTION
READY!!!1111111 COMPANIES USE D!!!!1111" which makes sense, but one
shouldn't forget why D was "invented", to improve the programming
situation and craft a language which is superior to all others and which
allows to write simple clean but yet powerful code!
Aug 08 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 08, 2013 at 10:15:28PM +0200, Tofu Ninja wrote:

 On Thursday, 8 August 2013 at 18:31:52 UTC, David wrote:

Am 08.08.2013 20:24, schrieb Adam D. Ruppe:

On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

I made a pull request improving the API a few weeks ago,
no one seems to really care.


Phobos needs a new dictator.


Either that, or I will soon start my own standard lib and stop
caring
about phobos. I think, I even know a few who would help me...


 
 It is really bad that people are actually talking about starting
 there own standard lib, I wasn't around for the whole phobos vs
 tango thing but from what I hear, it wasn't pretty. If there is
 problems with phobos or the way its managed, I feel like we should
 try and fix them and not try to replace it.


Yeah, another Phobos/Tango (or whatever it may be) split is probably
going to be the thing that finally kills off D before it ever reaches
widespread adoption. Please, let's not go there.

Here's another approach. If you feel something is wrong with Phobos and
your pulls aren't getting accepted for whatever reason, why not package
it up nicely in a form that's easily installable by users, and publish
it? Say you have a replacement for std.json that you feel is far
superior, well, then make it installable as a standalone library (say
alt.json), and promote it here and get many people to start to use it.
Once it's widely-used, I suspect it will be far easier to convince
people to put it into Phobos. :-)


T

-- 
"I'm running Windows '98." "Yes." "My computer isn't working now." "Yes, you
already said that." -- User-Friendly
Aug 08 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 08, 2013 at 08:24:19PM +0200, Adam D. Ruppe wrote:

 On Thursday, 8 August 2013 at 17:33:38 UTC, David wrote:

I made a pull request improving the API a few weeks ago,
no one seems to really care.


 
 Phobos needs a new dictator.


Or rather, Phobos needs more reviewers/committers. In the past 2 weeks
or so, pull requests have been piling up, and lots of them have a green
on the autotester, yet very few requests are actually being pulled.
Sounds like the usually-active Phobos devs are busy with other things.


T

-- 
In a world without fences, who needs Windows and Gates? -- Christian Surchi
Aug 08 2013
prev sibling next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 8 August 2013 at 13:56:15 UTC, Dicebot wrote:

 On Thursday, 8 August 2013 at 13:49:22 UTC, bearophile wrote:

 In my opinion we should follow the formal JSON grammar.


 This. Anyone who wants JavaScript behavior can use own 
 third-party library bust standard library must behave according 
 to published standards and specifications.


A formal grammar can be found here starting on page 202 but I 
don't know enough about grammars to be able to interpret it.
http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf

Im starting to become less sure if its a bug or not...
Aug 08 2013
next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 08 Aug 2013 17:17:38 +0200
"Tofu Ninja" <emmons0 purdue.edu> wrote:


 On Thursday, 8 August 2013 at 13:56:15 UTC, Dicebot wrote:

 On Thursday, 8 August 2013 at 13:49:22 UTC, bearophile wrote:

 In my opinion we should follow the formal JSON grammar.


 This. Anyone who wants JavaScript behavior can use own 
 third-party library bust standard library must behave according 
 to published standards and specifications.


 
 A formal grammar can be found here starting on page 202 but I 
 don't know enough about grammars to be able to interpret it.
 http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf
 
 Im starting to become less sure if its a bug or not...


I think that makes it clear that whitespace is NOT allowed within
numeric or fractional literals. Note that the JSONNumber and
JSONFraction tokens are defined in the lexical section (15.12.1.1). In
a lexer, whitespace is something that is always handled explicitly (as
they do, in both the JSONWhiteSpace and JSONStringCharacter tokens).

Down in the "Syntactic Grammar" section (15.12.1.2), they don't
explicitly mention anything about whitespace (for example, in the
JSONElementList token, where we already *know* whitespace is allowed).
This is undoubtedly because (aside from lexing) whitespace is generally
NOT explicit, but rather implied to optionally exist before/after each
token.

It sounds like the bug is that std.json is currently treating JSONNumber
(and maybe JSONFraction too) as a parser-handled nonterminal, instead
of a lexer-handled terminal as the spec indicates.
Aug 08 2013
next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 8 August 2013 at 15:45:22 UTC, Nick Sabalausky wrote:

 On Thu, 08 Aug 2013 17:17:38 +0200
 "Tofu Ninja" <emmons0 purdue.edu> wrote:


 On Thursday, 8 August 2013 at 13:56:15 UTC, Dicebot wrote:

 On Thursday, 8 August 2013 at 13:49:22 UTC, bearophile wrote:

 In my opinion we should follow the formal JSON grammar.


 This. Anyone who wants JavaScript behavior can use own 
 third-party library bust standard library must behave 
 according to published standards and specifications.


 
 A formal grammar can be found here starting on page 202 but I 
 don't know enough about grammars to be able to interpret it.
 http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf
 
 Im starting to become less sure if its a bug or not...


 I think that makes it clear that whitespace is NOT allowed 
 within
 numeric or fractional literals. Note that the JSONNumber and
 JSONFraction tokens are defined in the lexical section 
 (15.12.1.1). In
 a lexer, whitespace is something that is always handled 
 explicitly (as
 they do, in both the JSONWhiteSpace and JSONStringCharacter 
 tokens).

 Down in the "Syntactic Grammar" section (15.12.1.2), they don't
 explicitly mention anything about whitespace (for example, in 
 the
 JSONElementList token, where we already *know* whitespace is 
 allowed).
 This is undoubtedly because (aside from lexing) whitespace is 
 generally
 NOT explicit, but rather implied to optionally exist 
 before/after each
 token.

 It sounds like the bug is that std.json is currently treating 
 JSONNumber
 (and maybe JSONFraction too) as a parser-handled nonterminal, 
 instead
 of a lexer-handled terminal as the spec indicates.


Just out of curiosity what is this sort of grammar format called 
or what would one want to try to look up to understand it? I just 
have seen it around a few times and never really knew what it was 
about other than some how it describes a language.
Aug 08 2013
parent "Wyatt" <wyatt.epp gmail.com> writes:
On Thursday, 8 August 2013 at 17:37:37 UTC, Tofu Ninja wrote:

 Just out of curiosity what is this sort of grammar format 
 called or what would one want to try to look up to understand 
 it? I just have seen it around a few times and never really 
 knew what it was about other than some how it describes a 
 language.


That'd be some flavour of BNF: 
https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form

Very handy thing to know how to read (though I prefer PEGs).

-Wyatt
Aug 08 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 08, 2013 at 11:45:19AM -0400, Nick Sabalausky wrote:

 On Thu, 08 Aug 2013 17:17:38 +0200
 "Tofu Ninja" <emmons0 purdue.edu> wrote:
 

 On Thursday, 8 August 2013 at 13:56:15 UTC, Dicebot wrote:

 On Thursday, 8 August 2013 at 13:49:22 UTC, bearophile wrote:

 In my opinion we should follow the formal JSON grammar.


 This. Anyone who wants JavaScript behavior can use own 
 third-party library bust standard library must behave according 
 to published standards and specifications.


 
 A formal grammar can be found here starting on page 202 but I 
 don't know enough about grammars to be able to interpret it.
 http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf
 
 Im starting to become less sure if its a bug or not...




[...]

I find that the spec is not very clear as to what is/isn't allowed.
RFC-4627 is more unambiguous: it explicitly states that insignificant
whitespace is allowed before or after any of the six structural
characters, which are defined (in section 2) to be:

	[ { ] } : ,

Section 2.4 in RFC-4627 defines the number format very clearly. There is
no mention of insignificant whitespace. This, plus the statement above
in section 2 about structural characters, makes it clear that whitespace
is NOT allowed inside a number literal.

So please file a bug against std.json.


T

-- 
If you compete with slaves, you become a slave. -- Norbert Wiener
Aug 08 2013
prev sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Thursday, 8 August 2013 at 15:17:40 UTC, Tofu Ninja wrote:

 A formal grammar can be found here starting on page 202 but I 
 don't know enough about grammars to be able to interpret it.
 http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf

 Im starting to become less sure if its a bug or not...


I don't recogise that exact BNF notation (single colon? New to 
me.), but unless it's fairly...unique, it's still a bug according 
to a strict reading.  (I feel the RFC version is somewhat more 
clear, honestly.)

-Wyatt
Aug 08 2013
prev sibling parent "Borislav Kosharov" <boby_dsm abv.bg> writes:
On Thursday, 8 August 2013 at 13:56:15 UTC, Dicebot wrote:

 On Thursday, 8 August 2013 at 13:49:22 UTC, bearophile wrote:

 In my opinion we should follow the formal JSON grammar.


 This. Anyone who wants JavaScript behavior can use own 
 third-party library bust standard library must behave according 
 to published standards and specifications.


I agree with Dicebot. I posted on the other thread for JSON real 
numbers and explained why it isn't a bug.
Aug 08 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 08, 2013 at 03:49:15PM +0200, bearophile wrote:

 David:
 

If we follow an existing implementation, we should follow
JavaScript


 
 In my opinion we should follow the formal JSON grammar.


[...]

+1.


T

-- 
Recently, our IT department hired a bug-fix engineer. He used to work for
Volkswagen.
Aug 08 2013