www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Newline character set in the D lexer - NEL

reply Cecil Ward <cecil cecilward.com> writes:
Would there be any benefit from the following suggestion? Add the 
character Unicode NEL U+0085 into the set of EndOfLine characters 
in the lexer ?

Cecil Ward.
Aug 30 2020
next sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Monday, 31 August 2020 at 01:49:06 UTC, Cecil Ward wrote:
 Would there be any benefit from the following suggestion? Add 
 the character Unicode NEL U+0085 into the set of EndOfLine 
 characters in the lexer ?

 Cecil Ward.
I personally think we should have these definitions: /* NUL EM SUB */ EndOfFile = { 0x00 | 0x19 | 0x1A | PhysicalEndOfFile }; /* LF FF CR CR LF NEL LSEP PSEP */ EndOfLine = { 0x0A | 0x0C | 0x0D | 0x0D 0x0A | 0x85 | 0x2028 | 0x2029 | EndOfFile }; /* HT VT SP NBSP NQSP MQSP ENSP EMSP 3/MSP */ WhiteSpace = { 0x09 | 0x0B | 0x20 | 0xA0 | 0x2000 | 0x2001 | 0x2002 | 0x2003 | 0x2004 /* 4/MSP 6/MSP FSP PSP THSP HSP ZWSP NNBSP */ | 0x2005 | 0x2006 | 0x2007 | 0x2008 | 0x2009 | 0x200A | 0x200B | 0x202F /* MMSP WJ IDSP ZWNBSP */ | 0x205F | 0x2060 | 0x3000 | 0xFEFF | EndOfLine }; The definition of D source files misses quite a lot of them :-( EM = end of medium (what if not this should end a file?!?) NEL = New Line LSEP = Line Separator PSEP = Paragraph Separator NBSP = non-braking space NQSP = ENSP = N-wide space MQSP = EMSP = M-wide space 3/MSP = 1/3 M-wide space (three spaces together are as wide as an M) 4/MSP = 1/4 M-wide space 6/MSP = 1/6 M-wide space FSP = figure space PSP = point space THSP = thin space HSP = hair space ZWSP = zero width space NNBSP = narrow non-braking space MMSP = mathematic space WJ = word joiner (invisible space that separate words for the spelling correction) IDSP = ideographic space (same width as a chinese character) ZWNBSP = zero-width non-braking space
Aug 31 2020
parent reply Cecil Ward <cecil cecilward.com> writes:
I agree with Dominikus

Note to earlier poster: NEL was used and just possibly may still 
be used by IBM mainframe users; XML 1.1 understands NEL iirc;

see https://www.w3.org/TR/newline/   and

        https://www.w3.org/International/questions/qa-controls
Sep 03 2020
parent reply NilsLankila <NilsLankila gmx.us> writes:
On Friday, 4 September 2020 at 00:48:59 UTC, Cecil Ward wrote:
 I agree with Dominikus

 Note to earlier poster: NEL was used and just possibly may 
 still be used by IBM mainframe users; XML 1.1 understands NEL 
 iirc;

 see https://www.w3.org/TR/newline/   and

        https://www.w3.org/International/questions/qa-controls
Given the lack of answers I would suggest to go ahead with a PR or at least open an issue. Lexing is not a big deal but if nobody cares this will never be done.
Sep 03 2020
parent reply Cecil Ward <cecil cecilward.com> writes:
On Friday, 4 September 2020 at 05:28:47 UTC, NilsLankila wrote:
 Given the lack of answers I would suggest to go ahead with a PR 
 or at least open an issue. Lexing is not a big deal but if 
 nobody cares this will never be done.
Agreed, Nils. Mind you someone cared enough to include U+2028 and U+2029 in the lexer spec. I have no idea how to initiate a "PR". Perhaps someone could help me with this?
Sep 07 2020
parent James Blachly <james.blachly gmail.com> writes:
PR = "Pull Request".

Easy way is to fork the project on github, clone your (forked version of 
the) project, make changes, push back. This could be in ~master on your 
own fork, or ideally in a separate branch.

Then on github, go to the original project and start a new pull request. 
It should automagically detect that you've made changes (again ideally 
in a branch of your fork), and offer to make a pull request with your 
changes against ~master (or whatever is set as the default branch for 
the project).

James

On 9/8/20 2:42 AM, Cecil Ward wrote:
 On Friday, 4 September 2020 at 05:28:47 UTC, NilsLankila wrote:
 Given the lack of answers I would suggest to go ahead with a PR or at 
 least open an issue. Lexing is not a big deal but if nobody cares this 
 will never be done.
Agreed, Nils. Mind you someone cared enough to include U+2028 and U+2029 in the lexer spec. I have no idea how to initiate a "PR". Perhaps someone could help me with this?
Sep 08 2020
prev sibling parent reply Nils Lankila <NilsLankila gmx.us> writes:
On Monday, 31 August 2020 at 01:49:06 UTC, Cecil Ward wrote:
 Would there be any benefit from the following suggestion? Add 
 the character Unicode NEL U+0085 into the set of EndOfLine 
 characters in the lexer ?

 Cecil Ward.
Pardon me but why bother while ascii gives already all we need to put spaces and new lines with fast decode (< 80h) ?
Aug 31 2020
parent Paul Backus <snarwin gmail.com> writes:
On Monday, 31 August 2020 at 09:39:12 UTC, Nils Lankila wrote:
 On Monday, 31 August 2020 at 01:49:06 UTC, Cecil Ward wrote:
 Would there be any benefit from the following suggestion? Add 
 the character Unicode NEL U+0085 into the set of EndOfLine 
 characters in the lexer ?

 Cecil Ward.
Pardon me but why bother while ascii gives already all we need to put spaces and new lines with fast decode (< 80h) ?
D already recognizes some non-ascii characters as spaces and line separators [1], so the decision to "bother" has already been made. [1] https://dlang.org/spec/lex.html#character_set
Aug 31 2020