www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DScanner is ready for use

reply "Brian Schott" <briancschott gmail.com> writes:
DScanner is a tool for analyzing D source code. It has the 
following features:

* Prints out a complete AST of a source file in XML format.
* Syntax checks code and prints warning/error messages
* Prints a listing of modules imported by a source file
* Syntax highlights code in HTML format
* Provides more meaningful "line of code" count than wc
* Counts tokens in a source file

The lexer/parser/AST are located in the "std/d" directory in the 
repository. These files should prove useful to anyone else 
working on D tooling.

https://github.com/Hackerpilot/Dscanner

Aside: the D grammar that I reverse-engineered can be located 
here: 
https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Jul 27 2013
next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Sun, 28 Jul 2013 00:27:34 +0200
"Brian Schott" <briancschott gmail.com> wrote:

 DScanner is a tool for analyzing D source code. It has the 
 following features:
 
 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file
 
 The lexer/parser/AST are located in the "std/d" directory in the 
 repository. These files should prove useful to anyone else 
 working on D tooling.
 
 https://github.com/Hackerpilot/Dscanner
 
 Aside: the D grammar that I reverse-engineered can be located 
 here: 
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Sweet, I was *just* thinking about writing a D -> HTML syntax highlighter no more than about five minutes ago. Glad to see I won't have to :) I'll definitely be checking this out.
Jul 27 2013
prev sibling next sibling parent "qznc" <qznc web.de> writes:
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
 DScanner is a tool for analyzing D source code. It has the 
 following features:

 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file

 The lexer/parser/AST are located in the "std/d" directory in 
 the repository. These files should prove useful to anyone else 
 working on D tooling.

 https://github.com/Hackerpilot/Dscanner

 Aside: the D grammar that I reverse-engineered can be located 
 here: 
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Great! :) I do not understand the LoC count, though. The description sounds like "Number of Statements"?
Jul 28 2013
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
 DScanner is a tool for analyzing D source code. It has the 
 following features:

 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file

 The lexer/parser/AST are located in the "std/d" directory in 
 the repository. These files should prove useful to anyone else 
 working on D tooling.

 https://github.com/Hackerpilot/Dscanner

 Aside: the D grammar that I reverse-engineered can be located 
 here: 
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Awesome! I hope it won't be forgotten by the time I need it :) By the way, how far is that "std.d.*" stuff from ongoing Phobos inclusion review? I suppose currently it does not do any semantical analysis? How hard it would be to implement dmd warnings on top of dscanner instead?
Jul 28 2013
parent reply "Jacob Carlborg" <doob me.com> writes:
On Sunday, 28 July 2013 at 12:49:34 UTC, Dicebot wrote:

 Awesome! I hope it won't be forgotten by the time I need it :)

 By the way, how far is that "std.d.*" stuff from ongoing Phobos 
 inclusion review?

 I suppose currently it does not do any semantical analysis? How 
 hard it would be to implement dmd warnings on top of dscanner 
 instead?
I don't think it's necessary for semantic analysis to be included in Phobos. It's enough to start with a lexer, then later add a parser and semantic analysis. -- /Jacob Carlborg
Jul 28 2013
parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 28 July 2013 at 13:44:03 UTC, Jacob Carlborg wrote:
 I don't think it's necessary for semantic analysis to be 
 included in Phobos. It's enough to start with a lexer, then 
 later add a parser and semantic analysis.
Those were 2 separate not related questions ;)
Jul 28 2013
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
 Aside: the D grammar that I reverse-engineered can be located 
 here: 
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Does arrayLiteral support stray comma?
Jul 28 2013
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
Also looks like enumBody supports multiple commas with nothing 
between them, but dmd doesn't support it. Sould be
'{' enumMember (',' enumMember)* ','? '}'
Jul 28 2013
parent reply "Kagamin" <spam here.lot> writes:
And arrayLiteral should be
'[' (assignExpression (',' assignExpression)* ','?)? ']'
Jul 28 2013
parent "Kagamin" <spam here.lot> writes:
The same for arrayInitializer:
'[' (arrayMemberInitialization (',' arrayMemberInitialization)* 
','?)? ']'
Jul 28 2013
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
Like typedef alias supports multiple declarators.
aliasDeclaration:
'alias' (aliasInitializer (',' aliasInitializer)* | type 
declarator (',' declarator)*) ';'

Also declarator supports initializer, but alias doesn't.
declarator:
     Identifier ('=' initializer)?
     ;

alias int a=1;
Error: alias cannot have initializer
Jul 28 2013
parent "Kagamin" <spam here.lot> writes:
Well, knowing the lean and mean way dmd is written, alias 
initializer can be a semantical error rather than syntactical.
Jul 28 2013
prev sibling next sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 28.07.2013 00:27, schrieb Brian Schott:
 DScanner is a tool for analyzing D source code. It has the
 following features:

 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file

 The lexer/parser/AST are located in the "std/d" directory in the
 repository. These files should prove useful to anyone else
 working on D tooling.

 https://github.com/Hackerpilot/Dscanner

 Aside: the D grammar that I reverse-engineered can be located
 here:
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
would be nice to habe some sort of binary output (beside xml and json) to ease (and maybe speed-up) the usage
Jul 28 2013
prev sibling next sibling parent reply "qznc" <qznc web.de> writes:
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
 DScanner is a tool for analyzing D source code. It has the 
 following features:

 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file

 The lexer/parser/AST are located in the "std/d" directory in 
 the repository. These files should prove useful to anyone else 
 working on D tooling.

 https://github.com/Hackerpilot/Dscanner

 Aside: the D grammar that I reverse-engineered can be located 
 here: 
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Dscanner looks like a good starting point for a code formatting tool (like gofmt). However, there seems to be a tradeoff with performance involved. For compilation you want a fast lexer and parser. For formatting you need to preserve comments, though. For example, convert this from source to AST to source without losing the comments: void /*hello*/ /*world*/ main () { }
Jul 29 2013
parent reply Rory McGuire <rjmcguire gmail.com> writes:
Any chance of you turning this into a daemon? Something likt margo or
gocode?
On 29 Jul 2013 11:05, "qznc" <qznc web.de> wrote:

 On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:

 DScanner is a tool for analyzing D source code. It has the following
 features:

 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file

 The lexer/parser/AST are located in the "std/d" directory in the
 repository. These files should prove useful to anyone else working on D
 tooling.

 https://github.com/**Hackerpilot/Dscanner<https://github.com/Hackerpilot/Dscanner>

 Aside: the D grammar that I reverse-engineered can be located here:
 https://rawgithub.com/**Hackerpilot/DGrammar/master/**grammar.html<https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html>
Dscanner looks like a good starting point for a code formatting tool (like gofmt). However, there seems to be a tradeoff with performance involved. For compilation you want a fast lexer and parser. For formatting you need to preserve comments, though. For example, convert this from source to AST to source without losing the comments: void /*hello*/ /*world*/ main () { }
Jul 31 2013
parent reply Justin Whear <justin economicmodeling.com> writes:
On Wed, 31 Jul 2013 20:30:17 +0200, Rory McGuire wrote:

 Any chance of you turning this into a daemon? Something likt margo or
 gocode?
The author has another project here: https://github.com/Hackerpilot/DCD
Jul 31 2013
parent reply "Brian Schott" <briancschott gmail.com> writes:
On Wednesday, 31 July 2013 at 18:41:17 UTC, Justin Whear wrote:
 On Wed, 31 Jul 2013 20:30:17 +0200, Rory McGuire wrote:

 Any chance of you turning this into a daemon? Something likt 
 margo or
 gocode?
The author has another project here: https://github.com/Hackerpilot/DCD
I wouldn't bother trying to use that yet. Maybe next week, but not now. When I get it working there will be a thread on D.announce.
Jul 31 2013
parent Rory McGuire <rjmcguire gmail.com> writes:
okay, I look forward to its release.
I'm really looking forward to code completion in Sublime Text 2



On Wed, Jul 31, 2013 at 10:44 PM, Brian Schott <briancschott gmail.com>wrote:

 On Wednesday, 31 July 2013 at 18:41:17 UTC, Justin Whear wrote:

 On Wed, 31 Jul 2013 20:30:17 +0200, Rory McGuire wrote:

  Any chance of you turning this into a daemon? Something likt margo or
 gocode?
The author has another project here: https://github.com/**Hackerpilot/DCD<https://github.com/Hackerpilot/DCD>
I wouldn't bother trying to use that yet. Maybe next week, but not now. When I get it working there will be a thread on D.announce.
Aug 01 2013
prev sibling next sibling parent "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
 DScanner is a tool for analyzing D source code. It has the 
 following features:

 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file

 The lexer/parser/AST are located in the "std/d" directory in 
 the repository. These files should prove useful to anyone else 
 working on D tooling.

 https://github.com/Hackerpilot/Dscanner

 Aside: the D grammar that I reverse-engineered can be located 
 here: 
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Thanks Brian! I do not know about others, but I will definitely use these modules in my projects! :) Keep up with the good work.
Aug 01 2013
prev sibling next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Sun, 28 Jul 2013 00:27:34 +0200
"Brian Schott" <briancschott gmail.com> wrote:

 DScanner is a tool for analyzing D source code.
[...]
When I try to compile it (DMD 2.063.2 Win32) I get this: C:\Users\Nick\AppData\Roaming\dvm\compilers\dmd-2.063.2\bin\..\src\phobos\std\array.d(321): Error: function core.memory.GC.malloc (uint sz, uint ba = 0u) is not callable using argument types (ulong, BlkAttr) C:\Users\Nick\AppData\Roaming\dvm\compilers\dmd-2.063.2\bin\..\src\phobos\std\array.d(272): Error: template instance std.array.arrayAllocImpl!(false, ubyte[], ulong) error instantiating main.d(77): instantiated from here: uninitializedArray!(ubyte[], ulong) main.d(77): Error: template instance std.array.uninitializedArray!(ubyte[], ulong) error instantiating
Aug 01 2013
parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Thu, 1 Aug 2013 21:52:00 -0400
Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> wrote:

 On Sun, 28 Jul 2013 00:27:34 +0200
 "Brian Schott" <briancschott gmail.com> wrote:
 
 DScanner is a tool for analyzing D source code.
[...]
When I try to compile it (DMD 2.063.2 Win32) I get this: C:\Users\Nick\AppData\Roaming\dvm\compilers\dmd-2.063.2\bin\..\src\phobos\std\array.d(321): Error: function core.memory.GC.malloc (uint sz, uint ba = 0u) is not callable using argument types (ulong, BlkAttr) C:\Users\Nick\AppData\Roaming\dvm\compilers\dmd-2.063.2\bin\..\src\phobos\std\array.d(272): Error: template instance std.array.arrayAllocImpl!(false, ubyte[], ulong) error instantiating main.d(77): instantiated from here: uninitializedArray!(ubyte[], ulong) main.d(77): Error: template instance std.array.uninitializedArray!(ubyte[], ulong) error instantiating
Actually, I dug into this more (it was problem on 32-bit) and made a pull request: https://github.com/Hackerpilot/Dscanner/pull/42
Aug 01 2013
prev sibling next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
 DScanner is a tool for analyzing D source code. It has the 
 following features:

 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file

 The lexer/parser/AST are located in the "std/d" directory in 
 the repository. These files should prove useful to anyone else 
 working on D tooling.

 https://github.com/Hackerpilot/Dscanner

 Aside: the D grammar that I reverse-engineered can be located 
 here: 
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Any idea on when we might see json output(i am not a fan of xml)? Other than that this is a fantastic project! I am planing some projects in the future and this will be of great help. Keep up the good work!
Aug 02 2013
parent reply "Brian Schott" <briancschott gmail.com> writes:
On Friday, 2 August 2013 at 13:52:14 UTC, Tofu Ninja wrote:
 Any idea on when we might see json output(i am not a fan of 
 xml)?
Roughly the same time somebody submits a pull request. I'm currently focusing my spare time on DCD, so the JSON output will happen after I'm able to get auto-completion working.
Aug 02 2013
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Friday, 2 August 2013 at 18:01:01 UTC, Brian Schott wrote:
 On Friday, 2 August 2013 at 13:52:14 UTC, Tofu Ninja wrote:
 Any idea on when we might see json output(i am not a fan of 
 xml)?
Roughly the same time somebody submits a pull request. I'm currently focusing my spare time on DCD, so the JSON output will happen after I'm able to get auto-completion working.
I will look into adding it my self if I get some time, but I don't think I will need to use this for a while. For what I want it for, there is a lot of legwork to be done before I get around to needing this. Also roughly how difficult would it be to re-create source code from the xml? And does the xml preserve comments and if so does it do anything with ddoc?
Aug 02 2013
parent "Brian Schott" <briancschott gmail.com> writes:
On Friday, 2 August 2013 at 18:12:15 UTC, Tofu Ninja wrote:
 I will look into adding it my self if I get some time, but I 
 don't think I will need to use this for a while. For what I 
 want it for, there is a lot of legwork to be done before I get 
 around to needing this.
The XML output is handled by this class: https://github.com/Hackerpilot/Dscanner/blob/master/astprinter.d It shouldn't be much more difficult than changing the print statements.
 Also roughly how difficult would it be to re-create source code 
 from the xml? And does the xml preserve comments and if so does 
 it do anything with ddoc?
Aside from the comments, it should be possible to recreate the source from the AST. If it's not, there's a bug in the AST output. Comments are skipped when syntax checking or generating the AST.
Aug 02 2013
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Brian Schott:

 DScanner is a tool for analyzing D source code. It has the 
 following features:
 ...
 https://github.com/Hackerpilot/Dscanner
I have just compiled it on Windows32 and tried it. The compilation using the given bat has failed to link: OPTLINK (R) for Win32 Release 8.00.15 Copyright (C) Digital Mars 1989-2013 All rights reserved. http://www.digitalmars.com/ctg/optlink.html dscanner.obj(dscanner) Error 42: Symbol Undefined _D8analysis10ifelsesame15IfElseSameCheck6__ctorMFAyaZC8analysis10ifelsesame15IfElseSameCheck dscanner.obj(dscanner) Error 42: Symbol Undefined _D8analysis12constructors16ConstructorCheck7__ClassZ dscanner.obj(dscanner) Error 42: Symbol Undefined _D8analysis12constructors16ConstructorCheck6__ctorMFAyaZC8analysis12constructors16ConstructorCheck dscanner.obj(dscanner) Error 42: Symbol Undefined _D8analysis10ifelsesame15IfElseSameCheck7__ClassZ --- errorlevel 4 But I have built the DScanner successfully using the old "bud" tool. I have seen that this basic usage is not supported: dscanner -s *.d I have seen it doesn't support source code with unicode identifiers or chars. How do you enable/disable specific tests when you use -s? This code gives four problems to Dscanner: void main() { auto x = float(5); auto r = 1. + 2; int items[5]; import std.stdio; int.max.writeln; } In about 25_000 CLOC lines of my code DScanner has found several usages of the old-style alias syntax, that I will fix. Plus it has found three usages of the implicit string concatenation, and two of them are (the same) bug! Andrei Alexandrescu was very wrong to think that implicit string concatenation is a speck of dust. I am not going to close down that enhancement request. Bye, bearophile
Apr 22 2014
parent reply "Brian Schott" <briancschott gmail.com> writes:
On 04/22/2014 07:08 PM, bearophile wrote:
 This code gives four problems to Dscanner:
 
 void main() {
      auto x = float(5);
This is not valid. DMD and the grammar spec both do not allow this.
      auto r = 1. + 2;
Fixed. (https://issues.dlang.org/show_bug.cgi?id=12623)
      int items[5];
I don't support C-style declarations. DMD doesn't really support them either. (https://issues.dlang.org/show_bug.cgi?id=953)
      import std.stdio;
      int.max.writeln;
Fixed.
 }
 Andrei Alexandrescu was very wrong to think that implicit 
 string concatenation is a speck of dust. I am not going to 
 close down that enhancement request.
It's a very bad feature. Every time I've ever used it has been a bug.
Apr 24 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Brian Schott:

 This is not valid. DMD and the grammar spec both do not allow 
 this.
This was changed weeks ago. Now D accepts that code.
     int items[5];
I don't support C-style declarations. DMD doesn't really support them either. (https://issues.dlang.org/show_bug.cgi?id=953)
Then I suggest DScanner to support the half-C-declarations, because D compilers digests them just fine and they are very common in D code you will find in the wild. A code analyzer has to accept the real world code people write, otherwise it's much less useful :-) Bye, bearophile
Apr 24 2014
parent reply "Brian Schott" <briancschott gmail.com> writes:
On Thursday, 24 April 2014 at 21:36:24 UTC, bearophile wrote:
 This was changed weeks ago. Now D accepts that code.
You want tooling to support language features that aren't released?
 Then I suggest DScanner to support the half-C-declarations, 
 because D compilers digests them just fine and they are very 
 common in D code you will find in the wild. A code analyzer has 
 to accept the real world code people write, otherwise it's much 
 less useful :-)
Several people have indicated that they care about C-style declarations more than I do, and yet they still don't care enough to implement support for them and create a pull request.
Apr 24 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 24 April 2014 at 22:19:29 UTC, Brian Schott wrote:
 Then I suggest DScanner to support the half-C-declarations, 
 because D compilers digests them just fine and they are very 
 common in D code you will find in the wild. A code analyzer 
 has to accept the real world code people write, otherwise it's 
 much less useful :-)
Several people have indicated that they care about C-style declarations more than I do, and yet they still don't care enough to implement support for them and create a pull request.
I think if such support is ever to be implemented it should result in immediate analysis message with demand to change it to D style declarations and never do it again. This is an unfortunate legacy that should not be encouraged.
Apr 24 2014
prev sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Brian Schott:

 You want tooling to support language features that aren't 
 released?
I think it's nice for a language tool to try to follow closely the improvements in the language. But you are right, the update rhythms of DScanner are left to you.
 Several people have indicated that they care about C-style 
 declarations more than I do, and yet they still don't care 
 enough to implement support for them and create a pull request.
Sorry, I understand the feeling, I was just trying to help a little, underlying a missing part in DScanner. There are several open source projects I report bugs to, but I don't sent patched to. Bye, bearophile
Apr 24 2014
prev sibling next sibling parent Jos van Uden <usenet fwend.com> writes:
On 28-7-2013 0:27, Brian Schott wrote:
 DScanner is a tool for analyzing D source code. It has the following features:

 * Prints out a complete AST of a source file in XML format.
 * Syntax checks code and prints warning/error messages
 * Prints a listing of modules imported by a source file
 * Syntax highlights code in HTML format
 * Provides more meaningful "line of code" count than wc
 * Counts tokens in a source file

 The lexer/parser/AST are located in the "std/d" directory in the repository.
These files should prove useful to anyone else working on D tooling.

 https://github.com/Hackerpilot/Dscanner

 Aside: the D grammar that I reverse-engineered can be located here:
https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
If I do a styleCheck on a source file that contains a BOM, I get the following message: test.d(1:14)[error]: Expected ; instead of .
Apr 24 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 27 July 2013 at 22:27:35 UTC, Brian Schott wrote:
 * Prints out a complete AST of a source file in XML format.
Neat feature! I met two bugs that need some love: https://github.com/Hackerpilot/Dscanner/issues/168 https://github.com/Hackerpilot/Dscanner/issues/169
 Aside: the D grammar that I reverse-engineered can be located 
 here: 
 https://rawgithub.com/Hackerpilot/DGrammar/master/grammar.html
Thanks for making this available! Very useful.
Apr 26 2014