www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ide - RFC: Descent's Action Plan

reply Ary Borenszweig <ary esperanto.org.ar> writes:
Just to make my mind clear about how to go on with Descent, and to 
explain a conclusion I've arrived while making it, I wrote a small article:

http://www.dsource.org/projects/descent/wiki/ActionPlan

It talks about why Descent doesn't work as expected most of the time, 
what a solution for that is, and how languages with compile-time 
generation of symbols make writing an IDE harder (for me, of course).

You don't need to understand every bit of it in order to understand the 
idea, but I'll gladly explain antyhing that isn't clear (it'll help me, 
and possible future developers).

Any comment or suggestion (or even English corrections! :)) is very 
appreciated.
Mar 26 2008
next sibling parent reply Tim Burrell <tim timburrell.net> writes:
Informative article Ary!

A couple thoughts come to mind:


 If a module A depends on B, which in turns depends on B, semantic analysis
will not be done correctly.


I assume you meant if module A depends on B, which in turn depends on A?


 So if Descent wants to follow JDT's approach, it can't do semantic analysis
only on the needed bits. It must fully do it for involved modules, and just
then, begin symbol lookup, type resolution, etc. Even if selective imports are
used, the selected symbols may be compile-time generated. So performance, in
comparison with JDT, should be relatively bad.


I'm curious, is performance really a big issue?  I'm thinking that the 
analysis step only happens when a file is saved?  So the parsing / 
analysis step isn't run that often correct?  Personally I'd be willing 
to take a pause every so often if it meant robust, and correct 
auto-completion.

Maybe this could even be done in the background on a separate thread 
unbeknownst to the user, with the results of the updated syntax tree 
being available when the thread finishes processing?

At any rate, it was interesting to hear some of the problems you 
encountered while writing Descent.  Keep up the good work, it's a great 
project!
Mar 27 2008
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Tim Burrell wrote:

 Informative article Ary!
 
 A couple thoughts come to mind:
 

 If a module A depends on B, which in turns depends on B, semantic 
 analysis will not be done correctly.


 
 I assume you meant if module A depends on B, which in turn depends on A?


Yes, I've corrected this. Thanks.


 

 So if Descent wants to follow JDT's approach, it can't do semantic 
 analysis only on the needed bits. It must fully do it for involved 
 modules, and just then, begin symbol lookup, type resolution, etc. 
 Even if selective imports are used, the selected symbols may be 
 compile-time generated. So performance, in comparison with JDT, should 
 be relatively bad.


 
 I'm curious, is performance really a big issue?  I'm thinking that the 
 analysis step only happens when a file is saved?  So the parsing / 
 analysis step isn't run that often correct?  Personally I'd be willing 
 to take a pause every so often if it meant robust, and correct 
 auto-completion.


Actually, semantic analysis is performed each time you stop typing for a 
while, or when you request autocompletion. It is also performed when 
doing "go to definition". This is working like that in JDT, and that's 
why it's also like that in Descent. If it takes a few milliseconds, than 
it doesn't matter running it often because you will not notice it.

I guess performance won't be that bad because the lexer/parser stage is 
already done, and also semantic analysis is only needed for "surface" 
declarations (not in function bodies, for example, although if a 
function needs to be compile-time evaluated...).

Has any one done a performance analysis comparing 
lexing/parsing/semantic analysis times?


 
 Maybe this could even be done in the background on a separate thread 
 unbeknownst to the user, with the results of the updated syntax tree 
 being available when the thread finishes processing?
 
 At any rate, it was interesting to hear some of the problems you 
 encountered while writing Descent.  Keep up the good work, it's a great 
 project!
Mar 27 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Ary Borenszweig wrote:


 Tim Burrell wrote:

 If a module A depends on B, which in turns depends on B, semantic
 analysis will not be done correctly.


 
 I assume you meant if module A depends on B, which in turn depends on A?


 
 Yes, I've corrected this. Thanks.


I'm pretty certain that the D standard says that A depending on B, depending
on A is illegal...  This resolves issues with translation unit ordering
such as variable initialization and static construction.  It also makes
ordering of unit tests logical.  I think A depending on B depending on A
should be flagged as an error.
Mar 27 2008
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Jason House escribió:

 Ary Borenszweig wrote:
 

 Tim Burrell wrote:

 If a module A depends on B, which in turns depends on B, semantic
 analysis will not be done correctly.


 I assume you meant if module A depends on B, which in turn depends on A?


 Yes, I've corrected this. Thanks.


 
 I'm pretty certain that the D standard says that A depending on B, depending
 on A is illegal...  This resolves issues with translation unit ordering
 such as variable initialization and static construction.  It also makes
 ordering of unit tests logical.  I think A depending on B depending on A
 should be flagged as an error.


I meant something like this:

---
module one;

import two;

class Parent {

   Child child;

}
---
module two;

import one;

class Child {

   Parent parent;

}
---

DMD correctly compiles this, while in Descent the problem I mentioned 
appears. It can be solved with the current approach, but it involves 
writing semantic code besides the one provided in DMD. Luckily, today in 
the bus I figured out a very clean and nice way to program this, which 
shouldn't take a lot of time, without writing any additional semantic 
code. :)

The only unknown thing, yet, is how performance will be in this new 
approach...


Since you mentioned order of compilation: can a module, compiled to an 
obj file, expose different interfaces (that it, top level classes, 
functions, variables, etc.) when compiled in different orders?

Well, I've just came up with this example (anyone else have a more 
interesting one?), in which compilation order matters:

---
module one;

import two;

mixin("class Parent { Child child; }");
---
module two;

import one;

mixin("class Child { }");
---

$ dmd one.d two.d -c

(compiles ok)

$ dmd two.d one.d -c

Error: identifier 'Child' is not defined
Error: Child is used as a type
one.d: variable one.Parent.child voids have no value

When I came up with it, I thought I understood why the error was 
happening. But... I actually don't. Is this a bug?
Mar 27 2008
parent Christopher Wright <dhasenan gmail.com> writes:
Ary Borenszweig wrote:

 DMD correctly compiles this, while in Descent the problem I mentioned 
 appears. It can be solved with the current approach, but it involves 
 writing semantic code besides the one provided in DMD. Luckily, today in 
 the bus I figured out a very clean and nice way to program this, which 
 shouldn't take a lot of time, without writing any additional semantic 
 code. :)


The easy way is just to catenate the modules internally and do analysis 
on that. Circular dependencies within a module are already handled, 
reducing the problem to an already solved one.


 Error: identifier 'Child' is not defined
 Error: Child is used as a type
 one.d: variable one.Parent.child voids have no value
 
 When I came up with it, I thought I understood why the error was 
 happening. But... I actually don't. Is this a bug?


Yes.
Mar 27 2008
prev sibling next sibling parent Jason House <jason.james.house gmail.com> writes:
Ary Borenszweig wrote:


 Just to make my mind clear about how to go on with Descent, and to
 explain a conclusion I've arrived while making it, I wrote a small
 article:
 
 http://www.dsource.org/projects/descent/wiki/ActionPlan
 
 It talks about why Descent doesn't work as expected most of the time,
 what a solution for that is, and how languages with compile-time
 generation of symbols make writing an IDE harder (for me, of course).
 
 You don't need to understand every bit of it in order to understand the
 idea, but I'll gladly explain antyhing that isn't clear (it'll help me,
 and possible future developers).
 
 Any comment or suggestion (or even English corrections! :)) is very
 appreciated.


I may be underestimating the compile-time capability of D, but what if
top-level structure including basic information on the template conditions,
static if's, and version/debug statements required for a particular piece?

This would allow two levels of auto-completion.  The top-level data can say
under which conditions particular items are available.  Follow-up analysis
can say which are applicable for the current situation.  Knowing what other
options exist with different versions may actually help the programmer
decide on how to structure their current code to take advantage?

PS: Keep up the good work on Descent.  I look forward to seeing this stuff
in action :)
Mar 27 2008
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Ary Borenszweig wrote:

 Just to make my mind clear about how to go on with Descent, and to 
 explain a conclusion I've arrived while making it, I wrote a small article:
 
 http://www.dsource.org/projects/descent/wiki/ActionPlan
 
 It talks about why Descent doesn't work as expected most of the time, 
 what a solution for that is, and how languages with compile-time 
 generation of symbols make writing an IDE harder (for me, of course).
 
 You don't need to understand every bit of it in order to understand the 
 idea, but I'll gladly explain antyhing that isn't clear (it'll help me, 
 and possible future developers).
 
 Any comment or suggestion (or even English corrections! :)) is very 
 appreciated.


I think I more or less understood the problems you mention, so I want to 
offer some general toughts.

I think that trying to offer 100% correct completion and semantic 
analysis may not be the right way to go. I say this because doing might 
make the IDE too slow and memory-bloated, and make it's code much more 
complex, whereas having 99% correctness might make the IDE much more 
simple, and not make that much of a difference to the programmer. I say 
this based on the experiences of the lead CDT developer (see relevant 
post here: http://cdtdoug.blogspot.com/2007/03/ego-less-development.html)
Now, since D is simpler than C++, it might be easier and worthwhile to 
achieve that goal of 100% accurateness in the IDE, but given the 
manpower available for the Eclipse D IDE project, it might not be worth 
it yet. So I personally wouldn't mind much an IDE was not able to 
process compile-time generate symbols and definitions, there are other 
things I'd rather have first. But that's my personal opinion. :)


-- 
Bruno Medeiros - Software Developer, MSc. in CS/E graduate
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Apr 27 2008
parent Ary Borenszweig <ary esperanto.org.ar> writes:
Bruno Medeiros escribió:

 Ary Borenszweig wrote:

 Just to make my mind clear about how to go on with Descent, and to 
 explain a conclusion I've arrived while making it, I wrote a small 
 article:

 http://www.dsource.org/projects/descent/wiki/ActionPlan

 It talks about why Descent doesn't work as expected most of the time, 
 what a solution for that is, and how languages with compile-time 
 generation of symbols make writing an IDE harder (for me, of course).

 You don't need to understand every bit of it in order to understand 
 the idea, but I'll gladly explain antyhing that isn't clear (it'll 
 help me, and possible future developers).

 Any comment or suggestion (or even English corrections! :)) is very 
 appreciated.


 
 I think I more or less understood the problems you mention, so I want to 
 offer some general toughts.
 
 I think that trying to offer 100% correct completion and semantic 
 analysis may not be the right way to go. I say this because doing might 
 make the IDE too slow and memory-bloated, and make it's code much more 
 complex, whereas having 99% correctness might make the IDE much more 
 simple, and not make that much of a difference to the programmer. I say 
 this based on the experiences of the lead CDT developer (see relevant 
 post here: http://cdtdoug.blogspot.com/2007/03/ego-less-development.html)
 Now, since D is simpler than C++, it might be easier and worthwhile to 
 achieve that goal of 100% accurateness in the IDE, but given the 
 manpower available for the Eclipse D IDE project, it might not be worth 
 it yet. So I personally wouldn't mind much an IDE was not able to 
 process compile-time generate symbols and definitions, there are other 
 things I'd rather have first. But that's my personal opinion. :)


Actually... We already implemented it, and made some optimizations to 
DMD's semantic code (for some symbols semantic is never run, if not 
needed). It offers 100% correctness, and it works pretty fast. Say, 
about 300~500 milliseconds for large files (for example, import dfl.all) 
from the moment you request autocompletion until the moment the IDE 
offers them to you, so for normal-small files it should be fast. I'd 
rather wait 100 milliseconds more, than rely on my memory or check the 
source code (both much slower than 100 milliseconds).

We hope to release soon, and then we'll see if it's true based on user 
experience. At least I tried it inside some of Tang's, Phobo's and DFL's 
code, and it works quite well.

The problem with not offering the correct autocompletion proposals, is 
that you confuse the user. She would start thinking "Wait, I'm sure this 
class has this method, why isn't the IDE showing it to me? Maybe I'm 
wrong... let's check the source code". And then, you are back to 
nothing, because without an IDE, checking the source code or the 
documentation is the only option, besides memory.
Apr 27 2008