www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP18: Non-GC threads

reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
http://www.prowiki.org/wiki4d/wiki.cgi?LanguageDevel/DIPs/DIP18

The proposal is to create a base for safe, non garbage-collected 
threads. This is done by adding nogc attribute for functions. nogc 
functions cannot perform operations that may allocate garbage collected 
memory. They are covariant with gc ones.

Any comments are welcome.
Aug 31 2012
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Piotr Szturmaj:


 This is done by adding nogc attribute for functions. nogc 
 functions cannot perform operations that may allocate garbage 
 collected memory.


I suggest to call that  nogc.

Time ago I have suggested a " noheap" attribute for functions, 
that is similar to that "nogc", see the discussion there:
http://d.puremagic.com/issues/show_bug.cgi?id=5219

The difference is that  noheap disallows C functions like C 
malloc(), etc. Stack allocation like alloca() and variable length 
arrays is allowed in  noheap functions.

 noheap/ nogc is probably inferred for templated functions, as 
pure/nothrow.

The main disadvantage of an annotation like this is the 
proliferation of kinds of functions. The advantage is a more 
precise control of the effects of functions.

The Koka language has a function annotation similar to  noheap, I 
have discussed a little about Koka here:
http://forum.dlang.org/thread/ocxmiolhlcysehbjtcop forum.dlang.org
but Koka has type inference for effects, so often you don't need 
to add that annotation to functions.

Bye,
bearophile
Aug 31 2012
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
One more comment, a quotation from the DIP18:


Currently, all functions can use garbage collected memory. 
Threads that call those functions must be managed by the garbage 
collector, which may occasionally suspend them to perform 
collection. This can lead to unwanted pauses, that are not 
acceptable in some situations, mainly real-time audio/video 
processing, gaming, and others.<


C heap functions like malloc(), calloc(), realloc() have the same 
problem, their run-time is not deterministic, so if you don't 
want pauses a  noheap is useful.

 noheap is not meant to forbid calling low-level system-specific 
memory allocation functions.

Bye,
bearophile
Aug 31 2012
parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Saturday, 1 September 2012 at 02:52:23 UTC, bearophile wrote:

 One more comment, a quotation from the DIP18:


Currently, all functions can use garbage collected memory. 
Threads that call those functions must be managed by the 
garbage collector, which may occasionally suspend them to 
perform collection. This can lead to unwanted pauses, that are 
not acceptable in some situations, mainly real-time audio/video 
processing, gaming, and others.<


 C heap functions like malloc(), calloc(), realloc() have the 
 same problem, their run-time is not deterministic, so if you 
 don't want pauses a  noheap is useful.

  noheap is not meant to forbid calling low-level 
 system-specific memory allocation functions.

 Bye,
 bearophile


It looks very much that several annotations are actually directed 
at the compiler or the runtime for specific optimisations.
The   could be used (or are used) as local compiler switches or 
runtime hints.
Sep 02 2012
prev sibling parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
bearophile wrote:

 Piotr Szturmaj:


 This is done by adding nogc attribute for functions. nogc functions
 cannot perform operations that may allocate garbage collected memory.


 I suggest to call that  nogc.

 Time ago I have suggested a " noheap" attribute for functions, that is
 similar to that "nogc", see the discussion there:
 http://d.puremagic.com/issues/show_bug.cgi?id=5219


Thanks, I didn't know that. There are some good arguments for  nogc.


 The difference is that  noheap disallows C functions like C malloc(),
 etc. Stack allocation like alloca() and variable length arrays is
 allowed in  noheap functions.

  noheap/ nogc is probably inferred for templated functions, as
 pure/nothrow.


I would prefer  nogc, because nogc functions can manually allocate. Not 
using malloc, or preallocating using it is a lot easier than checking 
for GC allocations and manually managing non-gc threads.

The main reason under  nogc is not non-deterministic allocation of GC or 
malloc. The whole idea is to guarantee that nogc threads are not 
suspended by the GC.


 The main disadvantage of an annotation like this is the proliferation of
 kinds of functions. The advantage is a more precise control of the
 effects of functions.


The advantage is that you'd get the static checks similar to nothrow and 
pure.


 The Koka language has a function annotation similar to  noheap, I have
 discussed a little about Koka here:
 http://forum.dlang.org/thread/ocxmiolhlcysehbjtcop forum.dlang.org
 but Koka has type inference for effects, so often you don't need to add
 that annotation to functions.


Well, __traits(hasGCallocations, func) will also do the trick.
Sep 01 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 01-Sep-12 15:47, Piotr Szturmaj wrote:

 bearophile wrote:


 The Koka language has a function annotation similar to  noheap, I have
 discussed a little about Koka here:
 http://forum.dlang.org/thread/ocxmiolhlcysehbjtcop forum.dlang.org
 but Koka has type inference for effects, so often you don't need to add
 that annotation to functions.


 Well, __traits(hasGCallocations, func) will also do the trick.


Make it __traits(hasAttribute, "nogc", func) and that might be something.
Actually, I'd love to see some proposal that
a) clarifies how  attributes are supposed to be used
b) adds generic infrastructure to to use them via __traits or whatever

-- 
Olshansky Dmitry
Sep 01 2012
parent deadalnix <deadalnix gmail.com> writes:
Le 01/09/2012 14:29, Dmitry Olshansky a écrit :

 On 01-Sep-12 15:47, Piotr Szturmaj wrote:

 bearophile wrote:


 The Koka language has a function annotation similar to  noheap, I have
 discussed a little about Koka here:
 http://forum.dlang.org/thread/ocxmiolhlcysehbjtcop forum.dlang.org
 but Koka has type inference for effects, so often you don't need to add
 that annotation to functions.


 Well, __traits(hasGCallocations, func) will also do the trick.


 Make it __traits(hasAttribute, "nogc", func) and that might be something.
 Actually, I'd love to see some proposal that
 a) clarifies how  attributes are supposed to be used
 b) adds generic infrastructure to to use them via __traits or whatever


+1
Sep 01 2012
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
 If some of the function's arguments take references (directly or 
 indirectly), then all of them are automatically added to the list of GC 
 roots (addRoot()).


You still have a problem: you might have added an object passed as an 
argument to the GC roots, preventing the GC from collecting it, but if 
any thread is allowed to mutate a pointer inside this object pointing 
to some other data, using this other data in your thread is not safe, 
unless you have added a root for this other pointer too. You'd have to 
recursively add roots for this to work.

There's also the same issue with global variables, the ones which are 
pointers to other pointers.

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca/
Aug 31 2012
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Michel Fortin wrote:

 If some of the function's arguments take references (directly or
 indirectly), then all of them are automatically added to the list of
 GC roots (addRoot()).


 You still have a problem: you might have added an object passed as an
 argument to the GC roots, preventing the GC from collecting it, but if
 any thread is allowed to mutate a pointer inside this object pointing to
 some other data, using this other data in your thread is not safe,
 unless you have added a root for this other pointer too. You'd have to
 recursively add roots for this to work.

 There's also the same issue with global variables, the ones which are
 pointers to other pointers.


Yes, there is no obvious solution, but I think it could be done. For 
example non-immutable references may be disallowed at all. The other 
solution is to copy the whole referenced object using malloc, and add it 
to the GC roots.
Sep 01 2012
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2012-09-01 11:54:10 +0000, Piotr Szturmaj <bncrbme jadamspam.pl> said:


 Michel Fortin wrote:

 If some of the function's arguments take references (directly or
 indirectly), then all of them are automatically added to the list of
 GC roots (addRoot()).


 
 You still have a problem: you might have added an object passed as an
 argument to the GC roots, preventing the GC from collecting it, but if
 any thread is allowed to mutate a pointer inside this object pointing to
 some other data, using this other data in your thread is not safe,
 unless you have added a root for this other pointer too. You'd have to
 recursively add roots for this to work.
 
 There's also the same issue with global variables, the ones which are
 pointers to other pointers.


 
 Yes, there is no obvious solution, but I think it could be done. For 
 example non-immutable references may be disallowed at all.


That would be sufficient to enforce the guaranties, but it's a severe 
limitation that'll cause people to question the usefulness of the whole 
thing.


 The other solution is to copy the whole referenced object using malloc, 
 and add it to the GC roots.


Having to copy everything is also severe limitation I think. And even 
then it's not that simple to correctly copy a data structure if you 
want that to be done automatically. If you want it to be done manually, 
then how can you tell that the resulting pointer, and everything it 
points to, is not allocated from the GC heap?

-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca/
Sep 01 2012
prev sibling next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
I think this is an unnecessary addition.

I use D for game dev, and I avoid the GC + malloc like the 
plague, however I don't want to have to litter my code with 
distractions like nogc,  noheap or whatever keyword we choose.

It's easy enough to avoid these without expanding the type 
system. I add trace statements to the GC in druntime, and just 
avoid calling malloc. I think this is a more practical solution 
to this problem than extending the language and cluttering my 
code.
Sep 01 2012
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Peter Alexander wrote:

 I think this is an unnecessary addition.

 I use D for game dev, and I avoid the GC + malloc like the plague,
 however I don't want to have to litter my code with distractions like
 nogc,  noheap or whatever keyword we choose.

 It's easy enough to avoid these without expanding the type system. I add
 trace statements to the GC in druntime, and just avoid calling malloc. I
 think this is a more practical solution to this problem than extending
 the language and cluttering my code.


It's similar behavior to nothrow and pure. Instead of manually avoiding 
GC allocations, compiler does this checks for you. Imagine D doesn't 
have nothrow. You'd have to check every called function to see if it 
doesn't throw. In big programs throwing function may be left unnoticed 
and this is why we have static nothrow checks in D.
Sep 01 2012
parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 1 September 2012 at 11:37:39 UTC, Piotr Szturmaj 
wrote:

 It's similar behavior to nothrow and pure. Instead of manually 
 avoiding GC allocations, compiler does this checks for you. 
 Imagine D doesn't have nothrow. You'd have to check every 
 called function to see if it doesn't throw. In big programs 
 throwing function may be left unnoticed and this is why we have 
 static nothrow checks in D.


I understand the benefit. However, there are at least two 
significant costs:

1. If I want my entire program to be GC free, I have to annotate 
every single function with 'nogc'. This is not something I want 
to do.

2. It's a new language feature and has all the associated costs: 
initial implementation, bug fixing, marking up of functions in 
Phobos, documentation, etc.

Yes, with my approach, a rare allocation may go unnoticed, and 
you end up with an undesirable GC collection sometime in the 
future. It's not great, but it's not the end of the world, and 
I'm willing to risk that to avoid the costs I mentioned above.
Sep 01 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 01-Sep-12 16:27, Peter Alexander wrote:

 On Saturday, 1 September 2012 at 11:37:39 UTC, Piotr Szturmaj wrote:

 It's similar behavior to nothrow and pure. Instead of manually
 avoiding GC allocations, compiler does this checks for you. Imagine D
 doesn't have nothrow. You'd have to check every called function to see
 if it doesn't throw. In big programs throwing function may be left
 unnoticed and this is why we have static nothrow checks in D.


 I understand the benefit. However, there are at least two significant
 costs:

 1. If I want my entire program to be GC free, I have to annotate every
 single function with 'nogc'. This is not something I want to do.


I'd say
 nogc:
at the top and deal is sealed.


 2. It's a new language feature and has all the associated costs: initial
 implementation, bug fixing, marking up of functions in Phobos,
 documentation, etc.



 Yes, with my approach, a rare allocation may go unnoticed, and you end
 up with an undesirable GC collection sometime in the future. It's not
 great, but it's not the end of the world, and I'm willing to risk that
 to avoid the costs I mentioned above.



-- 
Olshansky Dmitry
Sep 01 2012
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 1 September 2012 at 12:39:41 UTC, Dmitry Olshansky 
wrote:

 I'd say
  nogc:
 at the top and deal is sealed.


BTW don't forget a  yesgc would be good to counter it. We need
a way to turn off each attribute to use this pattern best.
Sep 01 2012
next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
C:

int mul(int x, int y);


Future D:

pure  safe nothrow nogc commutative associative distributive 
 forceinline mayoverflow int mul(int x, int y);
Sep 01 2012
prev sibling next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
C:

int mul(int x, int y);


Future D:

pure  safe nothrow nogc commutative associative distributive 
 forceinline mayoverflow int mul(int x, int y);
Sep 01 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 01-Sep-12 19:58, Peter Alexander wrote:

 C:

 int mul(int x, int y);


 Future D:

 pure  safe nothrow nogc commutative associative distributive
  forceinline mayoverflow int mul(int x, int y);


No it's GNU C++ of today just replace 'pure' with __attribute__((pure))
and you are done :)
And don't even get me started at MS extensions...

Either way this is a kind of thing that only some users need but std 
library have to use all of them just so that those 'some users' can use it.

In fact if pure  safe nothrow could be combined in a user defined tag:

alias pure  safe nothrow nogc nice_func;

nice_func int mul(int x, int y);

and even C has it via macros, damn...

-- 
Olshansky Dmitry
Sep 01 2012
next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Saturday, 1 September 2012 at 16:08:08 UTC, Dmitry Olshansky 
wrote:


 In fact if pure  safe nothrow could be combined in a user 
 defined tag:

 alias pure  safe nothrow nogc nice_func;

 nice_func int mul(int x, int y);


That would be awesome.
Sep 02 2012
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On 1 September 2012 19:08, Dmitry Olshansky <dmitry.olsh gmail.com> wrote:


 On 01-Sep-12 19:58, Peter Alexander wrote:


 C:

 int mul(int x, int y);


 Future D:

 pure  safe nothrow nogc commutative associative distributive
  forceinline mayoverflow int mul(int x, int y);


 No it's GNU C++ of today just replace 'pure' with __attribute__((pure))
 and you are done :)
 And don't even get me started at MS extensions...

 Either way this is a kind of thing that only some users need but std
 library have to use all of them just so that those 'some users' can use it.

 In fact if pure  safe nothrow could be combined in a user defined tag:

 alias pure  safe nothrow nogc nice_func;

 nice_func int mul(int x, int y);

 and even C has it via macros, damn...



Aside the context of this thread, this would actually be REALLY useful. I
think I've asked about it a few times before.
There is already a fairly annoying problem where GDC supports GNU
attributes that DMD doesn't know about, which are basically impossible to
use; there's no nice way to use compiler-specific attributes without some
sort of alias like this.
I think the only current way is to duplicate the definitions in different
version blocks.
An attribute alias would be super handy, particularly when abstracting
support for multiple compilers.
Sep 03 2012
prev sibling parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Dmitry Olshansky wrote:

 On 01-Sep-12 19:58, Peter Alexander wrote:

 C:

 int mul(int x, int y);


 Future D:

 pure  safe nothrow nogc commutative associative distributive
  forceinline mayoverflow int mul(int x, int y);


 Either way this is a kind of thing that only some users need but std
 library have to use all of them just so that those 'some users' can use it.


The DIP states that nogc should be inferred by the compiler to avoid 
marking all existing functions.

But, I know there are some problems with inferring non templated 
methods: 
http://www.digitalmars.com/d/archives/digitalmars/D/Why_isn_t_purity_co._inferred_for_all_functions_166946.html.
Sep 03 2012
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 01-Sep-12 17:01, Adam D. Ruppe wrote:

 On Saturday, 1 September 2012 at 12:39:41 UTC, Dmitry Olshansky wrote:

 I'd say
  nogc:
 at the top and deal is sealed.


 BTW don't forget a  yesgc would be good to counter it. We need
 a way to turn off each attribute to use this pattern best.


What I see with this  nogc proposal is that that problem it tries to 
solve (and tool used to do so) is far more interesting and general.

Namely the problem is to specify that some functions can call only 
specific subset of functions and able to use only specific subset of 
language features. It's more far reaching then just gc, one may want to 
get  noblocking  or  async attribute to statically check e.g. that GUI 
thread can't ever block.
(it would however require to identify API calls that don't block, not 
possible on every OS, might entail some wrappers etc. but is very desirable)

To peek at what's more there that you can get this way see C++ AMP on 
why restricting a block of code or function is useful
e.g. here by Herb Sutter (including nice demo!):
http://www.youtube.com/watch?v=HK9rJFpX32Y

BTW  safe is in the same bucket, it does allow only to call  safe & 
 trusted (read as "subset of ") functions and use a subset of language 
features.

Actually exactly the same goes for pure (noglobal whatever) you just 
state this piece of code can't use this language feature (global 
variables, static variables, you name it), same for nothrow.

And not long ago David Nadlinger showed that function level trusted 
doesn't help much with templates that are  trusted for their own code, 
but need normal safety guarantee from foreign code brought by 
parameters. Thus restriction has to be able to work on a block scope too.

So what I want with this? I want to bring together C++ AMP restrict and 
 safe/ trusted/ system/pure/nothrow machinery in one clean and 
user-extensible future-proof feature.

The idea in a nutshell:

- specify a way attach tags to function declarations (to "paint" functions)

- allow specifying is-a relation between tags (supertags and subtags).

- add restrict specification that allows user to enforce on a block of 
code the following:
	a) disallow use of certain language features
	b) allow(or disallow) calling function with specific tags

- fit with existing world and provide clean migration path, specify defaults

I claim that it's enough to implement pure/safe/trusted/nothrow/nogc and 
proposed nogc.


The following is the draft of proposal.

1. Introduce a restrict keyword. Restrict applies to a block of code or 
a function:

auto foo(...) restrict(<restriction spec>)
{
	
}

auto bar(...)
{
	restrict(<restriction spec>){
	}
}

And the usual

restrict(<restrict spec>):

to affect all subsequent declarations in the module.

Multiple restrict blocks on the same declaration are treated as single 
restrict with restrict specifications concatenated.

The effect of such a clause is that declarations and statements inside 
of a block have to pass restriction check in order to compile.
The kind of check is described in restrict specification.

2. Functions declared with restrict(<restrict spec>) can only be bound 
by function pointer of more permissive type w.r.t. spec:

(assume A,B,C,D,E are elements of spec)
void funcABCD() restrict(A,B,C,D);
void funcACD() restrict(A,C,D);


function void() restrict(A,B,D) fp;
fp = &funcABCD; ///okay, fp is more permissive
fp = &funcACD; //fail, B is allowed inside of funcACD but not by fp

3. Function tags. Tag is a special literal associated with function 
declaration and type of function pointer. Any FP or function 
declarations can have an arbitrary amount of tags attached:

 tag(mytag, safe, nothrow)
void foobar();

function void ()  tag(default, nothrow)
Every function without tags has one implicit 'default' tag (name clashes 
with 'default' keyword may require another name). To only add tags 
include 'default' among others and to override it just don't list default.

User defined tags are specified as follows:

 tag tag_name; //new tag called tag_name

 tag tag_name : supertag; //tag_name is a subtag of super tag

The following tags are declared in object.d:

//pure is subtag of default, address of pure function can be assigned to 
default FP
 tag pure : default;
 tag system : default; //and system is subtag of default
 tag nothrow: default;
 tag trusted: system;
 tag safe: trusted;

Any assignments of FP allows only compatible pointer types.
Namely for the left hand side of assignment exp (LHS) and the right hand 
side (RHS):
all of RHS tags must have super tag or direct match in LHS's set of tags.

4. Restrict specifications

Introduce a list of language features that can be enabled/disabled with 
descriptive names, the list may be extended in the future.
Common suspects are:
	- heap stack frames and implicit GC actions
	- ability to declare/access kinds of variables (immutable, shared, 
static ...)
	- pointers to functions
	- pointers to ...
	- virtual functions
	- labels & goto
	- RTTI
	- inline assembler
	- pointer arithmetic
	- unions & reinterpret casts
	- C and D style varargs
	- other casts (integer/float coercing, const, dynamic)
	- exceptions
	...

each of those should have an IDs akin to __traits like 'virtual_calls', 
'gc' etc. Also all of the built-in names are reserved in a tags namespace.

Restrict specification is a comma separated list of:
1) tag - allow to call only functions that have tag 'tag' or have 
supertag of 'tag'

2) language_feature_id - disable feature with name language_feature_id

5. Backward compatibility.
 safe,  trusted,  system, nothrow and pure work as aliases respectively:
 safe -  tag(safe) restrict(safe)
 trusted -  tag(trusted) restrict(trusted)
 system -  tag(system) restrict(system)
pure -  tag(pure) restrict(globals, static, pure)
nothrow -  tag(nothrow) restrict(nothrow)

Sometime later these could be deprecated or left as simple shortcuts.

6. Probably we need a way to alias a list of restictions to ease the use 
of frequent combinations. (No idea, but any sane syntax around alias 
will do)

-- 
Olshansky Dmitry
Sep 01 2012
next sibling parent reply "foobar" <foo bar.com> writes:
On Saturday, 1 September 2012 at 16:20:14 UTC, Dmitry Olshansky
wrote:

 On 01-Sep-12 17:01, Adam D. Ruppe wrote:

 On Saturday, 1 September 2012 at 12:39:41 UTC, Dmitry 
 Olshansky wrote:

 I'd say
  nogc:
 at the top and deal is sealed.


 BTW don't forget a  yesgc would be good to counter it. We need
 a way to turn off each attribute to use this pattern best.


 What I see with this  nogc proposal is that that problem it 
 tries to solve (and tool used to do so) is far more interesting 
 and general.

 Namely the problem is to specify that some functions can call 
 only specific subset of functions and able to use only specific 
 subset of language features. It's more far reaching then just 
 gc, one may want to get  noblocking  or  async attribute to 
 statically check e.g. that GUI thread can't ever block.
 (it would however require to identify API calls that don't 
 block, not possible on every OS, might entail some wrappers 
 etc. but is very desirable)

 To peek at what's more there that you can get this way see C++ 
 AMP on why restricting a block of code or function is useful
 e.g. here by Herb Sutter (including nice demo!):
 http://www.youtube.com/watch?v=HK9rJFpX32Y

 BTW  safe is in the same bucket, it does allow only to call 
  safe &  trusted (read as "subset of ") functions and use a 
 subset of language features.

 Actually exactly the same goes for pure (noglobal whatever) you 
 just state this piece of code can't use this language feature 
 (global variables, static variables, you name it), same for 
 nothrow.

 And not long ago David Nadlinger showed that function level 
 trusted doesn't help much with templates that are  trusted for 
 their own code, but need normal safety guarantee from foreign 
 code brought by parameters. Thus restriction has to be able to 
 work on a block scope too.

 So what I want with this? I want to bring together C++ AMP 
 restrict and  safe/ trusted/ system/pure/nothrow machinery in 
 one clean and user-extensible future-proof feature.

 The idea in a nutshell:


<snip details>

I skipped the details but I do agree with the general sentiment.
This long post begs the question - Why a general annotation
mechanism isn't enough to justify adding so much complexity to
the language?

Me thinks that simply tagging malloc/alloca/GC.allocate/etc..
with a _user defined_ annotation should be sufficient without the
need to introduce all of the above syntax to the language. In
fact, I have a vague memory of reading about green/red marking of
code via templates in plain old c++. No extra syntax required.

Here's the link:
http://www.artima.com/cppsource/codefeaturesP.html
Sep 01 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 02-Sep-12 01:13, foobar wrote:
[snip]

 I skipped the details but I do agree with the general sentiment.
 This long post begs the question - Why a general annotation
 mechanism isn't enough to justify adding so much complexity to
 the language?


It's quite short compared to all of the features it replaces and it had 
to allow for backwards compatibility. And it had to leave place for 
other  annotations as I felt restricting blocks of code is not the only 
use case for annotations.

Also sadly it reiterates a lot of trivia around how tagged functions and 
pointers interact with each another.

About general annotation mechanism. Nobody yet put together a proper 
proposal (if any at all), all I see is people that keep saying: "it's 
simple - just add the general annotation mechanism!".



 Me thinks that simply tagging malloc/alloca/GC.allocate/etc..
 with a _user defined_ annotation should be sufficient without the
 need to introduce all of the above syntax to the language.


Simple tags have no much use on their own - need syntax to use them.
The compiler can deduce their meaning so you also need a way tell the 
compiler: this function can only use "yellow" and "green" functions. 
Then some tags need to be compatible (or convertible with others), then 
there is a need to bundle tags together and so on.

I agree that simply tagging functions with red/green is attractive and 
simple concept but to implement things like say  safe you need more then 
that. You need to tweak what compiler puts in there for you (and 
forbid/allow as it suits your needs).

  In

 fact, I have a vague memory of reading about green/red marking of
 code via templates in plain old c++. No extra syntax required.


But xxtra works obviously :)
The method entails carrying around tag arguments apparently. It could be 
labeled as creative but a very backward way to do something simple. More 
specifically it shows that this has to be a language feature so that 
developers can focus on using it instead of being busy implementing it 
by hand.


 Here's the link:
 http://www.artima.com/cppsource/codefeaturesP.html



-- 
Olshansky Dmitry
Sep 01 2012
parent reply "foobar" <foo bar.com> writes:
On Saturday, 1 September 2012 at 21:50:33 UTC, Dmitry Olshansky 
wrote:

 On 02-Sep-12 01:13, foobar wrote:
 [snip]

 I skipped the details but I do agree with the general


 sentiment.

 This long post begs the question - Why a general annotation
 mechanism isn't enough to justify adding so much complexity to
 the language?


 It's quite short compared to all of the features it replaces 
 and it had to allow for backwards compatibility. And it had to 
 leave place for other  annotations as I felt restricting blocks 
 of code is not the only use case for annotations.

 Also sadly it reiterates a lot of trivia around how tagged 
 functions and pointers interact with each another.

 About general annotation mechanism. Nobody yet put together a 
 proper proposal (if any at all), all I see is people that keep 
 saying: "it's simple - just add the general annotation 
 mechanism!".


This basically reiterates the original post without answering my 
question.
Regarding the general mechanism - Why do we have to re-invent the 
wheel?

even C++11 (with slight variations). My "proper proposal" would 



 Me thinks that simply tagging malloc/alloca/GC.allocate/etc..
 with a _user defined_ annotation should be sufficient without


 the

 need to introduce all of the above syntax to the language.


 Simple tags have no much use on their own - need syntax to use 
 them.
 The compiler can deduce their meaning so you also need a way 
 tell the compiler: this function can only use "yellow" and 
 "green" functions. Then some tags need to be compatible (or 
 convertible with others), then there is a need to bundle tags 
 together and so on.

 I agree that simply tagging functions with red/green is 
 attractive and simple concept but to implement things like say 
  safe you need more then that. You need to tweak what compiler 
 puts in there for you (and forbid/allow as it suits your needs).


We already have at least part of the syntax in D.
There is no need to "tweak the compiler", everything can be 
implemented in user code as shown in the link I provided. 
Including a proof of concept implementation in D, thanks to 
Bartosz.


  In

 fact, I have a vague memory of reading about green/red


 marking of

 code via templates in plain old c++. No extra syntax required.


 But xxtra works obviously :)
 The method entails carrying around tag arguments apparently. It 
 could be labeled as creative but a very backward way to do 
 something simple. More specifically it shows that this has to 
 be a language feature so that developers can focus on using it 
 instead of being busy implementing it by hand.


 Here's the link:
 http://www.artima.com/cppsource/codefeaturesP.html




Again, this is a proof of concept that shows that even current D 
is sufficient to implement this sort of restrictions. Add 
annotations to the mix and you could implement this with a better 
syntax without the need to add extra parameters to functions 
(which is quite an elegant solution imo)
No need for developers to implement this "by hand". Any language 
with a meta-data facility also provides standard annotations in 
its stdlib. If such restrictions are general purpose and useful 
enough they'll end up in Phobos anyway, allowing developers to 
concentrate on using them.
Sep 02 2012
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09/02/2012 12:08 PM, foobar wrote:

 This basically reiterates the original post without answering my question.
 Regarding the general mechanism - Why do we have to re-invent the wheel?


No need to re-invent the wheel, but it needs to be tweaked because the
vehicle it should be attached to has different features than other cars.
Sep 02 2012
prev sibling next sibling parent reply Maxim Fomin <maxim maxim-fomin.ru> writes:
2012/9/1 Dmitry Olshansky <dmitry.olsh gmail.com>:

 On 01-Sep-12 17:01, Adam D. Ruppe wrote:

 On Saturday, 1 September 2012 at 12:39:41 UTC, Dmitry Olshansky wrote:

 I'd say
  nogc:
 at the top and deal is sealed.



 BTW don't forget a  yesgc would be good to counter it. We need
 a way to turn off each attribute to use this pattern best.



 What I see with this  nogc proposal is that that problem it tries to solve
 (and tool used to do so) is far more interesting and general.


This reminds me custom attributes discussion which can be found in NG archive.
Requested language additions relevant to attributes may be divided
into three groups:

1 (tags). We want to tag some (likely function) declarations like
 nogc. This addition has several issues:
- can tags carry additional information (for e.x  tag(atrr1=opened))?
- are they part of type or not (similar problems with default arguments)?
- are they user-defined or entirely built in the language?
- how to distinguish between restricted and permissive semantic (when
tagged function care about
whether they call non-tagged functions or not)?

In this case possible solutions are to link user-defined enumeration
types to existing  syntax
or to create entirely new syntax for purpose of holding additional
information ( tag(atrr1=opened)
where  attr1 refers to enum attr1{} ). Simply tagged functions
( tag(name)) would have restrictive
semantics. Another possible solution would be that construction
" tag(attr1=value) type symbol;" is rewritten as
struct __name {type symbol; enum att1 = value; alias type this; } -
just not to add brand new features
which are hard to implement and easy to introduce bugs.

2 (extending type). We want to define type (likely classes) and then
extend its functionality.

this case (likely class) type implicitly derives
from attribute class. Obviously this also related to compile time
because D being static typed language and has no
way to be aware at run time that type functionality was extended. I
think that it is not worth embedding in the language, because
it is already possible to derive from base class, or interface, or
instantiated template or to use mixin.

3 (extending instance). We want to make some instance of (extremely
likely class) type to extend functionality
while still remaining in original type without affecting type.
Obviously this can be done only in run-time
where actual instance of type is created.

In this case possible solutions are to put associative array or some
container in druntime (object.Object) which export
attributes for e.x. through .attributeof property.

I offer simple solutions because I estimate probability of introducing
complex addition to the language now as
extremely low.
Sep 02 2012
parent reply "foobar" <foo bar.com> writes:
On Sunday, 2 September 2012 at 16:59:19 UTC, Maxim Fomin wrote:


 This reminds me custom attributes discussion which can be found 
 in NG archive.
 Requested language additions relevant to attributes may be 
 divided
 into three groups:

 1 (tags). We want to tag some (likely function) declarations 
 like  nogc.


IMO, this is a poor choice.


 2 (extending type). We want to define type (likely classes) and 
 then
 extend its functionality.

 wrong). In
 this case (likely class) type implicitly derives
 from attribute class. Obviously this also related to compile 
 time
 because D being static typed language and has no
 way to be aware at run time that type functionality was 
 extended. I
 think that it is not worth embedding in the language, because
 it is already possible to derive from base class, or interface, 
 or
 instantiated template or to use mixin.


This is my preferred solution. This is also similar to Java. In 

semantics. This also includes other languages on the matching 
platforms, E.g. Nemerle is a .net language and provides the same 

with Java annotations.


class Whatever : Annotation { // user defined annotation type
}

Java uses  interface to define a user defined annotation.
 interface whatever { // user defined annotation type
}

I don't get your last point at all regarding inheritance - in 

annotations. No need to add a new mechanism for it.
class Foo : whatever {} // inheritance

Really, the main things that are needed inside the compiler are a 
mechanism to attach custom attributes to various lingual parts, a 
way to mark such custom attributes by e.g. deriving from a 
special class (not unlike the IUnknown interface in D for COM) 
and standard compiler APIs for usage inside such an attribute.
One extra step is defining an execution model for custom 
attributes:
Java way is multi-pass - compiler executes the annotations which 
generate new version of the code, which is run through the 
compiler again, and so forth.
Nemerle way - attributes can be macros which in turn can 
manipulate the AST.
etc..


 3 (extending instance). We want to make some instance of 
 (extremely likely class) type to extend functionality
 while still remaining in original type without affecting type.


I don't get this at all. What's the purpose of having a single 
type?


 I offer simple solutions because I estimate probability of 
 introducing
 complex addition to the language now as
 extremely low.
Sep 02 2012
parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Monday, 3 September 2012 at 00:01:09 UTC, foobar wrote:

 2 (extending type). We want to define type (likely classes) 
 and then
 extend its functionality.

 wrong). In
 this case (likely class) type implicitly derives
 from attribute class. Obviously this also related to compile 
 time
 because D being static typed language and has no
 way to be aware at run time that type functionality was 
 extended. I
 think that it is not worth embedding in the language, because
 it is already possible to derive from base class, or 
 interface, or
 instantiated template or to use mixin.


 This is my preferred solution. This is also similar to Java. In 

 semantics. This also includes other languages on the matching 
 platforms, E.g. Nemerle is a .net language and provides the 

 same with Java annotations.


 class Whatever : Annotation { // user defined annotation type
 }

 Java uses  interface to define a user defined annotation.
  interface whatever { // user defined annotation type
 }

 I don't get your last point at all regarding inheritance - in 

 annotations. No need to add a new mechanism for it.
 class Foo : whatever {} // inheritance

 Really, the main things that are needed inside the compiler are 
 a mechanism to attach custom attributes to various lingual 
 parts, a way to mark such custom attributes by e.g. deriving 
 from a special class (not unlike the IUnknown interface in D 
 for COM) and standard compiler APIs for usage inside such an 
 attribute.
 One extra step is defining an execution model for custom 
 attributes:
 Java way is multi-pass - compiler executes the annotations 
 which generate new version of the code, which is run through 
 the compiler again, and so forth.
 Nemerle way - attributes can be macros which in turn can 
 manipulate the AST.
 etc..


The question with this approach is follows: if attributes are 
just another way of deriving why not use old syntax?


 3 (extending instance). We want to make some instance of 
 (extremely likely class) type to extend functionality
 while still remaining in original type without affecting type.


 I don't get this at all. What's the purpose of having a single 
 type?


It is not a single type, it is extending functionality of 
instance at run time which can be simulated by containers.
Sep 03 2012
parent "foobar" <foo bar.com> writes:
On Monday, 3 September 2012 at 16:21:08 UTC, Maxim Fomin wrote:

 The question with this approach is follows: if attributes are 
 just another way of deriving why not use old syntax?


I think you conflate two concepts - the custom attributes 
_themselves_ can use the old syntax for polymorphism and reuse. 
This is however completely orthogonal from the action of 
attaching a custom attribute to an object (function, variable, 
type, etc). This is close to AOP and cross cutting concerns.
an example (perhaps a bit convoluted, just to illustrate a point):
// I want to have all my logging related code in one place.
class Log : Attribute {}
// inheritance of attributes
class LegalLog : Log {} // for legal department
class DeveloperLog : Log {} // for developers

// usage
 LegalLog(params)
class Foo : Bar {}

 DeveloperLog(params)
class Goo : Foo {}

The idea is that all the specifics of logging are concentrated in 
one place and not spread throughout the application. Say I want 
to change logging file format, I can do it by simply changing the 
relevant attributes without touching the main business logic of 
my application. It make no sense in this scenario to derive my 
business object classes from the Logging classes.
Sep 03 2012
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Saturday, 1 September 2012 at 16:20:14 UTC, Dmitry Olshansky 
wrote:
....

 6. Probably we need a way to alias a list of restictions to 
 ease the use of frequent combinations. (No idea, but any sane 
 syntax around alias will do)


So you can tailor precisely the language to your own 
requirements, by basically inserting compiler switches in the 
code.
That is a very smart idea, but I don't see how this could play 
well with most of the std lib.
Sep 02 2012
prev sibling next sibling parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Dmitry Olshansky wrote:

 On 01-Sep-12 17:01, Adam D. Ruppe wrote:

 On Saturday, 1 September 2012 at 12:39:41 UTC, Dmitry Olshansky wrote:

 I'd say
  nogc:
 at the top and deal is sealed.


 BTW don't forget a  yesgc would be good to counter it. We need
 a way to turn off each attribute to use this pattern best.


 What I see with this  nogc proposal is that that problem it tries to
 solve (and tool used to do so) is far more interesting and general.

 Namely the problem is to specify that some functions can call only
 specific subset of functions and able to use only specific subset of
 language features. It's more far reaching then just gc, one may want to
 get  noblocking  or  async attribute to statically check e.g. that GUI
 thread can't ever block.
 (it would however require to identify API calls that don't block, not
 possible on every OS, might entail some wrappers etc. but is very
 desirable)


[snip]


 The idea in a nutshell:


[snip]

How the attribute inferring works in this proposal?
Sep 03 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 03-Sep-12 16:32, Piotr Szturmaj wrote:

 Dmitry Olshansky wrote:

 On 01-Sep-12 17:01, Adam D. Ruppe wrote:

 On Saturday, 1 September 2012 at 12:39:41 UTC, Dmitry Olshansky wrote:

 I'd say
  nogc:
 at the top and deal is sealed.


 BTW don't forget a  yesgc would be good to counter it. We need
 a way to turn off each attribute to use this pattern best.


 What I see with this  nogc proposal is that that problem it tries to
 solve (and tool used to do so) is far more interesting and general.

 Namely the problem is to specify that some functions can call only
 specific subset of functions and able to use only specific subset of
 language features. It's more far reaching then just gc, one may want to
 get  noblocking  or  async attribute to statically check e.g. that GUI
 thread can't ever block.
 (it would however require to identify API calls that don't block, not
 possible on every OS, might entail some wrappers etc. but is very
 desirable)


 [snip]


 The idea in a nutshell:


 [snip]

 How the attribute inferring works in this proposal?


Interesting... and it seems like user-defined tags can't be inferred or 
that it would need more thought.

However restrictions can be inferred by observing which things which 
template instance does and the tags of functions it calls. It's exactly 
the same as now but every restriction out of a set (say  safe) now get 
inferred separately.

-- 
Olshansky Dmitry
Sep 03 2012
prev sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On 2012-09-01, 18:20, Dmitry Olshansky wrote:


 The idea in a nutshell:

 - specify a way attach tags to function declarations (to "paint"  
 functions)

 - allow specifying is-a relation between tags (supertags and subtags).

 - add restrict specification that allows user to enforce on a block of  
 code the following:
 	a) disallow use of certain language features
 	b) allow(or disallow) calling function with specific tags

 - fit with existing world and provide clean migration path, specify  
 defaults

 I claim that it's enough to implement pure/safe/trusted/nothrow/nogc and  
 proposed nogc.



I'm sorry I have not gotten around to commenting on this before, but I
*really* like this proposal. As foobar has said, a general annotation
system would be real nice, but as far as I can see, it can be built on
top of this, at a later point.

Also, I cannot see that this has been turned into a proper DIP. :p

http://prowiki.org/wiki4d/wiki.cgi?LanguageDevel/DIPs

-- 
Simen
Sep 22 2012
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Saturday, 1 September 2012 at 00:40:23 UTC, Piotr Szturmaj 
wrote:

 http://www.prowiki.org/wiki4d/wiki.cgi?LanguageDevel/DIPs/DIP18


How this nogc attribute will work with phobos and druntime?
Sep 05 2012
parent Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Kagamin wrote:

 On Saturday, 1 September 2012 at 00:40:23 UTC, Piotr Szturmaj wrote:

 http://www.prowiki.org/wiki4d/wiki.cgi?LanguageDevel/DIPs/DIP18


 How this nogc attribute will work with phobos and druntime?


I don't know what do you mean exactly. Some of the phobos and druntime 
functions will certainly not work, but some will do. Also, I think that 
nogc/pure/nothrow should be inferred, so they can be used without 
marking so many functions.
Sep 05 2012