www.digitalmars.com         C & C++   DMDScript  

digitalmars.dip.ideas - Add Implicit Context System

reply jmh530 <john.michael.hall gmail.com> writes:
D should consider adding an implicit context system.



In `std.experimental.allocator`, key functions like `make` have 
`auto ref Allocator alloc` as the first item in the function 
parameter list. From the documentation for the module, it would 
get used like below

```
int* p = theAllocator.make!int(42);
```

this means that if there is a a function that calls functions 
like `make`, it is generally recommended to be consistent and 
also pass `auto ref Allocator alloc` as the first parameter.

The implication of this is that all of the user's function calls 
end up looking like `theallocator.funcA(x)` instead of `x.funcA`. 
This is less intuitive for most purposes and ends up complicating 
the way a lot of code looks.







The default calling convention in Odin passes an implicit context 
pointer on each call [1].

 From the documentation:


 In each scope, there is an implicit value named context. This 
 context variable is local to each scope and is implicitly 
 passed by pointer to any procedure call in that scope (if the 
 procedure has the Odin calling convention).



 The main purpose of the implicit context system is for the 
 ability to intercept third-
 party code and libraries and modify their functionality.


They provide the following example:

```odin
main :: proc() {
	c := context // copy the current scope's context

	context.user_index = 456
	{
		context.allocator = my_custom_allocator()
		context.user_index = 123
		supertramp() // the `context` for this scope is implicitly 
passed to `supertramp`
	}

	// `context` value is local to the scope it is in
	assert(context.user_index == 456)
}

supertramp :: proc() {
	c := context // this `context` is the same as the parent 
procedure that it was called from
	// From this example, context.user_index == 123
	// A context.allocator is assigned to the return value of 
`my_custom_allocator()`

	// The memory management procedure uses the `context.allocator` 
by default unless explicitly specified otherwise
	ptr := new(int)
	free(ptr)
}
```

The `Context` struct appears in their runtime library [2] and 
looks like below.

```
Context :: struct {
	allocator:              Allocator,
	temp_allocator:         Allocator,
	assertion_failure_proc: Assertion_Failure_Proc,
	logger:                 Logger,
	random_generator:       Random_Generator,
	user_ptr:               rawptr,
	user_index:             int,
	// Internal use only
	_internal:              rawptr,
}
```

It includes two fields associated with allocators, but also 
fields associated with assertion handling, logging, random number 
generators, and some others.



With respect to allocators [3], a call to `ptr := new(int)` in 
Odin is equivalent to `ptr := new(int, context.allocator)`.

The context system stores two different kinds of allocation. The 
`context.allocator`, which is a general allocator assigned from 
the OS, and a `context.temp_allocator`, which is meant for 
temporary short-lived allocations and is a growing arena-based 
allocator (`free_all` must be used to free it).



To incorporate this kind of system into D without breaking any 
existing code requires two main things:
1. make an implicit, scope-level `Context` value,
2. allow users to opt-in to this type of calling convention.

On 1, the way that it works in Odin is that each lexical scope 
has an implicit identifier named `context`. That `context` is 
local to the scope. So for instance, something like
```
{
     int a;
}
```
would get re-written to something like
```
{
     auto local_context = parent_context.copy;
     int a;
}
```
But from the user's perspective, it would all be called 
`context`. So that implies it would be a little bit special. For 
instance, the user should be able to do

```
{
     auto c = context; // make a copy of the current context
     context.user_index = 456; // set the user_index (or something 
similar)
     {
         // implicitly a new context is created here copy-on-write 
from the parent
         context.allocator = my_custom_allocator();
         context.user_index = 123; // the local context is 
overwriting its user_index
     }
     assert(context.user_index == 456); // but the parent context 
is NOT overwritten
}
```

In terms of performance, Odin uses a copy-on-write approach in 
order to limit copies. This could be implemented for D too in the 
compiler. But if the user isn't using context anywhere, then the 
compiler should also be optimizing it away.

On 2, users would need to opt-in to this type of calling 
convention. functions would still default to `extern(D)`, but 
users could optionally declare something like 
`extern(D_context)`, which is exactly the same as `extern(D)`, 
except that it implicitly passes a pointer to the context 
parameter as the final parameter in the function parameter list. 
This would basically require adding a new linkage type.

In addition, as discussed above, the Odin `Context` struct is 
defined in the runtime library. We would also need to add 
something similar. Their `Context` also enables logging and other 
functionality to be incorporated in. How much functionality we 
would want to enable with this depends on how we construct the 
`Context` struct (of which `context` would be an instance). I 
would think it could make sense to copy from their approach. One 
complication of this is that their `Context` struct has fields 
that are an `Allocator` type. In D, we don't have the same kind 
of `Allocator` type and instead our version would likely need to 
be templated.

[1] https://odin-lang.org/docs/overview/#implicit-context-system
[2] https://pkg.odin-lang.org/base/runtime/#Context
[3] https://odin-lang.org/docs/overview/#allocators
Dec 21 2025
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
I wrote this elsewhere, but there is a simpler approach to this, that 
I've slightly copied from Swift.

When you access a global variable, copy it and reference that instead.

```d
private State* gstate;

struct Thing {
	private {
		State state;
		State* old;
	}

	this(ref Thing other) {
		old = gstate;
		gstate = &state;
	}

	~this() {
		state = old;	
	}
}

 implicitCopy
Thing thing;

void usage() {
	// Thing temp = .thing;
	thing.write(); // temp.write();
}
```

No need to mess around with calling conventions, and this is fully in 
user control.
Dec 21 2025
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 22 December 2025 at 02:16:03 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 I wrote this elsewhere, but there is a simpler approach to 
 this, that I've slightly copied from Swift.


Do you mind pointing me in the direction where you wrote about 
this elsewhere? Thanks.
Dec 26 2025
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 27/12/2025 5:09 AM, jmh530 wrote:

 On Monday, 22 December 2025 at 02:16:03 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 I wrote this elsewhere, but there is a simpler approach to this, that 
 I've slightly copied from Swift.


 
 Do you mind pointing me in the direction where you wrote about this 
 elsewhere? Thanks.


https://forum.dlang.org/post/10i07v1$2c14$1 digitalmars.com
Dec 26 2025
parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 26 December 2025 at 19:01:16 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 On 27/12/2025 5:09 AM, jmh530 wrote:

 On Monday, 22 December 2025 at 02:16:03 UTC, Richard (Rikki) 
 Andrew Cattermole wrote:

 I wrote this elsewhere, but there is a simpler approach to 
 this, that I've slightly copied from Swift.


 
 Do you mind pointing me in the direction where you wrote about 
 this elsewhere? Thanks.


 https://forum.dlang.org/post/10i07v1$2c14$1 digitalmars.com


Thanks. My issue with this is that you still need to refer to 
‘thing’. Would you want some kind of super-powered ‘with’ 
statement.
Dec 26 2025
prev sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Monday, 22 December 2025 at 02:16:03 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 I wrote this elsewhere, but there is a simpler approach to 
 this, that I've slightly copied from Swift.

 When you access a global variable, copy it and reference that 
 instead.

 ```d
 private State* gstate;

 struct Thing {
 	private {
 		State state;
 		State* old;
 	}

 	this(ref Thing other) {
 		old = gstate;
 		gstate = &state;
 	}

 	~this() {
 		state = old;	
 	}
 }

  implicitCopy
 Thing thing;


How is this better than copy on write?
Jan 06
prev sibling next sibling parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
On Monday, 22 December 2025 at 02:06:17 UTC, jmh530 wrote:

 D should consider adding an implicit context system.


I absolutely agree.

However the real power is in being able to easily specify what 
you want done when you enter or exit the context, including the 
context initialisation (that can fail for various reasons).

On top of that context is perfect for width statement that should 
be used to directly access context values.

I know there will be people who will argue that we do not need 
this as we have scope blocks, and there will be another fraction 
that believes RIIA is the way to go. Both have pros-and cons, 
while properly done context handling covers all use-cases I think.
Dec 22 2025
parent reply Kapendev <alexandroskapretsos gmail.com> writes:
On Monday, 22 December 2025 at 14:34:24 UTC, Dejan Lekic wrote:

 On Monday, 22 December 2025 at 02:06:17 UTC, jmh530 wrote:

 D should consider adding an implicit context system.


 I absolutely agree.

 However the real power is in being able to easily specify what 
 you want done when you enter or exit the context, including the 
 context initialisation (that can fail for various reasons).

 On top of that context is perfect for width statement that 
 should be used to directly access context values.

 I know there will be people who will argue that we do not need 
 this as we have scope blocks, and there will be another 
 fraction that believes RIIA is the way to go. Both have 
 pros-and cons, while properly done context handling covers all 
 use-cases I think.


Rikki's idea is fine if people **really** need something like 
this.
Personally, I think that if someone wants to deal with allocators 
and a generic allocator API, it should be done explicitly (like 
Zig). It's much simpler and easier to control. No weird edge 
cases where X doesn't work, Y is missing or Z is unsafe.

Also from a library design point of view, having something like 
this means every library ends up with an extra API member that an 
user can just use. Same X, Y and Z apply here. Where are the 
`private` fans when you need them XD
Dec 22 2025
next sibling parent reply sighoya <sighoya gmail.com> writes:
On Monday, 22 December 2025 at 17:31:02 UTC, Kapendev wrote:

 Also from a library design point of view, having something like 
 this means every library ends up with an extra API member that 
 an user can just use. Same X, Y and Z apply here. Where are the 
 `private` fans when you need them XD


Can't they just push and pop a new context and ignoring the 
provided one in such a case?
Dec 22 2025
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Monday, 22 December 2025 at 17:34:03 UTC, sighoya wrote:

 On Monday, 22 December 2025 at 17:31:02 UTC, Kapendev wrote:

 Also from a library design point of view, having something 
 like this means every library ends up with an extra API member 
 that an user can just use. Same X, Y and Z apply here. Where 
 are the `private` fans when you need them XD


 Can't they just push and pop a new context and ignoring the 
 provided one in such a case?


What if the library takes a user-supplied callback, which might 
modify the context? Or calls a method of a class that the user 
might have overriden, or instantiates a template with a 
user-supplied type.

In theory, if you write your library code carefully enough, you 
can guard against this sort of thing. In practice, there are so 
many opportunities for mistakes that even the most vigilant 
programmer will inevitably let some slip through.
Dec 22 2025
prev sibling parent Kapendev <alexandroskapretsos gmail.com> writes:
On Monday, 22 December 2025 at 17:34:03 UTC, sighoya wrote:

 On Monday, 22 December 2025 at 17:31:02 UTC, Kapendev wrote:

 Also from a library design point of view, having something 
 like this means every library ends up with an extra API member 
 that an user can just use. Same X, Y and Z apply here. Where 
 are the `private` fans when you need them XD


 Can't they just push and pop a new context and ignoring the 
 provided one in such a case?


You could, but that would remove the ability to "intercept 
third-party code", which seems to be the goal behind this idea.
Dec 22 2025
prev sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 23/12/2025 6:31 AM, Kapendev wrote:

 On Monday, 22 December 2025 at 14:34:24 UTC, Dejan Lekic wrote:

 On Monday, 22 December 2025 at 02:06:17 UTC, jmh530 wrote:

 D should consider adding an implicit context system.


 I absolutely agree.

 However the real power is in being able to easily specify what you 
 want done when you enter or exit the context, including the context 
 initialisation (that can fail for various reasons).

 On top of that context is perfect for width statement that should be 
 used to directly access context values.

 I know there will be people who will argue that we do not need this as 
 we have scope blocks, and there will be another fraction that believes 
 RIIA is the way to go. Both have pros-and cons, while properly done 
 context handling covers all use-cases I think.


 
 Rikki's idea is fine if people **really** need something like this.
 Personally, I think that if someone wants to deal with allocators and a 
 generic allocator API, it should be done explicitly (like Zig). It's 
 much simpler and easier to control. No weird edge cases where X doesn't 
 work, Y is missing or Z is unsafe.
 
 Also from a library design point of view, having something like this 
 means every library ends up with an extra API member that an user can 
 just use. Same X, Y and Z apply here. Where are the `private` fans when 
 you need them XD


It is a lot worse than what has been realized just yet.

That custom calling convention? Yeah that is highly infectious.
Every function has to be annotated with it.

Once applied it basically cannot come off and it is explicit, it cannot 
be inferred.
Dec 22 2025
next sibling parent reply Guillaume Piolat <first.nam_e gmail.com> writes:
On Monday, 22 December 2025 at 19:21:33 UTC, Richard (Rikki) 
Andrew Cattermole wrote:


 It is a lot worse than what has been realized just yet.

 That custom calling convention? Yeah that is highly infectious.
 Every function has to be annotated with it.

 Once applied it basically cannot come off and it is explicit, 
 it cannot be inferred.


https://code.dlang.org/packages/implicit-context

The problem with implicit context is that once done it's not a 
whole lot better than TLS.
Perhaps a convention for what TLS like theAllocator should 
provide (such as pushAllocator/popAllocator) would replace the 
need for implicit context, if need there is.

...perhaps the compiler should auto-generates those function if 
the TLS variable starts with "theXXXX" :)

But the calling convention having one parameter just for this 
features is not a clear win.
Dec 26 2025
parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 26 December 2025 at 11:25:42 UTC, Guillaume Piolat 
wrote:

 On Monday, 22 December 2025 at 19:21:33 UTC, Richard (Rikki) 
 Andrew Cattermole wrote:


 [...]


 https://code.dlang.org/packages/implicit-context

 The problem with implicit context is that once done it's not a 
 whole lot better than TLS.
 Perhaps a convention for what TLS like theAllocator should 
 provide (such as pushAllocator/popAllocator) would replace the 
 need for implicit context, if need there is.

 ...perhaps the compiler should auto-generates those function if 
 the TLS variable starts with "theXXXX" :)

 But the calling convention having one parameter just for this 
 features is not a clear win.


Interesting.
Dec 26 2025
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 22 December 2025 at 19:21:33 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 On 23/12/2025 6:31 AM, Kapendev wrote:

 On Monday, 22 December 2025 at 14:34:24 UTC, Dejan Lekic wrote:

 [...]


 
 Rikki's idea is fine if people **really** need something like 
 this.
 Personally, I think that if someone wants to deal with 
 allocators and a generic allocator API, it should be done 
 explicitly (like Zig). It's much simpler and easier to 
 control. No weird edge cases where X doesn't work, Y is 
 missing or Z is unsafe.
 
 Also from a library design point of view, having something 
 like this means every library ends up with an extra API member 
 that an user can just use. Same X, Y and Z apply here. Where 
 are the `private` fans when you need them XD


 It is a lot worse than what has been realized just yet.

 That custom calling convention? Yeah that is highly infectious.
 Every function has to be annotated with it.

 Once applied it basically cannot come off and it is explicit, 
 it cannot be inferred.


What if you make the rule that if a function doesn't have an 
extern declaration, it is inferred to be `extern(D)` unless it 
calls an `extern(D_context)` (or whatever) function and then it 
is `extern(D_context)`?
Dec 26 2025
prev sibling next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 22 December 2025 at 02:06:17 UTC, jmh530 wrote:

 D should consider adding an implicit context system.



 [...]


```d
with(theAllocator) {
     auto p = make!int(42);
}
```
Jan 05
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 05/01/2026 11:15 PM, Atila Neves wrote:

 On Monday, 22 December 2025 at 02:06:17 UTC, jmh530 wrote:

 D should consider adding an implicit context system.



 [...]


 
 ```d
 with(theAllocator) {
      auto p = make!int(42);
 }
 ```


That does not replace the stack data structure supporting a push and a pop.

Nor is that being requested.
Jan 05
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 5 January 2026 at 10:15:13 UTC, Atila Neves wrote:

 On Monday, 22 December 2025 at 02:06:17 UTC, jmh530 wrote:

 D should consider adding an implicit context system.



 [...]


 ```d
 with(theAllocator) {
     auto p = make!int(42);
 }
 ```


When I run the following in run.dlang.io
```
import std.experimental.allocator;

void main() {
     with(theAllocator) {
         auto p = make!int(42);
     }
}
```
This the result:

```
/dlang/dmd/linux/bin64/../../src/phobos/std/experimental/alloca
or/package.d(1183): Error: no property `allocate` for `alloc` of type `int`
         auto m = alloc.allocate(max(stateSize!T, 1));
                       ^
/dlang/dmd/linux/bin64/../../src/phobos/std/experimental/alloca
or/package.d(1210): Error: no property `deallocate` for `alloc` of type `int`
                 alloc.deallocate(m);
                      ^
onlineapp.d(7): Error: template instance 
`std.experimental.allocator.make!(int, int)` error instantiating
         auto p = make!int(42);
```

My understanding here is that `with` changes implicit name look 
up, but it doesn't just inject `theAllocator` as the implicit 
first argument into functions. That's why it works for members 
and member functions.

Regardless, supposing that were somehow changed, the `with` 
statement wouldn't help you if you had another function in the 
with statement that you want to use the same allocator (`helper1` 
in code below). You would still need to make it explicit if you 
want the other functions to use `theAllocator` (`helper2`).

```
import core.memory;
import std.experimental.allocator;

void helper1() {
     auto q = make!int(64);
}
void helper2(Allocator)(Allocator a) {
     auto q = a.make!int(64);
}

void main() {
     with (theAllocator) {
         helper1(); // error
         helper2(theAllocator); // have to be explicit, what's the 
point?
     }
}
```
Jan 05
prev sibling parent reply libxmoc <libxmoc gmail.com> writes:
Implicit context is a disaster for readability.

It hides dependencies, breaks determinism, and makes testing a 
nightmare. You can't tell what a function needs just by looking 
at it.

Explicit parameters are honest. If it's required, it belongs in 
the signature.
Jan 14
parent user1234 <user1234 12.de> writes:
On Wednesday, 14 January 2026 at 16:26:44 UTC, libxmoc wrote:

 Implicit context is a disaster for readability.

 It hides dependencies, breaks determinism, and makes testing a 
 nightmare. You can't tell what a function needs just by looking 
 at it.

 Explicit parameters are honest. If it's required, it belongs in 
 the signature.


the lifting trick...

problem is: you want to have higienic lambdas. The thing that's 
been discussed here is the context, "the context could carry a 
pointer to an allocator"... but you actually raise the problem of 
captures, which is very important for lambdas, i.e function as 
expressions. Then they become much less appealing.
Jan 15