digitalmars.dip.ideas - Objective-C Interop Extensions
- Luna (48/48) Nov 16 ## Proposal
- Nicholas Wilson (27/76) Nov 16 This would be easy to do, synchronised works in pretty much the
- Luna (16/104) Nov 16 There's a few subtleties that would make this change in
- Luna (16/65) Nov 16 I will add that the code I shared on the D Language Code Club
- Richard (Rikki) Andrew Cattermole (56/56) Nov 16 Me and Luna have talked a bit on Discord about this, and I've done quite...
Currently DLang has, in some compilers, support for ObjectiveC types being declared. However, modern Objective-C code is strongly dependent on autorelease and auto release pools. Without them Objective-C code leaks memory (see: https://developer.apple.com/documentation/quartzcore/cametallayer?language=objc). To alleviate this, clang implements the concept of an ` autoreleasepool` block, this block frees all Objective-C classes which have called `autorelease`. In the case of Metal, for example, a lot of internal data uses autorelease calls. The ` autoreleasepool` blocks in clang are implemented via 2 helper functions declared in the objective-c runtime `libobjc.dylib` or the like in alternate runtime implementations. ```d extern(C) extern void* objc_autoreleasePoolPush(); extern(C) extern void objc_autoreleasePoolPop(void*); ``` My proposal is to add 2 new hooks to the D runtime emitted by the compiler as well as a new keyword, `autoreleasepool`. This keyword should be followed by a block. The runtime hooks should have a default implementation which do nothing. ```d void* _d_pool_enter() { return null; } void _d_pool_leave(void*) { } ``` Whenever an `autoreleasepool` block is encountered by the compiler, the following code should be emitted. ```d // Note: __ctx_ptr is just a placeholder, the name of the variable should be unique. void* __ctx_ptr = _d_pool_enter(); // Code goes here _d_pool_leave(__ctx_ptr); ``` autorelease pools should additionally, on macOS, automatically be generated in the runtime entry point and wrap the user D main function, and threads created in D should also include pool enter and leave calls. If the D compiler links to a library defining `objc_autoreleasePoolPush` and `objc_autoreleasePoolPop`, the D runtime hooks should link against those, instead. The `Foundation` library includes a class called NSAutoreleasePool, however i've found that if Objective-C's runtime is compiled with ARC (Automatic Reference Counting) support, then NSAutoreleasePool becomes no-op in some instances, which in turn causes memory leaks.
Nov 16
On Saturday, 16 November 2024 at 22:29:36 UTC, Luna wrote:Currently DLang has, in some compilers, support for ObjectiveC types being declared. However, modern Objective-C code is strongly dependent on autorelease and auto release pools. Without them Objective-C code leaks memory (see: https://developer.apple.com/documentation/quartzcore/cametallayer?language=objc). To alleviate this, clang implements the concept of an ` autoreleasepool` block, this block frees all Objective-C classes which have called `autorelease`. In the case of Metal, for example, a lot of internal data uses autorelease calls. The ` autoreleasepool` blocks in clang are implemented via 2 helper functions declared in the objective-c runtime `libobjc.dylib` or the like in alternate runtime implementations. ```d extern(C) extern void* objc_autoreleasePoolPush(); extern(C) extern void objc_autoreleasePoolPop(void*); ``` My proposal is to add 2 new hooks to the D runtime emitted by the compiler as well as a new keyword, `autoreleasepool`. This keyword should be followed by a block. The runtime hooks should have a default implementation which do nothing. ```d void* _d_pool_enter() { return null; } void _d_pool_leave(void*) { } ``` Whenever an `autoreleasepool` block is encountered by the compiler, the following code should be emitted. ```d // Note: __ctx_ptr is just a placeholder, the name of the variable should be unique. void* __ctx_ptr = _d_pool_enter(); // Code goes here _d_pool_leave(__ctx_ptr); ``` autorelease pools should additionally, on macOS, automatically be generated in the runtime entry point and wrap the user D main function, and threads created in D should also include pool enter and leave calls. If the D compiler links to a library defining `objc_autoreleasePoolPush` and `objc_autoreleasePoolPop`, the D runtime hooks should link against those, instead. The `Foundation` library includes a class called NSAutoreleasePool, however i've found that if Objective-C's runtime is compiled with ARC (Automatic Reference Counting) support, then NSAutoreleasePool becomes no-op in some instances, which in turn causes memory leaks.This would be easy to do, synchronised works in pretty much the exact same way, but why does this need to be a compiler thing? i.e. what does it offer that ```d struct AutoreleasePool { private void* __p; static AutoreleasePool opCall() { AutoreleasePool ret; ret.__p = objc_autoreleasePoolPush(); return ret; } ~this () { objc_autoreleasePoolPop(__p); } } void main() { auto x = AutoreleasePool(); } ``` does not? I think I saw some of your code that used a scope delegate as `autorelease((){ ... });` why is that not sufficient? Given how niche this is, there had better be a very good reason for adding it, e.g. generality with regular D code, not just Obj-C.
Nov 16
On Saturday, 16 November 2024 at 22:47:09 UTC, Nicholas Wilson wrote:On Saturday, 16 November 2024 at 22:29:36 UTC, Luna wrote:There's a few subtleties that would make this change in particular good to have on macOS at least. Given that, as said, threads and the runtime entry and exits needs to have implicit autoreleasepools to avoid potentially leaking both memory, but more importantly: kernel handles. A lot of functionality in macOS relies on getting shared memory handles that are wrapped in autorelease handlers; and without a pool to clean them up you can end up with a bunch of zombie handles exhausting the available handles in the system. Having this implicit and explicit control over autorelease behaviour would be a massive benefit and memory safety improvement when working on cross-platform software in D, as you can not avoid relying on Objective-C, eg. if you're making graphical applications.Currently DLang has, in some compilers, support for ObjectiveC types being declared. However, modern Objective-C code is strongly dependent on autorelease and auto release pools. Without them Objective-C code leaks memory (see: https://developer.apple.com/documentation/quartzcore/cametallayer?language=objc). To alleviate this, clang implements the concept of an ` autoreleasepool` block, this block frees all Objective-C classes which have called `autorelease`. In the case of Metal, for example, a lot of internal data uses autorelease calls. The ` autoreleasepool` blocks in clang are implemented via 2 helper functions declared in the objective-c runtime `libobjc.dylib` or the like in alternate runtime implementations. ```d extern(C) extern void* objc_autoreleasePoolPush(); extern(C) extern void objc_autoreleasePoolPop(void*); ``` My proposal is to add 2 new hooks to the D runtime emitted by the compiler as well as a new keyword, `autoreleasepool`. This keyword should be followed by a block. The runtime hooks should have a default implementation which do nothing. ```d void* _d_pool_enter() { return null; } void _d_pool_leave(void*) { } ``` Whenever an `autoreleasepool` block is encountered by the compiler, the following code should be emitted. ```d // Note: __ctx_ptr is just a placeholder, the name of the variable should be unique. void* __ctx_ptr = _d_pool_enter(); // Code goes here _d_pool_leave(__ctx_ptr); ``` autorelease pools should additionally, on macOS, automatically be generated in the runtime entry point and wrap the user D main function, and threads created in D should also include pool enter and leave calls. If the D compiler links to a library defining `objc_autoreleasePoolPush` and `objc_autoreleasePoolPop`, the D runtime hooks should link against those, instead. The `Foundation` library includes a class called NSAutoreleasePool, however i've found that if Objective-C's runtime is compiled with ARC (Automatic Reference Counting) support, then NSAutoreleasePool becomes no-op in some instances, which in turn causes memory leaks.This would be easy to do, synchronised works in pretty much the exact same way, but why does this need to be a compiler thing? i.e. what does it offer that ```d struct AutoreleasePool { private void* __p; static AutoreleasePool opCall() { AutoreleasePool ret; ret.__p = objc_autoreleasePoolPush(); return ret; } ~this () { objc_autoreleasePoolPop(__p); } } void main() { auto x = AutoreleasePool(); } ``` does not? I think I saw some of your code that used a scope delegate as `autorelease((){ ... });` why is that not sufficient? Given how niche this is, there had better be a very good reason for adding it, e.g. generality with regular D code, not just Obj-C.
Nov 16
Luna <luna foxgirls.gay> On Sunday, 17 November 2024 at 00:09:06 UTC, Luna wrote:On Saturday, 16 November 2024 at 22:47:09 UTC, Nicholas Wilson wrote:I will add that the code I shared on the D Language Code Club discord still leaks ~10 objects on exit (but no kernel handles) due to a pool not being created before the objective-c runtime is loaded. This however is serviceable, as the memory gets freed anyway when the process exits. However if you are opening file handles without an autorelease pool you may end up leaking handles if you forget creating one. Having that handled automatically would be of great benefit in ensuring correct behaviour. However there's a single exception to autorelease pools, and that's Objective-C exceptions. Those implicitly free pool objects and don't need special handling, from my understanding. As such pools should only be freed when its scope exits *without any Objective-C exception being thrown*. D exceptions don't trigger this behaviour though, so there might need to be special handling for uncaught exceptions and errors.On Saturday, 16 November 2024 at 22:29:36 UTC, Luna wrote:There's a few subtleties that would make this change in particular good to have on macOS at least. Given that, as said, threads and the runtime entry and exits needs to have implicit autoreleasepools to avoid potentially leaking both memory, but more importantly: kernel handles. A lot of functionality in macOS relies on getting shared memory handles that are wrapped in autorelease handlers; and without a pool to clean them up you can end up with a bunch of zombie handles exhausting the available handles in the system. Having this implicit and explicit control over autorelease behaviour would be a massive benefit and memory safety improvement when working on cross-platform software in D, as you can not avoid relying on Objective-C, eg. if you're making graphical applications.[...]This would be easy to do, synchronised works in pretty much the exact same way, but why does this need to be a compiler thing? i.e. what does it offer that ```d struct AutoreleasePool { private void* __p; static AutoreleasePool opCall() { AutoreleasePool ret; ret.__p = objc_autoreleasePoolPush(); return ret; } ~this () { objc_autoreleasePoolPop(__p); } } void main() { auto x = AutoreleasePool(); } ``` does not? I think I saw some of your code that used a scope delegate as `autorelease((){ ... });` why is that not sufficient? Given how niche this is, there had better be a very good reason for adding it, e.g. generality with regular D code, not just Obj-C.
Nov 16
Me and Luna have talked a bit on Discord about this, and I've done quite
a bit of reading.
With ARC (adding of retain/release automatically) we do not need release
pool support. However while we might not need it, there is a lot of code
out there that is not compiled with ARC which we will want to use.
This does place an importance on us adding reference counting that would
be objective-c/COM aware.
Regarding code-gen, it is equivalent to a ``scope(exit)`` to perform the
pool leaving.
```d
void main() autorelease {
}
```
Is equivalent to:
```d
void main() {
void* __generatedIdentifier1 = objc_autoreleasePoolPush();
bool __generatedIdentifier2 = true;
try {
} catch (NSException e) {
__generatedIdentifier2 = false;
throw e;
} finally {
if (__generatedIdentifier2)
objc_autoreleasePoolPop(__generatedIdentifier);
}
}
```
The reason for the guard variable is that NSException effectively
inherits from ``Error`` not ``Exception``. It is not meant to be caught.
The lifetime of the auto release memory must outlive the catch.
Due to this exception behavior, this would be a valuable addition to our
Objective-c support.
For the grammar:
```diff
ScopeStatement:
+ autorelease BlockStatement
AtAttribute:
+ autorelease
```
```d
void func() autorelease {
autorelease {
}
}
```
Mangling is unaffected by the attribute on the function.
https://clang.llvm.org/docs/AutomaticReferenceCounting.html
As for:
```d
void* _d_pool_enter() { return null; }
void _d_pool_leave(void*) { }
```
I would suggest not doing it, it doesn't add anything.
Either its linked in on non-Apple platforms and then turned on by
compiler switch, or it is not emitted.
Nov 16