digitalmars.D - LDC PGO on Windows

Imperatorn (6/6) Oct 16 2023 Does PGO work on Windows with LDC?

Imperatorn (9/15) Oct 16 2023 With msvc I get

Imperatorn (25/44) Oct 16 2023 I get the same on Linux:

Imperatorn (13/53) Oct 16 2023 After some experimentation it turned out it was working after

Imperatorn (20/23) Oct 17 2023 ------------------------------------------------------------------------...

Imperatorn (69/73) Oct 17 2023 If you are having trouble using the above, try the following

jmh530 (7/12) Oct 17 2023 Is there any way to incorporate some of what you figured out into

Sergey (3/19) Oct 21 2023 Put another link here. For future "new documentation project":

Imperatorn (2/16) Oct 21 2023 Thanks for the links!

Imperatorn (2/18) Nov 11 2023 Yeah, if I get the time I will

Imperatorn (10/15) Oct 17 2023 A typo:

Imperatorn <johan_forsberg_86 hotmail.com> writes:

Does PGO work on Windows with LDC?

If so, how do I use it? Been trying to follow the information 
available but get various errors related to linker or runtime 
errors.

Is it supported on Windows?

Thanks

Oct 16 2023

Imperatorn <johan_forsberg_86 hotmail.com> writes:

On Monday, 16 October 2023 at 17:00:41 UTC, Imperatorn wrote:
 Does PGO work on Windows with LDC?

 If so, how do I use it? Been trying to follow the information 
 available but get various errors related to linker or runtime 
 errors.

 Is it supported on Windows?

 Thanks

With msvc I get
unresolved external symbol llvm_profile_instrument_target 
referenced in function _D4hash7keynode9xtoHashFNbNeKxSQBfQBdZm
unresolved external symbol llvm_profile_runtime referenced in 
function llvm_profile_runtime_user

And with link-internally I get
lld-link: error: undefined symbol: llvm_profile_runtime
lld-link: error: undefined symbol: llvm_profile_instrument_target

Oct 16 2023

Imperatorn <johan_forsberg_86 hotmail.com> writes:

On Monday, 16 October 2023 at 17:13:04 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:00:41 UTC, Imperatorn wrote:
 Does PGO work on Windows with LDC?

 If so, how do I use it? Been trying to follow the information 
 available but get various errors related to linker or runtime 
 errors.

 Is it supported on Windows?

 Thanks

 With msvc I get
 unresolved external symbol llvm_profile_instrument_target 
 referenced in function _D4hash7keynode9xtoHashFNbNeKxSQBfQBdZm
 unresolved external symbol llvm_profile_runtime referenced in 
 function llvm_profile_runtime_user

 And with link-internally I get
 lld-link: error: undefined symbol: llvm_profile_runtime
 lld-link: error: undefined symbol: 
 llvm_profile_instrument_target

I get the same on Linux:
undefined reference to `__llvm_profile_instrument_target'

And trying a minimal example gives me:

Exception Code: 0xC000001D
0x00007FF603511EDE, C:\Program Files\LDC 
1.35\bin\ldc2.exe(0x00007FF600310000) + 0x3201EDE byte(s)
0x00007FF60318F052, C:\Program Files\LDC 
1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2E7F052 byte(s)
0x00007FF6032A01DA, C:\Program Files\LDC 
1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F901DA byte(s)
0x00007FF60329FE2F, C:\Program Files\LDC 
1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F8FE2F byte(s)
0x00007FF6032A0135, C:\Program Files\LDC 
1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F90135 byte(s)
0x00007FF6031884DD, C:\Program Files\LDC 
1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2E784DD byte(s)
0x00007FF60353F4C4, C:\Program Files\LDC 
1.35\bin\ldc2.exe(0x00007FF600310000) + 0x322F4C4 byte(s)
0x00007FFF17AB7344, 
C:\windows\System32\KERNEL32.DLL(0x00007FFF17AA0000) + 0x17344 
byte(s), BaseThreadInitThunk() + 0x14 byte(s)
0x00007FFF18F426B1, 
C:\windows\SYSTEM32\ntdll.dll(0x00007FFF18EF0000) + 0x526B1 
byte(s), RtlUserThreadStart() + 0x21 byte(s)

Oct 16 2023

Imperatorn <johan_forsberg_86 hotmail.com> writes:

On Monday, 16 October 2023 at 17:43:21 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:13:04 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:00:41 UTC, Imperatorn wrote:
 [...]

 With msvc I get
 unresolved external symbol llvm_profile_instrument_target 
 referenced in function _D4hash7keynode9xtoHashFNbNeKxSQBfQBdZm
 unresolved external symbol llvm_profile_runtime referenced in 
 function llvm_profile_runtime_user

 And with link-internally I get
 lld-link: error: undefined symbol: llvm_profile_runtime
 lld-link: error: undefined symbol: 
 llvm_profile_instrument_target

 I get the same on Linux:
 undefined reference to `__llvm_profile_instrument_target'

 And trying a minimal example gives me:

 Exception Code: 0xC000001D
 0x00007FF603511EDE, C:\Program Files\LDC 
 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x3201EDE byte(s)
 0x00007FF60318F052, C:\Program Files\LDC 
 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2E7F052 byte(s)
 0x00007FF6032A01DA, C:\Program Files\LDC 
 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F901DA byte(s)
 0x00007FF60329FE2F, C:\Program Files\LDC 
 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F8FE2F byte(s)
 0x00007FF6032A0135, C:\Program Files\LDC 
 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2F90135 byte(s)
 0x00007FF6031884DD, C:\Program Files\LDC 
 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x2E784DD byte(s)
 0x00007FF60353F4C4, C:\Program Files\LDC 
 1.35\bin\ldc2.exe(0x00007FF600310000) + 0x322F4C4 byte(s)
 0x00007FFF17AB7344, 
 C:\windows\System32\KERNEL32.DLL(0x00007FFF17AA0000) + 0x17344 
 byte(s), BaseThreadInitThunk() + 0x14 byte(s)
 0x00007FFF18F426B1, 
 C:\windows\SYSTEM32\ntdll.dll(0x00007FFF18EF0000) + 0x526B1 
 byte(s), RtlUserThreadStart() + 0x21 byte(s)

After some experimentation it turned out it was working after 
all, but not with any code and dependencies.

For example in my case, if I removed asdf deserialize, it worked, 
but if I introduced it again, I got a bunch of errors like this

```
lld-link: error: undefined symbol: ClassInfo for 
asdf.asdf.AsdfSerdeException
 [...]



lld-link: error: undefined symbol: vtable for 
asdf.asdf.AsdfSerdeException
 [...]



lld-link: error: undefined symbol: initializer for 
asdf.asdf.AsdfSerdeException
```

Oct 16 2023

Imperatorn <johan_forsberg_86 hotmail.com> writes:

On Monday, 16 October 2023 at 18:10:21 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:43:21 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:13:04 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:00:41 UTC, Imperatorn wrote:





--------------------------------------------------------------------------------------------

For documentation purposes I will share the solution here.

The problem was I was using dub and didn't know how to forward 
everything correctly as dflags.

In your dflags, supply something like this, do *not* add -of=test 
or whatever, dub will overwrite it:

```d
"dflags": ["-m64", "-release" ,"-fprofile-instr-generate"]
```

Then just:
```d
dub build --compiler=ldc2
```

This will produce an executable with the same name as you 
specified in your dub config.

Now you run that and it will produce an output file that you will 
convert.

The rest of the steps can be read here:

https://johanengelen.github.io/ldc/2016/07/15/Profile-Guided-Optimization-with-LDC.html

Oct 17 2023

Imperatorn <johan_forsberg_86 hotmail.com> writes:

On Tuesday, 17 October 2023 at 08:59:52 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 18:10:21 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:43:21 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:13:04 UTC, Imperatorn wrote:
 On Monday, 16 October 2023 at 17:00:41 UTC, Imperatorn wrote:





If you are having trouble using the above, try the following 
instead:

```d
dub build --compiler=ldc2 --build-mode=allAtOnce --combined
```

The complete process for dub (replace {programname} with your 
program name):

1. Specify a name and dflags
```d
	"dflags": ["-fprofile-instr-generate"],
	"name": "{programname}",
```

2. Build
```d
dub build --compiler=ldc2 --build-mode=allAtOnce --combined
```

3. Execute
```
./{programname}
```

4. Convert
```
ldc-profdata merge default.profraw -output converted.profdata
```

5. Change your dub.json to use the instrumented configuration and 
change the name
```d
	"dflags": ["-fprofile-instr-use=converted.profdata"],
	"name": "{programname}_optimized",
```

6. Build
```d
dub build --compiler=ldc2 --build-mode=allAtOnce --combined
```

7. Enjoy your PGO executable

-------------------------------------------------------------------------------------------


All the above steps in a script, but without dub, assuming app.d 
is in source

Windows:

```d
echo "Building app_normal.exe for comparison"
ldc2 -of="app_normal.exe" -O -g -gc -Isource source\app.d

echo "Building app_instrumented.exe with profiling"
ldc2 -fprofile-instr-generate -O -g -gc 
-of="app_instrumented.exe" -Isource source\app.d

echo "Running app_instrumented.exe"
app_instrumented.exe

ldc-profdata merge default.profraw -output converted.profdata

echo "Building PGO executable"
ldc2 -fprofile-instr-use=converted.profdata -O 
-of="app_optimized.exe" -Isource source\app.d -flto=full 
-defaultlib=phobos2-ldc-lto,druntime-ldc-lto
```

*nix:

```bash
echo "Building app_normal for comparison"
ldc2 -of="app_normal" -O -g -gc -Isource source/app.d

echo "Building app_instrumented with profiling"
ldc2 -fprofile-instr-generate -O -g -gc -of="app_instrumented" 
-Isource source/app.d

echo "Running app_instrumented"
./app_instrumented

ldc-profdata merge default.profraw -output converted.profdata

echo "Building PGO executable"
ldc2 -fprofile-instr-use=converted.profdata -O 
-of="app_optimized" -Isource source/app.d -flto=full 
-defaultlib=phobos2-ldc-lto,druntime-ldc-lto
```

Oct 17 2023

jmh530 <john.michael.hall gmail.com> writes:

On Tuesday, 17 October 2023 at 13:00:54 UTC, Imperatorn wrote:
 On Tuesday, 17 October 2023 at 08:59:52 UTC, Imperatorn wrote:
 [...]





 If you are having trouble using the above, try the following 
 instead:

 [...]

Is there any way to incorporate some of what you figured out into
https://wiki.dlang.org/LDC_LLVM_profiling_instrumentation

It might also make sense to file an issue with dub about 
streamlining the process. I only see one issue currently that 
references PGO.
https://github.com/dlang/dub/issues/951

Oct 17 2023

Sergey <kornburn yandex.ru> writes:

On Tuesday, 17 October 2023 at 13:42:24 UTC, jmh530 wrote:
 On Tuesday, 17 October 2023 at 13:00:54 UTC, Imperatorn wrote:
 On Tuesday, 17 October 2023 at 08:59:52 UTC, Imperatorn wrote:
 [...]





 If you are having trouble using the above, try the following 
 instead:

 [...]

 Is there any way to incorporate some of what you figured out 
 into
 https://wiki.dlang.org/LDC_LLVM_profiling_instrumentation

 It might also make sense to file an issue with dub about 
 streamlining the process. I only see one issue currently that 
 references PGO.
 https://github.com/dlang/dub/issues/951

Put another link here. For future "new documentation project":
https://archive.fosdem.org/2017/schedule/event/ldc_d_optimization/attachments/slides/1819/export/events/attachments/ldc_d_optimization/slides/1819/FOSDEM_2017.pdf

Oct 21 2023

Imperatorn <johan_forsberg_86 hotmail.com> writes:

On Saturday, 21 October 2023 at 11:31:54 UTC, Sergey wrote:
 On Tuesday, 17 October 2023 at 13:42:24 UTC, jmh530 wrote:
 On Tuesday, 17 October 2023 at 13:00:54 UTC, Imperatorn wrote:
 [...]

 Is there any way to incorporate some of what you figured out 
 into
 https://wiki.dlang.org/LDC_LLVM_profiling_instrumentation

 It might also make sense to file an issue with dub about 
 streamlining the process. I only see one issue currently that 
 references PGO.
 https://github.com/dlang/dub/issues/951

 Put another link here. For future "new documentation project":
 https://archive.fosdem.org/2017/schedule/event/ldc_d_optimization/attachments/slides/1819/export/events/attachments/ldc_d_optimization/slides/1819/FOSDEM_2017.pdf

Thanks for the links!

Oct 21 2023

Imperatorn <johan_forsberg_86 hotmail.com> writes:

On Tuesday, 17 October 2023 at 13:42:24 UTC, jmh530 wrote:
 On Tuesday, 17 October 2023 at 13:00:54 UTC, Imperatorn wrote:
 On Tuesday, 17 October 2023 at 08:59:52 UTC, Imperatorn wrote:
 [...]





 If you are having trouble using the above, try the following 
 instead:

 [...]

 Is there any way to incorporate some of what you figured out 
 into
 https://wiki.dlang.org/LDC_LLVM_profiling_instrumentation

 It might also make sense to file an issue with dub about 
 streamlining the process. I only see one issue currently that 
 references PGO.
 https://github.com/dlang/dub/issues/951

Yeah, if I get the time I will

Nov 11 2023

Imperatorn <johan_forsberg_86 hotmail.com> writes:

On Tuesday, 17 October 2023 at 13:00:54 UTC, Imperatorn wrote:
 On Tuesday, 17 October 2023 at 08:59:52 UTC, Imperatorn wrote:
 [...]





 If you are having trouble using the above, try the following 
 instead:

 [...]

A typo:

It should be

```
-fprofile-instr-use="converted.profdata"
```

not

```
-fprofile-instr-use=converted.profdata
```

Oct 17 2023

D Programming

C/C++ Programming

Other

digitalmars.D - LDC PGO on Windows