digitalmars.D - DMD Performance Regression Publisher [GSoC 2026]
- Abul Hossain Khan (51/51) Jun 12 Hi everyone,
- Dmitry Olshansky (4/8) Jun 15 Have you thought about using Linux’s perf tool it has interesting
- Abul Hossain Khan (4/8) Jun 16 We started with Valgrind because it's giving us quite good and
- Abul Hossain Khan (46/46) Jun 25 Hi everyone,
- monkyyy (3/9) Jun 25 binary size of dmd is much less important then compile speed
- FinalEvilution (12/22) Jun 26 +1 for PGO.
- Abul Hossain Khan (2/2) Jul 03 Thank u so much for the suggestions.
- Abul Hossain Khan (51/51) Jul 10 Hi everyone,
- Abul Hossain Khan (52/52) Jul 20 Hi everyone,
- libxmoc (5/5) Jul 20 This is a much needed project, thanks for the PR!
- Adam D. Ruppe (2/3) Jul 20 When you run out of memory, compile speed increases to infinity.
- user1234 (14/22) Jul 21 This is very good work. the CI aspects have been a bit left aside
Hi everyone, I am working on the Performance Regression Publisher project under the mentorship of Dennis. My progress so far: the initial end-to-end pipeline has been built and is working on my fork. The bot builds DMD at a PR's merge-base and at its head, measures a small set of metrics under cachegrind, and posts a single sticky comment with the diff. **What's done** The harness is in `tools/perfrunner/` and is written in D (dub project), - `app.d` — CLI, takes the two already-built dmd binaries + metadata and writes `results.json`. - `cachegrind.d` — runs the compile under cachegrind and reads the instruction count. - `metrics.d` — the five metrics below. - `report.d` / `stats.d` — the schema-v1 JSON and the percent-delta math. - `workloads/hello.d` — the single workload for now will add more soon. Around it, `.github/workflows/perf.yml` runs on every PR (and on pushes to master), builds both refs with the existing `ci/run.sh`, runs the harness, and hands the result to `.github/scripts/perf_comment.py`, which upserts one sticky comment so force-pushes don't spam the thread. **The metrics it reports(PR Comment) right now Looks like this:** | Metric | Base | PR | delta | |--------|------|----|-------| | compile hello.d (instr) | 422.9 M | 457.9 M | +8.27% | | compile hello.d -O (instr) | 446.1 M | 481.1 M | +7.84% | | dmd binary size (stripped) | 11.91 MB | 11.91 MB | 0.00% | | hello binary size | 0.72 MB | 0.72 MB | 0.00% | | peak RSS (compile hello.d) | 56 MB | 55 MB | -2.17% | I tested the whole path end to end: On my fork with a deliberate busy loop in `compiler/src/main.d`, and the bot reported +8.27% / +7.84% instructions consistently across reruns while size stayed flat. Code - https://github.com/abulgit/dmd/pull/39 **what's next -** 1. We are currently building both with DMD 2.112.0 as the host compiler. Dennis suggested moving to `ldc2 -O3` with PGO so the binary we measure matches a real release build and the optimizer doesn't make a harmless PR look like a regression. Also he suggested that we should to do this early, in case valgrind has any trouble with ldc2. 2. After that, We will add more Realistic Workloads there like Phobos etc. 3. And then Building the dashboard that will show the historical Data. That's the plan we have right now, and I'll try to post weekly updates here as work progresses. Feel free to leave any feedback or suggestions!
Jun 12
On Friday, 12 June 2026 at 18:58:03 UTC, Abul Hossain Khan wrote:Hi everyone, I am working on the Performance Regression Publisher project under the mentorship of Dennis. [...]Have you thought about using Linux’s perf tool it has interesting stats about performance counters. Valgrind is sensible tool but being intrusive it may distort profile a little bit.
Jun 15
Abul Hossain Khan <abulkhan19175 gmail.com> Have you thought about using Linux’s perf tool it has interesting stats about performance counters. Valgrind is sensible tool but being intrusive it may distort profile a little bit.We started with Valgrind because it's giving us quite good and stable results so far, but we'll definitely look into perf as well, especially once we switch to ldc2 + PGO and can compare the approaches.
Jun 16
Hi everyone, Last time I posted, we had the basic pipeline working: the bot builds DMD at the base and head commits, runs Cachegrind, and posts a sticky comment. At that point, everything was being built with DMD 2.112.0 as the host compiler. Now we are building with `ldc2-1.42.0` as the host compiler, so the binary we measure more closely resembles a real release build. I got that working, and the numbers improved quite a bit. | Metric | DMD Host | LDC2 Host | Improvement | | -------------------------- | -------: | ----------: | ----------: | | compile hello.d (instr) | 422.8 M | 257.6 M | -39.1% | | compile hello.d -O (instr) | 445.9 M | 277.9 M | -37.7% | | dmd binary size (stripped) | 11.93 MB | 6.96 MB | -41.7% | | hello binary size | 0.72 MB | 0.72 MB | - | | peak RSS | 56 MB | 49 MB | -12.5% | Switching from DMD as the host compiler to LDC2 reduced instruction counts by about 39%, cut the compiler binary size by 41%, and slightly reduced peak memory usage. The results are also very consistent across reruns, and the workflow runs faster as well. Now as my mentor also suggested adding PGO (Profile Guided Optimization) with the LDC2 build for even better results. I didn't know much about PGO at first, but Dennis helped me understand it and also shared his local build script with PGO enabled with me. I think I've almost figured it out, so that will be the next thing I work on. * Finish the PGO integration ASAP. * Add more realistic workloads, like Phobos etc. and then we will start working on the dashboard to display historical results. Yes I will have my semester exams from June 27 to July 3, so I will be unavailable for about a week. Hopefully that's okay. I'll make up for it afterward if needed, and I will make sure to finish the project on time (or even ahead of schedule). Also, I want to thank my mentor, Dennis. He has been incredibly supportive throughout. As always, feel free to leave any feedback or suggestions!
Jun 25
On Thursday, 25 June 2026 at 17:51:10 UTC, Abul Hossain Khan wrote:Hi everyone, Last time I posted, we had the basic pipeline working: the bot builds DMD at the base and head commits, runs Cachegrind, and posts a sticky comment. At that point, everything was being built with DMD 2.112.0 as the host compiler. [...]binary size of dmd is much less important then compile speed
Jun 25
On Thursday, 25 June 2026 at 17:51:10 UTC, Abul Hossain Khan wrote:Now as my mentor also suggested adding PGO (Profile Guided Optimization) with the LDC2 build for even better results. I didn't know much about PGO at first, but Dennis helped me understand it and also shared his local build script with PGO enabled with me. I think I've almost figured it out, so that will be the next thing I work on.+1 for PGO. As far as I'm concerned everyone who is seriously optimizing for performance should be using PGO. Modifying a build script to run the benchmark suite (if you don't have one what are you optimizing) and compile a second time is a pretty simple change, but I've seen up to a 30% decrease in run time... For just a makefile edit.**Semester exam** Yes I will have my semester exams from June 27 to July 3, so I will be unavailable for about a week.You're going to ace it.As always, feel free to leave any feedback or suggestions!If memory serves perf has the ability to diff the performance of 2 separate profiles on a per function basis. Might be handy.
Jun 26
Thank u so much for the suggestions. Exams are finally over now and they went pretty well.
Jul 03
Hi everyone, Last time we had switched the host compiler from DMD to ldc2-1.42.0, which gave a nice drop in instruction counts and compiler size. Since then I finished my semester exams and got back to the work. DMD already has a `dmd-pgo` build target that does the full LDC PGO cycle, So the first approach was just use that from the workflow for base and head separately. Got that working, but the training was actually failing silently cause Phobos was building after `dmd-pgo`, so the instrumented compiler couldn't import stdlib during training, so it barely trained on anything. Then I fixed the ordering and it worked! But then another problem was that it was showing weird result after that like ~5.5% delta on sizes and also the Instruction Count was showing weird deltas. That's because base and head were each training their own PGO profile independently, in parallel, the merge data comes out slightly different every single run. So the two sides end up with two different profiles even on identical source, which shows up as noise in the diff. Fixed it by training PGO once and reusing that same profile for both builds. After that It was stable across the runs. Here's how the numbers moved across every stage so far: | Metric | DMD host | LDC2 host | LDC2 + PGO | |--------|----------|-----------|------------| | compile hello.d (instr) | 422.8 M | 257.6 M | 224.1 M | | compile hello.d -O (instr) | 445.9 M | 277.9 M | 242.7 M | | dmd binary size (stripped) | 11.93 MB | 6.96 MB | 6.89 MB | | hello binary size | 0.72 MB | 0.72 MB | 0.72 MB | | peak RSS | 56 MB | 50 MB | 50 MB | The system works fine on my forked repo, but a PR that comes from a fork repo gets a read-only token, so the comment step would just fail there. Did some research and the usual fix for this is to split it into two workflows - `perf.yml` - does the build + measure and just uploads the results as an artifact, `perf-comment.yml` - runs on `workflow_run` after that finishes, in the base repo context, so it gets a writable token, downloads the artifact and posts the comment. It should works fine. Now the thing is, `workflow_run` only triggers off the copy of the workflow that's on the default branch. So until this is merged to `master`, `perf-comment.yml` just never fires, even though `perf.yml` runs fine and uploads the artifact. PR link - https://github.com/dlang/dmd/pull/23380 - After the PR got Reviewed and merged We can add more workloads into it, and start working on the dashboard Thanks again to Dennis for the amazing mentorship. As always, feel free to leave any feedback or suggestions!
Jul 10
Hi everyone, Last time I posted update about we Integrated PGO in our Build process, and the PR was waiting for review. Now the PR is merged into DMD master. Here is what we've done since then: We added Phobos compilation metrics to perfrunner. We reused the Phobos checkout already present in CI by passing `--base-phobos` and `--head-phobos` into perfrunner and compiling Phobos's `std/package.d` with `-i=std -preview=dip1000`. Measuring Phobos compilation gives us a much more realistic compiler benchmark (~5 billion instructions and ~633 MB peak RSS) compared to just compiling a hello world program. Here is what the bot comment output looks like now with Phobos included: | Metric | Base | PR | delta | |--------|------|----|-------| | compile hello.d (instr) | 215.6 M | 215.6 M | 0.00% | | compile hello.d -O (instr) | 234.1 M | 234.1 M | 0.00% | | compile Phobos (instr) | 5,078.7 M | 5,078.7 M | 0.00% | | dmd binary size (stripped) | 6.87 MB | 6.87 MB | 0.00% | | hello binary size | 0.72 MB | 0.72 MB | 0.00% | | peak RSS (compile hello.d) | 43 MB | 44 MB | +0.33% | | peak RSS (compile Phobos) | 633 MB | 633 MB | -0.07% | **PR(Merged)**: https://github.com/dlang/dmd/pull/23432 Getting a PR comment about perf every time after opening a PR can be very annoying, especially when there aren't any significant deltas. the bot now skips posting if all metric deltas stay within defined threshold limits. We defined the initial threshold limits as: ```python THRESHOLDS = {"cachegrind": 0.1, "stat": 0.1, "time -v": 2.0} ``` Also threshold values need to be determined experimentally over time by trial and error. So we will observe PRs and can tweak them accordingly. Also fixed a small UI issue where zero deltas were showing up as `-0.00%`. So we add an `abs()` guard for near-zero values, in comment poster. **PR(Merged)**: https://github.com/dlang/dmd/pull/23434 - Will see how the bot performs overtime. - I want to add a couple more real-world workloads, such as `vibe.d`. If anyone has suggestions for other D libraries/dub packages we should benchmark, please let me know! - My mentor gave me an awesome idea that instead of just displaying raw high-level metrics, we can add stage breakdown (using `-ftime-trace`). I don't know much about `-ftime-trace`, so it is going to be very interesting for me to implement it. Thanks again to Dennis for the guidance and reviews! And as always, feel free to leave any feedback or suggestions.
Jul 20
This is a much needed project, thanks for the PR! However, memory usage is not a useful metric. How long it takes to compile community projects is much better one, as others have suggested, the main use of dmd is fast iteration time, so we should benchmark as such.
Jul 20
On Monday, 20 July 2026 at 17:44:16 UTC, libxmoc wrote:However, memory usage is not a useful metric.When you run out of memory, compile speed increases to infinity.
Jul 20
On Monday, 20 July 2026 at 18:44:39 UTC, Adam D. Ruppe wrote:On Monday, 20 July 2026 at 17:44:16 UTC, libxmoc wrote:I just download more ramHowever, memory usage is not a useful metric.When you run out of memory, compile speed increases to infinity.
Jul 20
On Monday, 20 July 2026 at 18:44:39 UTC, Adam D. Ruppe wrote:On Monday, 20 July 2026 at 17:44:16 UTC, libxmoc wrote:I didn't mean it should be dropped as a metric, only that compile time ranks higher in priority.However, memory usage is not a useful metric.When you run out of memory, compile speed increases to infinity.
Jul 22
On Friday, 12 June 2026 at 18:58:03 UTC, Abul Hossain Khan wrote:Hi everyone, I am working on the Performance Regression Publisher project under the mentorship of Dennis. My progress so far: the initial end-to-end pipeline has been built and is working on my fork. The bot builds DMD at a PR's merge-base and at its head, measures a small set of metrics under cachegrind, and posts a single sticky comment with the diff.This is very good work. the CI aspects have been a bit left aside since a few years, but while the initial impact is awesome (I daily check what's going on on GH for DMD, and others..) I have a few questions. 1. how is this reliable ? Are you sure that the hardware used is always the same, i.e when testing master vs head. 2. How the metric is relevant ? Is a PR always supposed to lower the cost in term of intruction refs ? This project put a bit of pressure in this that a new feature will look like a regression. 3. Ecological cost That pumps energy, is that worth ?
Jul 21