digitalmars.D - A big update on 2.110
- Dennis (175/175) Mar 06 Hello everyone,
- Richard (Rikki) Andrew Cattermole (8/10) Mar 06 This isn't entirely true.
- Sergey (2/5) Mar 06 Thanks Dennis and Iain for your great work!
- Hipreme (8/14) Mar 06 Yes, there's no need to release 2.110.1
- Bastiaan Veelo (18/28) Mar 06 Thank you for doing this very important work, it is good to see
- jmh530 (2/6) Mar 06 Thank you and Iain for all your hard work.
- M. M. (8/13) Mar 06 Thank you for the update and the details behind. It is much
- kinke (1/1) Mar 06 Much obliged!
- Salih Dincer (17/28) Mar 06 It is clear that you enjoyed what you wrote (especially the
- Dukc (21/36) Mar 06 Thanks for the writeup - and especially the work to do this. Wow,
- Dennis (20/26) Mar 06 Indeed, that was one of the reasons I chose NixOS even though I'm
- matheus (7/9) Mar 07 Nice story and a bit scary one. There must have be good relief in
- deadalnix (11/15) Mar 07 Unfortunately, what is released as 2.110 differs from what's in
- deadalnix (6/22) Mar 07 I was able to reduce this to
- Walter Bright (3/3) Mar 07 I big thank you to Dennis and Iain for the tremendous amount of work sol...
Hello everyone, As you all know, the official DMD releases got severely delayed. I think you're owed an explanation of why it took so long. Iain Buclaw, our release manager, was working on streamlining the release process to make it more automated and reduce the '[bus factor](https://en.wikipedia.org/wiki/Bus_factor)'. Then he got 'hit by a bus'. Development on dmd and automatic [nightly](https://github.com/dlang/dmd/releases/tag/nightly) builds continued, but the official releases for 2.110 and beyond got delayed and delayed, as nobody was around with the know-how and access to make a proper release. Luckily, as the [November update](https://forum.dlang.org/post/xlsnoajvuphefxlestwu forum.dlang.org) alluded to, Iain was going to slowly return and on January 8, he uploaded the release candidate for 2.110. While doing that, he walked me through the process in a Slack huddle / tmate session, so I could take over the task for the time being. I was optimistic that things were going to be back on track now, and updated the release schedule with 2.110.0 due for February 1st 2025. However, once that day passed, it turned out we were not in the clear yet. At first I thought there was just a small wait for [the new changelog generator after our migration from Bugzilla to GitHub issues](https://github.com/dlang/tools/pull/466), but no: 2.110.0 still needed to be built and uploaded, so this week I made it my priority to get everything sorted out. Feel free to skip to the next paragraph, this is just a bunch of technical details for your amusement / understanding of why it was so hard to complete a release for me. So like I said, Iain and I scheduled an online meeting to go over the release process on January 8. Right from the start, Slack huddles don't work in Firefox, so I downloaded the client, which crashed really badly when I started sharing my screen through it, requiring a reboot of my computer to fix my audio. That's not relevant, but it set the tone wonderfully for everyone that was about to come. So the release orchestration is largely handled by one big [release.sh](https://gist.github.com/ibuclaw/35d19c5ff64ecea25e4a7c7ce6a79eac) bash script with about 20 sequential steps, but it can't just be run on any computer: it has a lot of dependencies on software being installed, configuration, authentication to various servers etc. So Iain started by giving me an account on his server from which he builds releases, that I can ssh into. Then we walked through each step, one by one. Because I had some trouble getting my git configuration/authentication/keys etc. working on the server, I ended up `rsync`'ing my user folder on the server to my own desktop so I could continue locally. That folder is 18 GB because it contains 4 different virtual machines for building the Linux, Windows, OSX and FreeBSD releases, but once the download completes, I should be able to build the release, right? Each step uncovered new missing dependencies / setup on my part. I installed `vagrant` and `virtualbox`, had to `modprobe vboxdrv`, but that required installing `linux-rt-headers`, wait it still doesn't work, oh the right package is `linux-headers` yadayadayada... Running the VM also requires enabling virtualization in your BIOS, a setting which I couldn't find at first because it's labeled 'SVM mode' on this particular motherboard. The biggest issue was that I have an AMD processor, and OSX refuses to boot on anything but an Intel processor. Apparently it's easy to fool the cpuid on libvirt, but not on virtualbox. That's how far we came in that huddle: I made progress, but ultimately couldn't build myself yet. When it became clear this week that I had to figure out a way to build myself, I started to look if I had a different computer laying around. Since my old desktop with an Intel processor has a broken motherboard, I dug up my old 2015 student laptop with Windows 10, and installed linux on a new partition. The USB drive from which I previously installed NixOS on a mini laptop suddenly failed to start the display manager. It looks like some bits got corrupted, because a decompression program reported bad file contents. I always suspected that particular USB stick to be unreliable, but the 2 other drives I tried wouldn't even get picked up by my Laptop's boot loader, so I copied a fresh NixOS installer on the sketchy one and hoped it would work. It did. I can't seem to boot it directly though, I have to boot Windows 10 first and then shift + restart => use device => Linux Boot Loader. Oh well. Now I had to configure everything on my new installation. Some ChatGPT assistance, [Stack Overflow](https://stackoverflow.com/questions/29933918/ssh-key-permissions-0644-for-id-rsa-pub-a e-too-open-on-mac), and an [MVP GitHub comment](https://github.com/NixOS/nixpkgs/issues/76108#issuecomment-592195955) later, I could finally boot the virtual boxes! Then it exceeded my 8GB of RAM and the OOM killer crashed the whole PC. I rebooted and set up an 8GB swap file. I retried, but suddenly the Virtual Boxes wouldn't boot anymore, now reporting a new error. Huh? When I retried, it failed even before getting to the command to start the VM. Can't write to `/tmp/XXX`? Oh, I see. I ran out of disk space. I only had room for a 58 GB partition, which I thought was enough for the 20 GB of VMs, but not when `~/VirtualBox VMs` and `/tmp` get filled up by running vagrant so many times. Some disk cleaning and 2 hours of anxiously watching my task manager showing 95% RAM usage later, it actually did it. It built releases for all platforms! That felt real good, but my troubleshooting hadn't come to an end just yet... To run the script that handles the release process, [to quote Atila](https://forum.dlang.org/post/fiyfbeczwihhsbiudtaw forum.dlang.org), you need a special ~~Windows~~ Linux box commisioned by the Vatican and blessed by the Pope himself. But after a ton of troubleshooting, my old laptop became such a blessed box. It's probably the third one on the planet, and I'm storing it in a safe in a nuclear bunker. :) Every release is accompanied with a PGP signature in the form a .sig file, which the `install.sh` script and the setup-dlang GitHub action uses to `gpg --verify` the executables with. Since I'm a new signer, I added my public key to the [gpg_keys](https://dlang.org/gpg_keys.html) page. However, I wasn't aware that I also had to manually update the binary `d-keyring.pgp` file, which is the source of public keys installation scripts actually use. So yesterday (March 5) you might have noticed the following error in your CI: ``` Run dlang-community/setup-dlang v1 Enabling dmd-latest Downloading https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.tar.xz Verifying the download with GPG gpg: Signature made Wed Mar 5 12:46:50 2025 UTC gpg: using RSA key F3F896F3274BBD9BBBA59058710592E7FB7AF6CA gpg: Can't check signature: No public key ``` I fixed that this morning (Mar 6 11:00 UTC) with assistance from Andrei Horodniceanu on Slack (thanks!). I'm sorry for the inconvenience. If you encounter this error while manually installing/verifying, ensure you are using the latest keyring. My current plans are: - Fix the changelog (either by trying Robert's new tool or writing something manually) - Formally announce the 2.110 release on Friday March 7, and publish the download links on the website - Update the [release archive index](https://downloads.dlang.org/releases/2.x/) (currently I hit a snag where [d-requests](https://code.dlang.org/packages/requests) can't find the right libssl.so) - Update the documentation archives - Document everything I learned about the release process for future maintainers - Do the releases for 2.111 onwards until Iain or someone else takes over again If you want to download 2.110 already, you can do so at: [libphobos2-109_2.110.0-0_i386.deb.sig](https://downloads.dlang.org/releases/2.x/2.110.0/libphobos2-109_2.110.0-0_i386.deb.sig) [libphobos2-109_2.110.0-0_i386.deb](https://downloads.dlang.org/releases/2.x/2.110.0/libphobos2-109_2.110.0-0_i386.deb) [libphobos2-109_2.110.0-0_amd64.deb.sig](https://downloads.dlang.org/releases/2.x/2.110.0/libphobos2-109_2.110.0-0_amd64.deb.sig) [libphobos2-109_2.110.0-0_amd64.deb](https://downloads.dlang.org/releases/2.x/2.110.0/libphobos2-109_2.110.0-0_amd64.deb) [dmd_2.110.0-0_i386.deb.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd_2.110.0-0_i386.deb.sig) [dmd_2.110.0-0_i386.deb](https://downloads.dlang.org/releases/2.x/2.110.0/dmd_2.110.0-0_i386.deb) [dmd_2.110.0-0_amd64.deb.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd_2.110.0-0_amd64.deb.sig) [dmd_2.110.0-0_amd64.deb](https://downloads.dlang.org/releases/2.x/2.110.0/dmd_2.110.0-0_amd64.deb) [dmd.2.110.0.windows.zip.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.windows.zip.sig) [dmd.2.110.0.windows.zip](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.windows.zip) [dmd.2.110.0.windows.7z.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.windows.7z.sig) [dmd.2.110.0.windows.7z](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.windows.7z) [dmd.2.110.0.osx.zip.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.zip.sig) [dmd.2.110.0.osx.zip](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.zip) [dmd.2.110.0.osx.tar.xz.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.tar.xz.sig) [dmd.2.110.0.osx.tar.xz](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.osx.tar.xz) [dmd.2.110.0.linux.zip.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.linux.zip.sig) [dmd.2.110.0.linux.zip](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.linux.zip) [dmd.2.110.0.linux.tar.xz.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.linux.tar.xz.sig) [dmd.2.110.0.linux.tar.xz](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.linux.tar.xz) [dmd.2.110.0.freebsd-64.zip.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.freebsd-64.zip.sig) [dmd.2.110.0.freebsd-64.zip](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.freebsd-64.zip) [dmd.2.110.0.freebsd-64.tar.xz.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.freebsd-64.tar.xz.sig) [dmd.2.110.0.freebsd-64.tar.xz](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.freebsd-64.tar.xz) [dmd.2.110.0.dmg.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.dmg.sig) [dmd.2.110.0.dmg](https://downloads.dlang.org/releases/2.x/2.110.0/dmd.2.110.0.dmg) [dmd-2.110.0.exe.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0.exe.sig) [dmd-2.110.0.exe](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0.exe) [dmd-2.110.0-0.openSUSE.x86_64.rpm.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.openSUSE.x86_64.rpm.sig) [dmd-2.110.0-0.openSUSE.x86_64.rpm](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.openSUSE.x86_64.rpm) [dmd-2.110.0-0.openSUSE.i386.rpm.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.openSUSE.i386.rpm.sig) [dmd-2.110.0-0.openSUSE.i386.rpm](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.openSUSE.i386.rpm) [dmd-2.110.0-0.fedora.x86_64.rpm.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.fedora.x86_64.rpm.sig) [dmd-2.110.0-0.fedora.x86_64.rpm](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.fedora.x86_64.rpm) [dmd-2.110.0-0.fedora.i386.rpm.sig](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.fedora.i386.rpm.sig) [dmd-2.110.0-0.fedora.i386.rpm](https://downloads.dlang.org/releases/2.x/2.110.0/dmd-2.110.0-0.fedora.i386.rpm) Note that 2.110 only contains bug fixes implemented in the last 7 months, not the new features of the last 7 months. Those are scheduled for 2.111, which brings up a question: Iain and I had the idea of skipping 2.110.1 and going straight to 2.111.0, since we're so far behind. There's also been talk of slowing down the amount of releases in general. What do you all think? I know the whole situation is very unprofessional. Personally, I'm working part time as Pull Request and Issue Manager for the D Language Foundation, and part time as D Programmer for [SARC](https://www.sarc.nl/), a Dutch maritime software company with a roughly similar head count as the DLF 'core team'. Every week, I go to SARC's office where there's a weekly meeting with everyone. You can see what everyone is working on, knowledge is easily shared, and issues can be solved together. While I think it's amazing that DLF operates completely remote, with volunteer contributions coming from all around the world, it is awful for handling situations like this 2.110 release debacle. But we are learning and improving overall I think. I want to thank Iain for doing the thankless job of being release manager, and helping us out recently when he really shouldn't have had to. Have a good day everyone. ~Dennis Korpel
Mar 06
On 07/03/2025 2:30 AM, Dennis wrote:Development on dmd and automatic nightly <https://github.com/dlang/dmd/ releases/tag/nightly> builds continuedThis isn't entirely true. Dmd nightlies broke for OSX for less than a day. By some very lucky chance I had been digging into the CI situation as part of making a release when the OSX image went bye byes. We didn't miss a single build! Anyway, thanks to both Dennis and Iain for working on this, I know how much work is needed to get all of this sorted out long term.
Mar 06
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:Hello everyone, Have a good day everyone. ~Dennis KorpelThanks Dennis and Iain for your great work!
Mar 06
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:Note that 2.110 only contains bug fixes implemented in the last 7 months, not the new features of the last 7 months. Those are scheduled for 2.111, which brings up a question: Iain and I had the idea of skipping 2.110.1 and going straight to 2.111.0, since we're so far behind. There's also been talk of slowing down the amount of releases in general. What do you all think?Yes, there's no need to release 2.110.1 I believe that the separation between people which updates their compiler is basically; 1 - Bug Fixes 2 - New features Since bugfixes entered 2.110.0, there isn't much point into releasing the minor
Mar 06
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:Hello everyone, As you all know, the official DMD releases got severely delayed. I think you're owed an explanation of why it took so long.[…]But we are learning and improving overall I think. I want to thank Iain for doing the thankless job of being release manager, and helping us out recently when he really shouldn't have had to. Have a good day everyone. ~Dennis KorpelThank you for doing this very important work, it is good to see this train being put back on the tracks again! Regarding preference for release cadence I think this should be mainly up to the person doing the work. I also think that it makes sense to prioritise making the process more robust and easier to operate over a high cadence. Also, it may be more valuable to have frequent bug fix releases than feature releases. And, should it take extra effort to prepare installer packages than the zip archives, it could be an option to have a lower cadence for these. People that require the latest and greatest are probably fine with a zip. At SARC we have automated compiler installation as part of the commit history, which relies solely on zip archives. This is probably a common approach in professional teams. — Bastiaan.
Mar 06
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:Hello everyone, As you all know, the official DMD releases got severely delayed. I think you're owed an explanation of why it took so long.Thank you and Iain for all your hard work.
Mar 06
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:Hello everyone, As you all know, the official DMD releases got severely delayed. I think you're owed an explanation of why it took so long. [...]Thank you for the update and the details behind. It is much appreciated. Thank you and Iain for doing this work. I hope you can manage to streamline some of that. With respect to the frequency of releases, I quite like the bi-monthly cadence, but really anything else that will be more than once or twice a year is great.
Mar 06
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:To run the script that handles the release process, [to quote Atila](https://forum.dlang.org/post/fiyfbeczwihhsbiudtaw forum.dlang.org), you need a special ~~Windows~~ Linux box commisioned by the Vatican and blessed by the Pope himself. But after a ton of troubleshooting, my old laptop became such a blessed box. It's probably the third one on the planet, and I'm storing it in a safe in a nuclear bunker. :)It is clear that you enjoyed what you wrote (especially the above), which I cannot quote in its entirety. Most of all, I'm glad you made it. I read all the lines that you sincerely wrote. This was difficult due to the fact that my native language is not English. However, the difficulties you are experiencing are insignificant compared to the difficulties you are experiencing. In summary, I congratulate you for what you have doneNote that 2.110 only contains bug fixes implemented in the last 7 months, not the new features of the last 7 months. Those are scheduled for 2.111, which brings up a question: Iain and I had the idea of skipping 2.110.1 and going straight to 2.111.0, since we're so far behind. There's also been talk of slowing down the amount of releases in general. What do you all think?Updating bugs was important. Maybe we should do a little review instead of skipping the version number. And no matter how difficult it is, version 2.110.1 can be completed with a small update, what do you think. In summary, it is better not to spoil what we are used to. Because the previous "versioning method" had already changed Thanks Dennis... SDB 79
Mar 06
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:To run the script that handles the release process, [to quote Atila](https://forum.dlang.org/post/fiyfbeczwihhsbiudtaw forum.dlang.org), you need a special ~~Windows~~ Linux box commisioned by the Vatican and blessed by the Pope himself. But after a ton of troubleshooting, my old laptop became such a blessed box. It's probably the third one on the planet, and I'm storing it in a safe in a nuclear bunker. :)Thanks for the writeup - and especially the work to do this. Wow, that was quite a campaign to get it working! Since your box runs NixOS as the host OS, I guess you can pass its `configuration.nix` to the Foundation, so when it comes a time to set up a fourth box someday it should be at least a bit easier.Note that 2.110 only contains bug fixes implemented in the last 7 months, not the new features of the last 7 months. Those are scheduled for 2.111, which brings up a question: Iain and I had the idea of skipping 2.110.1 and going straight to 2.111.0, since we're so far behind. There's also been talk of slowing down the amount of releases in general. What do you all think?Good idea. There are a varying number of patch versions for each major version anyway. It doesn't bother me if that number is sometimes 1.I know the whole situation is very unprofessional.I don't think the delay itself is that unprofessional. Iain has an incredible load of responsibilities so it would be a surprise if something like this _didn't_ happen sometimes. For a small organisation like the Foundation I don't expect that every role can have a timely back up that would be there in an ideal big company. The only gripe I have is about keeping the users up to date when the release doesn't happen when it was planned to. Why? What's the new ETA? Or if there's no ETA currently, what's the ETA for the new ETA? But you just remedied that with your post.I want to thank Iain for doing the thankless job of being release manager, and helping us out recently when he really shouldn't have had to.+1
Mar 06
On Thursday, 6 March 2025 at 20:33:20 UTC, Dukc wrote:Since your box runs NixOS as the host OS, I guess you can pass its `configuration.nix` to the Foundation, so when it comes a time to set up a fourth box someday it should be at least a bit easier.Indeed, that was one of the reasons I chose NixOS even though I'm more experienced with Debian and Arch at the moment: I wanted to have a comprehensive and reproducible list of dependencies.The only gripe I have is about keeping the users up to date when the release doesn't happen when it was planned to. Why?I understand. I think this also comes down to the delay in communication you get from distributed remote work. Concretely: I was on vacation on February 1st, and missed the DLF monthly meeting on February 7th. On the 14th I had my bi-weekly check-in with Mike and Razvan, and I heard that the 2.110 release situation came up in the monthly meeting, and that Robert and Iain were working on the changelog generator. I guess at that point I could have made an announcement, but I had hardly any details. I couldn't tell if it was going to be sorted out next week or actually another major delay. It wasn't the top idea in my mind either: I had my own work to get back to. With online correspondence it's so easy for days to pass between messages, or even weeks when one party forgets to follow up. It's like what Walter always stresses: issues that are going on for months can be hashed out at DConf in minutes, where we're all in person.
Mar 06
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:Hello everyone, ...Nice story and a bit scary one. There must have be good relief in the end. We should be glad to have people that give up their own time to maintain projects like this. Thanks for your work, Matheus.
Mar 07
On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:Hello everyone, [...] ~Dennis KorpelUnfortunately, what is released as 2.110 differs from what's in the 2.110.rc.1 . Not by much, but by enough so that the release doesn't build SDC even though rc.1 does. I'm a bit frustrated because all my CI pipelines are now broken, even though I test the beta channel and rc to make sure I don't find myself in that spot. What's the point of making a rc if this is not going to be what is shipped? Anyways, this is the failure I'm getting: https://github.com/snazzy-d/sdc/actions/runs/13713868682/job/38372203222
Mar 07
On Friday, 7 March 2025 at 12:24:54 UTC, deadalnix wrote:On Thursday, 6 March 2025 at 13:30:51 UTC, Dennis wrote:I was able to reduce this to https://github.com/dlang/dmd/issues/20965 . The reduced error happens in 2.109 as well, and I assume it now triggers in the overall project in 2.110 and not 2.109 because the template instactiation order changed somehow.Hello everyone, [...] ~Dennis KorpelUnfortunately, what is released as 2.110 differs from what's in the 2.110.rc.1 . Not by much, but by enough so that the release doesn't build SDC even though rc.1 does. I'm a bit frustrated because all my CI pipelines are now broken, even though I test the beta channel and rc to make sure I don't find myself in that spot. What's the point of making a rc if this is not going to be what is shipped? Anyways, this is the failure I'm getting: https://github.com/snazzy-d/sdc/actions/runs/13713868682/job/38372203222
Mar 07
I big thank you to Dennis and Iain for the tremendous amount of work solving this problem! I hope Iain gets better soon!
Mar 07