D.gnu - Restructuring the download archive
- Johannes Pfau (94/94) Jul 10 2015 I thinks it's time to push some new binary releases to
- Iain Buclaw via D.gnu (46/75) Jul 10 2015 I think we can break anything that isn't a short name (should be a symli...
- Johannes Pfau (30/77) Jul 11 2015 Isn't the frontend version usually more important for the user? The
- Iain Buclaw via D.gnu (13/76) Jul 11 2015 For us, it's toolchain that is more important. The GCC version does affe...
- Johannes Pfau (9/91) Jul 12 2015 OK. OTOH this suggests we should also build new frontend releases for
- Iain Buclaw via D.gnu (33/124) Jul 12 2015 Shouldn't be too much, currently there is not a high frequency of fronte...
I thinks it's time to push some new binary releases to gdcproject.org (gcc-5, intrinsics for checkedint, ...). We should first get some structure into our download archive though: ftp://ftp.gdcproject.org/binaries/ The first question is whether we can break old download links. I've hardcoded one or two toolchain URLs in the build scripts, but that could be changed easily. Another question is how exactly the new directory layout should look like. I think we definitely need host folders as a top-level structure: /binaries |- x86_64-linux-gnu |- x86_64-w64-mingw32 |- arm-linux-gnueabi Do we want to have the target as a directory? GCC version? Frontend version? I think the frontend version is usually more important for users than gcc version. So if we don't use the GCC version we still have two options. dmdversion/target or target/dmdversion: /binaries |- x86_64-linux-gnu | |- 2.061 | | |- x86_64-w64-mingw32 | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz | | |- arm-linux-gnueabi | |- 2.062 |- x86_64-w64-mingw32 |- arm-linux-gnueabi Another question: DMD frontend sub-releases in one folder? (2.061|2.061.1|2.061.2) We could also use symlinks for tools to point to the latest releases: /binaries |- x86_64-linux-gnu | |- 2.061 | | |- x86_64-w64-mingw32 | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz | | |- arm-linux-gnueabi | | |- x86_64-w64-mingw32_2.061_LATEST.tar.xz | |- 2.062 | |- x86_64-w64-mingw32_LATEST.tar.xz |- x86_64-w64-mingw32 |- arm-linux-gnueabi And how should we present naive releases? Some folder structure and target as usual or we could use some other name for target (e.g. native). We could also skip the target directory for native builds: /binaries |- x86_64-linux-gnu | |- 2.061 | | |- x86_64-w64-mingw32 | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz | | |- arm-linux-gnueabi | | |- x86_64-w64-mingw32_2.061_LATEST.tar.xz | | |- x86_64-linux-gnu_2.061_gcc4.9.2_AAAA_20150405.tar.xz | | |- x86_64-linux-gnu_2.061_gcc4.9.2_BBBB_20150405.tar.xz | | |- x86_64-linux-gnu_2.061_gcc4.9.2_CCCC_20150405.tar.xz | |- 2.062 | |- x86_64-w64-mingw32_LATEST.tar.xz |- x86_64-w64-mingw32 |- arm-linux-gnueabi However, this could get confusing as we'll have many files in the dmdver folders. We could also move the -LATEST links into the target dirs for all builds except for native ones. This way native builds will be more prominent: /binaries |- x86_64-linux-gnu | |- 2.061 | | |- x86_64-w64-mingw32 | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz | | | |- x86_64-w64-mingw32_2.061_LATEST.tar.xz | | |- arm-linux-gnueabi | | |- x86_64-linux-gnu | | | |- x86_64-linux-gnu_2.061_gcc4.9.2_AAAA_20150405.tar.xz | | | |- x86_64-linux-gnu_2.061_gcc4.9.2_BBBB_20150405.tar.xz | | | |- x86_64-linux-gnu_2.061_gcc4.9.2_CCCC_20150405.tar.xz | | |- x86_64-linux-gnu_2.061_LATEST.tar.xz | |- 2.062 | |- x86_64-w64-mingw32_LATEST.tar.xz |- x86_64-w64-mingw32 |- arm-linux-gnueabi |- x86_64-linux-gnu_LATEST.tar.xz
Jul 10 2015
On 10 July 2015 at 14:45, Johannes Pfau via D.gnu <d.gnu puremagic.com> wrote:I thinks it's time to push some new binary releases to gdcproject.org (gcc-5, intrinsics for checkedint, ...). We should first get some structure into our download archive though: ftp://ftp.gdcproject.org/binaries/ The first question is whether we can break old download links. I've hardcoded one or two toolchain URLs in the build scripts, but that could be changed easily.I think we can break anything that isn't a short name (should be a symlink on the server). We need to keep at least these for Travis to continue working. http://ftp.digitalmars.com/LATEST_GDC By all means, we can raise a PR with Travis to move it elsewhere, then remove the rest of the old structure later.Another question is how exactly the new directory layout should look like. I think we definitely need host folders as a top-level structure: /binaries |- x86_64-linux-gnu |- x86_64-w64-mingw32 |- arm-linux-gnueabi Do we want to have the target as a directory? GCC version? Frontend version? I think the frontend version is usually more important for users than gcc version. So if we don't use the GCC version we still have two options. dmdversion/target or target/dmdversion: /binaries |- x86_64-linux-gnu | |- 2.061 | | |- x86_64-w64-mingw32 | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz | | | |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz | | |- arm-linux-gnueabi | |- 2.062 |- x86_64-w64-mingw32 |- arm-linux-gnueabi Another question: DMD frontend sub-releases in one folder? (2.061|2.061.1|2.061.2)I think GCC releases should take precedence over D Frontend and architecture IMO. As that is the most recognisable (and important) aspect of the name. eg: /binaries |- 4.8.4 | |- x86_64-linux-gnu | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz | |- arm-linux-gnueabi | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | |- x86_64-w64-mingw32 | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz |- 4.9.2 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 |- 5.1.0 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 I want to avoid a situation where there is both gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the same directory. We are already overcrowding the name with a version number pair, adding in a timestamp just makes it unreadable. So, rather than having timestamps, I think we should just co-ordinate our binary releases around GCC major and minor updates. My preference being, if we really must push in an update tarball whose GCC/DMD version pair is already on our FTP server, I'd rather just overwrite the previous version, maybe moving the original into an old-releases directory. Regards Iain
Jul 10 2015
Am Fri, 10 Jul 2015 16:26:35 +0200 schrieb "Iain Buclaw via D.gnu" <d.gnu puremagic.com>:Isn't the frontend version usually more important for the user? The frontend version determines if user code compiles or not and the frontend version is used to compare compilers. The GCC version has very little effect for users. This way if users are looking for a specific frontend version they'll have to walk all the gcc version folders.Another question: DMD frontend sub-releases in one folder? (2.061|2.061.1|2.061.2)I think GCC releases should take precedence over D Frontend and architecture IMO. As that is the most recognisable (and important) aspect of the name.eg: /binaries |- 4.8.4 | |- x86_64-linux-gnu | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz | |- arm-linux-gnueabi | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | |- x86_64-w64-mingw32 | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz |- 4.9.2 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 |- 5.1.0 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 I want to avoid a situation where there is both gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the same directory. We are already overcrowding the name with a version number pair, adding in a timestamp just makes it unreadable. So, rather than having timestamps, I think we should just co-ordinate our binary releases around GCC major and minor updates.And frontend releases? If the dmd team really implements a 2-month release schedule we'd miss some frontend versions otherwise.My preference being, if we really must push in an update tarball whose GCC/DMD version pair is already on our FTP server, I'd rather just overwrite the previous version, maybe moving the original into an old-releases directory.We can probably avoid the timestamp and git revision. We shouldn't overwrite files though: users might check checksums for downloads or do URL-based caching. I'd add a counter for further releases at the end instead: gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r2.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r3.tar.xz (The reason why I initially included both git-id and build date is this: If there's a problem with the build system we might need to rebuild the same revision of a toolchain. We than have a higher build count for one toolchain, but it's probably not a big issue). BTW: Maybe we can implement a download mask for older releases at some point: ______________________________________ | | | Host: ______________ | | Target: ______________ | | Frontend version: ______________ | | Release: LATEST________ | | | ---------------------------------------
Jul 11 2015
On 11 July 2015 at 10:01, Johannes Pfau via D.gnu <d.gnu puremagic.com> wrote:Am Fri, 10 Jul 2015 16:26:35 +0200 schrieb "Iain Buclaw via D.gnu" <d.gnu puremagic.com>:For us, it's toolchain that is more important. The GCC version does affect users, but in more subtle ways. For instance, a new optimisation pass may corrupt programs, or if they get a version of gdc built against gcc-5.1 and debug a program on Ubuntu 12.04LTS, that gdb won't really work very well at all. The overall experience is much better if you install the correct version that works with your running system.Isn't the frontend version usually more important for the user? The frontend version determines if user code compiles or not and the frontend version is used to compare compilers. The GCC version has very little effect for users. This way if users are looking for a specific frontend version they'll have to walk all the gcc version folders.Another question: DMD frontend sub-releases in one folder? (2.061|2.061.1|2.061.2)I think GCC releases should take precedence over D Frontend and architecture IMO. As that is the most recognisable (and important) aspect of the name.I still feel that we should avoid imposing a three layer versioning system. Either build a new tarball after each GCC major/minor release, or a new tarball when we update to the next D version (and it has matured for a bit). Iain.eg: /binaries |- 4.8.4 | |- x86_64-linux-gnu | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz | |- arm-linux-gnueabi | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | |- x86_64-w64-mingw32 | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz |- 4.9.2 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 |- 5.1.0 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 I want to avoid a situation where there is both gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the same directory. We are already overcrowding the name with a version number pair, adding in a timestamp just makes it unreadable. So, rather than having timestamps, I think we should just co-ordinate our binary releases around GCC major and minor updates.And frontend releases? If the dmd team really implements a 2-month release schedule we'd miss some frontend versions otherwise.My preference being, if we really must push in an update tarball whose GCC/DMD version pair is already on our FTP server, I'd rather just overwrite the previous version, maybe moving the original into an old-releases directory.We can probably avoid the timestamp and git revision. We shouldn't overwrite files though: users might check checksums for downloads or do URL-based caching. I'd add a counter for further releases at the end instead: gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r2.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r3.tar.xz
Jul 11 2015
Am Sat, 11 Jul 2015 13:37:42 +0200 schrieb "Iain Buclaw via D.gnu" <d.gnu puremagic.com>:On 11 July 2015 at 10:01, Johannes Pfau via D.gnu <d.gnu puremagic.com> wrote:OK. OTOH this suggests we should also build new frontend releases for older GCC versions for these users? But this would probably eat quite some disk space.Isn't the frontend version usually more important for the user? The frontend version determines if user code compiles or not and the frontend version is used to compare compilers. The GCC version has very little effect for users. This way if users are looking for a specific frontend version they'll have to walk all the gcc version folders.For us, it's toolchain that is more important. The GCC version does affect users, but in more subtle ways. For instance, a new optimisation pass may corrupt programs, or if they get a version of gdc built against gcc-5.1 and debug a program on Ubuntu 12.04LTS, that gdb won't really work very well at all. The overall experience is much better if you install the correct version that works with your running system.Sure. I just meant that if we have to provide new releases for some reason (like if a toolchain doesn't work at all cause we forgot to install some files or similar cases) we should never simply override releases. But normally we should ship only one release without counter.I still feel that we should avoid imposing a three layer versioning system. Either build a new tarball after each GCC major/minor release, or a new tarball when we update to the next D version (and it has matured for a bit).eg: /binaries |- 4.8.4 | |- x86_64-linux-gnu | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz | |- arm-linux-gnueabi | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | |- x86_64-w64-mingw32 | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz |- 4.9.2 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 |- 5.1.0 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 I want to avoid a situation where there is both gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the same directory. We are already overcrowding the name with a version number pair, adding in a timestamp just makes it unreadable. So, rather than having timestamps, I think we should just co-ordinate our binary releases around GCC major and minor updates.And frontend releases? If the dmd team really implements a 2-month release schedule we'd miss some frontend versions otherwise.My preference being, if we really must push in an update tarball whose GCC/DMD version pair is already on our FTP server, I'd rather just overwrite the previous version, maybe moving the original into an old-releases directory.We can probably avoid the timestamp and git revision. We shouldn't overwrite files though: users might check checksums for downloads or do URL-based caching. I'd add a counter for further releases at the end instead: gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r2.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r3.tar.xz
Jul 12 2015
On 12 July 2015 at 10:33, Johannes Pfau via D.gnu <d.gnu puremagic.com> wrote:Am Sat, 11 Jul 2015 13:37:42 +0200 schrieb "Iain Buclaw via D.gnu" <d.gnu puremagic.com>:Shouldn't be too much, currently there is not a high frequency of frontend updates (once every 8 or so months), and there are typically 2 minor GCC releases a year for a particular version. I think it's good as a general rule of thumb to only update/maintain the D frontend for the last three GCC releases.On 11 July 2015 at 10:01, Johannes Pfau via D.gnu <d.gnu puremagic.com> wrote:OK. OTOH this suggests we should also build new frontend releases for older GCC versions for these users? But this would probably eat quite some disk space.Isn't the frontend version usually more important for the user? The frontend version determines if user code compiles or not and the frontend version is used to compare compilers. The GCC version has very little effect for users. This way if users are looking for a specific frontend version they'll have to walk all the gcc version folders.For us, it's toolchain that is more important. The GCC version does affect users, but in more subtle ways. For instance, a new optimisation pass may corrupt programs, or if they get a version of gdc built against gcc-5.1 and debug a program on Ubuntu 12.04LTS, that gdb won't really work very well at all. The overall experience is much better if you install the correct version that works with your running system.If the toolchain is broken or files are missing, I'd much rather remove the broken tarballs if we discover such problems quickly (ie: within a week or two). Otherwise it's delay until next GCC/DMD release. If upstream DMD actually commit to a bi-monthly schedule, this probably won't be so bad. It would mean that there are 7/8 releases a year per GCC version. Think in the order of: gcc-5.1+2.068 gcc-5.1+2.069 gcc-5.1+2.070 gcc-5.2+2.070 gcc-5.2+2.071 gcc-5.2+2.072 gcc-5.2+2.073 gcc-5.3+2.073 However I hope that people don't get comfortable with this kind of high maintenance. If we were part of mainline GCC, I'd expect that we won't have the opportunity to do a major frontend update once a release goes out (example, Ubuntu 14.04LTS has GDC with 2.064.2, and this will continue to be supported until 2020=E2=80=A0), point releases would be OK though. =E2=80=A0 I had given forewarning of this, but no one actually considered maintaining 2.064 support in their projects. http://forum.dlang.org/post/lif4uj$2qt7$2 digitalmars.com (It would be nice if the core library maintainers gave this consideration though before breaking ABI) Regards IainSure. I just meant that if we have to provide new releases for some reason (like if a toolchain doesn't work at all cause we forgot to install some files or similar cases) we should never simply override releases. But normally we should ship only one release without counter.I still feel that we should avoid imposing a three layer versioning system. Either build a new tarball after each GCC major/minor release, or a new tarball when we update to the next D version (and it has matured for a bit).eg: /binaries |- 4.8.4 | |- x86_64-linux-gnu | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz | |- arm-linux-gnueabi | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | |- x86_64-w64-mingw32 | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz |- 4.9.2 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 |- 5.1.0 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 I want to avoid a situation where there is both gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the same directory. We are already overcrowding the name with a version number pair, adding in a timestamp just makes it unreadable. So, rather than having timestamps, I think we should just co-ordinate our binary releases around GCC major and minor updates.And frontend releases? If the dmd team really implements a 2-month release schedule we'd miss some frontend versions otherwise.My preference being, if we really must push in an update tarball whose GCC/DMD version pair is already on our FTP server, I'd rather just overwrite the previous version, maybe moving the original into an old-releases directory.We can probably avoid the timestamp and git revision. We shouldn't overwrite files though: users might check checksums for downloads or do URL-based caching. I'd add a counter for further releases at the end instead: gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r2.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r3.tar.xz
Jul 12 2015