www.digitalmars.com         C & C++   DMDScript  

D.gnu - Restructuring the download archive

reply "Johannes Pfau" <nospam example.com> writes:
I thinks it's time to push some new binary releases to 
gdcproject.org (gcc-5, intrinsics for checkedint, ...). We should 
first get some structure into our download archive though: 
ftp://ftp.gdcproject.org/binaries/

The first question is whether we can break old download links. 
I've hardcoded one or two toolchain URLs in the build scripts, 
but that could be changed easily.


Another question is how exactly the new directory layout should 
look like. I think we definitely need host folders as a top-level 
structure:

/binaries
  |- x86_64-linux-gnu
  |- x86_64-w64-mingw32
  |- arm-linux-gnueabi

Do we want to have the target as a directory? GCC version? 
Frontend version?
I think the frontend version is usually more important for users 
than gcc version. So if we don't use the GCC version we still 
have two options. dmdversion/target or target/dmdversion:

/binaries
  |- x86_64-linux-gnu
  |  |- 2.061
  |  |  |- x86_64-w64-mingw32
  |  |  |  |- 
x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz
  |  |  |  |- 
x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz
  |  |  |- arm-linux-gnueabi
  |  |- 2.062
  |- x86_64-w64-mingw32
  |- arm-linux-gnueabi

Another question: DMD frontend sub-releases in one folder? 
(2.061|2.061.1|2.061.2)

We could also use symlinks for tools to point to the latest 
releases:

/binaries
  |- x86_64-linux-gnu
  |  |- 2.061
  |  |  |- x86_64-w64-mingw32
  |  |  |  |- 
x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz
  |  |  |  |- 
x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz
  |  |  |- arm-linux-gnueabi
  |  |  |- x86_64-w64-mingw32_2.061_LATEST.tar.xz
  |  |- 2.062
  |  |- x86_64-w64-mingw32_LATEST.tar.xz
  |- x86_64-w64-mingw32
  |- arm-linux-gnueabi

And how should we present naive releases? Some folder structure 
and target as usual or we could use some other name for target 
(e.g. native). We could also skip the target directory for native 
builds:

/binaries
  |- x86_64-linux-gnu
  |  |- 2.061
  |  |  |- x86_64-w64-mingw32
  |  |  |  |- 
x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz
  |  |  |  |- 
x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz
  |  |  |- arm-linux-gnueabi
  |  |  |- x86_64-w64-mingw32_2.061_LATEST.tar.xz
  |  |  |- x86_64-linux-gnu_2.061_gcc4.9.2_AAAA_20150405.tar.xz
  |  |  |- x86_64-linux-gnu_2.061_gcc4.9.2_BBBB_20150405.tar.xz
  |  |  |- x86_64-linux-gnu_2.061_gcc4.9.2_CCCC_20150405.tar.xz
  |  |- 2.062
  |  |- x86_64-w64-mingw32_LATEST.tar.xz
  |- x86_64-w64-mingw32
  |- arm-linux-gnueabi

However, this could get confusing as we'll have many files in the 
dmdver folders. We could also move the -LATEST links into the 
target dirs for all builds except for native ones. This way 
native builds will be more prominent:

/binaries
  |- x86_64-linux-gnu
  |  |- 2.061
  |  |  |- x86_64-w64-mingw32
  |  |  |  |- 
x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz
  |  |  |  |- 
x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz
  |  |  |  |- x86_64-w64-mingw32_2.061_LATEST.tar.xz
  |  |  |- arm-linux-gnueabi
  |  |  |- x86_64-linux-gnu
  |  |  |  |- x86_64-linux-gnu_2.061_gcc4.9.2_AAAA_20150405.tar.xz
  |  |  |  |- x86_64-linux-gnu_2.061_gcc4.9.2_BBBB_20150405.tar.xz
  |  |  |  |- x86_64-linux-gnu_2.061_gcc4.9.2_CCCC_20150405.tar.xz
  |  |  |- x86_64-linux-gnu_2.061_LATEST.tar.xz
  |  |- 2.062
  |  |- x86_64-w64-mingw32_LATEST.tar.xz
  |- x86_64-w64-mingw32
  |- arm-linux-gnueabi
  |- x86_64-linux-gnu_LATEST.tar.xz
Jul 10 2015
parent reply "Iain Buclaw via D.gnu" <d.gnu puremagic.com> writes:
On 10 July 2015 at 14:45, Johannes Pfau via D.gnu <d.gnu puremagic.com>
wrote:

 I thinks it's time to push some new binary releases to gdcproject.org
 (gcc-5, intrinsics for checkedint, ...). We should first get some structure
 into our download archive though: ftp://ftp.gdcproject.org/binaries/

 The first question is whether we can break old download links. I've
 hardcoded one or two toolchain URLs in the build scripts, but that could be
 changed easily.
I think we can break anything that isn't a short name (should be a symlink on the server). We need to keep at least these for Travis to continue working. http://ftp.digitalmars.com/LATEST_GDC By all means, we can raise a PR with Travis to move it elsewhere, then remove the rest of the old structure later.
 Another question is how exactly the new directory layout should look like.
 I think we definitely need host folders as a top-level structure:

 /binaries
  |- x86_64-linux-gnu
  |- x86_64-w64-mingw32
  |- arm-linux-gnueabi

 Do we want to have the target as a directory? GCC version? Frontend
 version?
 I think the frontend version is usually more important for users than gcc
 version. So if we don't use the GCC version we still have two options.
 dmdversion/target or target/dmdversion:

 /binaries
  |- x86_64-linux-gnu
  |  |- 2.061
  |  |  |- x86_64-w64-mingw32
  |  |  |  |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_AAAA_20140430.tar.xz
  |  |  |  |- x86_64-w64-mingw32_2.061.2_gcc4.8.2_BBBB_20140430.tar.xz
  |  |  |- arm-linux-gnueabi
  |  |- 2.062
  |- x86_64-w64-mingw32
  |- arm-linux-gnueabi

 Another question: DMD frontend sub-releases in one folder?
 (2.061|2.061.1|2.061.2)
I think GCC releases should take precedence over D Frontend and architecture IMO. As that is the most recognisable (and important) aspect of the name. eg: /binaries |- 4.8.4 | |- x86_64-linux-gnu | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz | | |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz | | |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz | |- arm-linux-gnueabi | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz | |- x86_64-w64-mingw32 | | |- gdc-4.8.4+2.061.2.tar.xz | | |- gdc-4.8.4+2.062.tar.xz |- 4.9.2 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 |- 5.1.0 | |- arm-linux-gnueabi | |- x86_64-linux-gnu | |- x86_64-w64-mingw32 I want to avoid a situation where there is both gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the same directory. We are already overcrowding the name with a version number pair, adding in a timestamp just makes it unreadable. So, rather than having timestamps, I think we should just co-ordinate our binary releases around GCC major and minor updates. My preference being, if we really must push in an update tarball whose GCC/DMD version pair is already on our FTP server, I'd rather just overwrite the previous version, maybe moving the original into an old-releases directory. Regards Iain
Jul 10 2015
parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 10 Jul 2015 16:26:35 +0200
schrieb "Iain Buclaw via D.gnu" <d.gnu puremagic.com>:

 Another question: DMD frontend sub-releases in one folder?
 (2.061|2.061.1|2.061.2)
I think GCC releases should take precedence over D Frontend and architecture IMO. As that is the most recognisable (and important) aspect of the name.
Isn't the frontend version usually more important for the user? The frontend version determines if user code compiles or not and the frontend version is used to compare compilers. The GCC version has very little effect for users. This way if users are looking for a specific frontend version they'll have to walk all the gcc version folders.
 eg:
 
 /binaries
  |- 4.8.4
  |  |- x86_64-linux-gnu
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
  |  |  |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz
  |  |  |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz
  |  |  |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz
  |  |  |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz
  |  |- arm-linux-gnueabi
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
  |  |- x86_64-w64-mingw32
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
   |- 4.9.2
  |  |- arm-linux-gnueabi
  |  |- x86_64-linux-gnu
  |  |- x86_64-w64-mingw32
   |- 5.1.0
  |  |- arm-linux-gnueabi
  |  |- x86_64-linux-gnu
  |  |- x86_64-w64-mingw32
 
 
 I want to avoid a situation where there is both
 gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the same
 directory.  We are already overcrowding the name with a version
 number pair, adding in a timestamp just makes it unreadable.
 
 So, rather than having timestamps, I think we should just co-ordinate
 our binary releases around GCC major and minor updates.
And frontend releases? If the dmd team really implements a 2-month release schedule we'd miss some frontend versions otherwise.
  My
 preference being, if we really must push in an update tarball whose
 GCC/DMD version pair is already on our FTP server, I'd rather just
 overwrite the previous version, maybe moving the original into an
 old-releases directory.
We can probably avoid the timestamp and git revision. We shouldn't overwrite files though: users might check checksums for downloads or do URL-based caching. I'd add a counter for further releases at the end instead: gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r2.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r3.tar.xz (The reason why I initially included both git-id and build date is this: If there's a problem with the build system we might need to rebuild the same revision of a toolchain. We than have a higher build count for one toolchain, but it's probably not a big issue). BTW: Maybe we can implement a download mask for older releases at some point: ______________________________________ | | | Host: ______________ | | Target: ______________ | | Frontend version: ______________ | | Release: LATEST________ | | | ---------------------------------------
Jul 11 2015
parent reply "Iain Buclaw via D.gnu" <d.gnu puremagic.com> writes:
On 11 July 2015 at 10:01, Johannes Pfau via D.gnu <d.gnu puremagic.com>
wrote:

 Am Fri, 10 Jul 2015 16:26:35 +0200
 schrieb "Iain Buclaw via D.gnu" <d.gnu puremagic.com>:

 Another question: DMD frontend sub-releases in one folder?
 (2.061|2.061.1|2.061.2)
I think GCC releases should take precedence over D Frontend and architecture IMO. As that is the most recognisable (and important) aspect of the name.
Isn't the frontend version usually more important for the user? The frontend version determines if user code compiles or not and the frontend version is used to compare compilers. The GCC version has very little effect for users. This way if users are looking for a specific frontend version they'll have to walk all the gcc version folders.
For us, it's toolchain that is more important. The GCC version does affect users, but in more subtle ways. For instance, a new optimisation pass may corrupt programs, or if they get a version of gdc built against gcc-5.1 and debug a program on Ubuntu 12.04LTS, that gdb won't really work very well at all. The overall experience is much better if you install the correct version that works with your running system.
 eg:

 /binaries
  |- 4.8.4
  |  |- x86_64-linux-gnu
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
  |  |  |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz
  |  |  |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz
  |  |  |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz
  |  |  |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz
  |  |- arm-linux-gnueabi
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
  |  |- x86_64-w64-mingw32
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
   |- 4.9.2
  |  |- arm-linux-gnueabi
  |  |- x86_64-linux-gnu
  |  |- x86_64-w64-mingw32
   |- 5.1.0
  |  |- arm-linux-gnueabi
  |  |- x86_64-linux-gnu
  |  |- x86_64-w64-mingw32


 I want to avoid a situation where there is both
 gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the same
 directory.  We are already overcrowding the name with a version
 number pair, adding in a timestamp just makes it unreadable.

 So, rather than having timestamps, I think we should just co-ordinate
 our binary releases around GCC major and minor updates.
And frontend releases? If the dmd team really implements a 2-month release schedule we'd miss some frontend versions otherwise.
  My
 preference being, if we really must push in an update tarball whose
 GCC/DMD version pair is already on our FTP server, I'd rather just
 overwrite the previous version, maybe moving the original into an
 old-releases directory.
We can probably avoid the timestamp and git revision. We shouldn't overwrite files though: users might check checksums for downloads or do URL-based caching. I'd add a counter for further releases at the end instead: gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r2.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r3.tar.xz
I still feel that we should avoid imposing a three layer versioning system. Either build a new tarball after each GCC major/minor release, or a new tarball when we update to the next D version (and it has matured for a bit). Iain.
Jul 11 2015
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 11 Jul 2015 13:37:42 +0200
schrieb "Iain Buclaw via D.gnu" <d.gnu puremagic.com>:

 On 11 July 2015 at 10:01, Johannes Pfau via D.gnu
 <d.gnu puremagic.com> wrote:
 
 Isn't the frontend version usually more important for the user? The
 frontend version determines if user code compiles or not and the
 frontend version is used to compare compilers. The GCC version has
 very little effect for users. This way if users are looking for a
 specific frontend version they'll have to walk all the gcc version
 folders.
For us, it's toolchain that is more important. The GCC version does affect users, but in more subtle ways. For instance, a new optimisation pass may corrupt programs, or if they get a version of gdc built against gcc-5.1 and debug a program on Ubuntu 12.04LTS, that gdb won't really work very well at all. The overall experience is much better if you install the correct version that works with your running system.
OK. OTOH this suggests we should also build new frontend releases for older GCC versions for these users? But this would probably eat quite some disk space.
 
 
 eg:

 /binaries
  |- 4.8.4
  |  |- x86_64-linux-gnu
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
  |  |  |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz
  |  |  |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz
  |  |  |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz
  |  |  |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz
  |  |- arm-linux-gnueabi
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
  |  |- x86_64-w64-mingw32
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
   |- 4.9.2
  |  |- arm-linux-gnueabi
  |  |- x86_64-linux-gnu
  |  |- x86_64-w64-mingw32
   |- 5.1.0
  |  |- arm-linux-gnueabi
  |  |- x86_64-linux-gnu
  |  |- x86_64-w64-mingw32


 I want to avoid a situation where there is both
 gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the
 same directory.  We are already overcrowding the name with a
 version number pair, adding in a timestamp just makes it
 unreadable.

 So, rather than having timestamps, I think we should just
 co-ordinate our binary releases around GCC major and minor
 updates.
And frontend releases? If the dmd team really implements a 2-month release schedule we'd miss some frontend versions otherwise.
  My
 preference being, if we really must push in an update tarball
 whose GCC/DMD version pair is already on our FTP server, I'd
 rather just overwrite the previous version, maybe moving the
 original into an old-releases directory.
We can probably avoid the timestamp and git revision. We shouldn't overwrite files though: users might check checksums for downloads or do URL-based caching. I'd add a counter for further releases at the end instead: gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r2.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r3.tar.xz
I still feel that we should avoid imposing a three layer versioning system. Either build a new tarball after each GCC major/minor release, or a new tarball when we update to the next D version (and it has matured for a bit).
Sure. I just meant that if we have to provide new releases for some reason (like if a toolchain doesn't work at all cause we forgot to install some files or similar cases) we should never simply override releases. But normally we should ship only one release without counter.
Jul 12 2015
parent "Iain Buclaw via D.gnu" <d.gnu puremagic.com> writes:
On 12 July 2015 at 10:33, Johannes Pfau via D.gnu <d.gnu puremagic.com>
wrote:

 Am Sat, 11 Jul 2015 13:37:42 +0200
 schrieb "Iain Buclaw via D.gnu" <d.gnu puremagic.com>:

 On 11 July 2015 at 10:01, Johannes Pfau via D.gnu
 <d.gnu puremagic.com> wrote:

 Isn't the frontend version usually more important for the user? The
 frontend version determines if user code compiles or not and the
 frontend version is used to compare compilers. The GCC version has
 very little effect for users. This way if users are looking for a
 specific frontend version they'll have to walk all the gcc version
 folders.
For us, it's toolchain that is more important. The GCC version does affect users, but in more subtle ways. For instance, a new optimisation pass may corrupt programs, or if they get a version of gdc built against gcc-5.1 and debug a program on Ubuntu 12.04LTS, that gdb won't really work very well at all. The overall experience is much better if you install the correct version that works with your running system.
OK. OTOH this suggests we should also build new frontend releases for older GCC versions for these users? But this would probably eat quite some disk space.
Shouldn't be too much, currently there is not a high frequency of frontend updates (once every 8 or so months), and there are typically 2 minor GCC releases a year for a particular version. I think it's good as a general rule of thumb to only update/maintain the D frontend for the last three GCC releases.
 eg:

 /binaries
  |- 4.8.4
  |  |- x86_64-linux-gnu
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
  |  |  |- gdc-4.8.4-arm-linux-gnueabi+2.061.2.tar.xz
  |  |  |- gdc-4.8.4-arm-linux-gnueabi+2.062.tar.xz
  |  |  |- gdc-4.8.4-x86_64-w64-mingw32+2.061.2.tar.xz
  |  |  |- gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz
  |  |- arm-linux-gnueabi
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
  |  |- x86_64-w64-mingw32
  |  |  |- gdc-4.8.4+2.061.2.tar.xz
  |  |  |- gdc-4.8.4+2.062.tar.xz
   |- 4.9.2
  |  |- arm-linux-gnueabi
  |  |- x86_64-linux-gnu
  |  |- x86_64-w64-mingw32
   |- 5.1.0
  |  |- arm-linux-gnueabi
  |  |- x86_64-linux-gnu
  |  |- x86_64-w64-mingw32


 I want to avoid a situation where there is both
 gdc-4.8.2-2.061.2-20140430 and gdc-4.8.2-2.061.2-20150430 in the
 same directory.  We are already overcrowding the name with a
 version number pair, adding in a timestamp just makes it
 unreadable.

 So, rather than having timestamps, I think we should just
 co-ordinate our binary releases around GCC major and minor
 updates.
And frontend releases? If the dmd team really implements a 2-month release schedule we'd miss some frontend versions otherwise.
  My
 preference being, if we really must push in an update tarball
 whose GCC/DMD version pair is already on our FTP server, I'd
 rather just overwrite the previous version, maybe moving the
 original into an old-releases directory.
We can probably avoid the timestamp and git revision. We shouldn't overwrite files though: users might check checksums for downloads or do URL-based caching. I'd add a counter for further releases at the end instead: gdc-4.8.4-x86_64-w64-mingw32+2.062.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r2.tar.xz gdc-4.8.4-x86_64-w64-mingw32+2.062_r3.tar.xz
I still feel that we should avoid imposing a three layer versioning system. Either build a new tarball after each GCC major/minor release, or a new tarball when we update to the next D version (and it has matured for a bit).
Sure. I just meant that if we have to provide new releases for some reason (like if a toolchain doesn't work at all cause we forgot to install some files or similar cases) we should never simply override releases. But normally we should ship only one release without counter.
If the toolchain is broken or files are missing, I'd much rather remove the broken tarballs if we discover such problems quickly (ie: within a week or two). Otherwise it's delay until next GCC/DMD release. If upstream DMD actually commit to a bi-monthly schedule, this probably won't be so bad. It would mean that there are 7/8 releases a year per GCC version. Think in the order of: gcc-5.1+2.068 gcc-5.1+2.069 gcc-5.1+2.070 gcc-5.2+2.070 gcc-5.2+2.071 gcc-5.2+2.072 gcc-5.2+2.073 gcc-5.3+2.073 However I hope that people don't get comfortable with this kind of high maintenance. If we were part of mainline GCC, I'd expect that we won't have the opportunity to do a major frontend update once a release goes out (example, Ubuntu 14.04LTS has GDC with 2.064.2, and this will continue to be supported until 2020=E2=80=A0), point releases would be OK though. =E2=80=A0 I had given forewarning of this, but no one actually considered maintaining 2.064 support in their projects. http://forum.dlang.org/post/lif4uj$2qt7$2 digitalmars.com (It would be nice if the core library maintainers gave this consideration though before breaking ABI) Regards Iain
Jul 12 2015