www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dlang.org endpoint for permanent links to DIPs

reply Dennis <dkorpel gmail.com> writes:
Given an issue number, getting the URL for that issue is not 
hard. Just put it into this string:
https://issues.dlang.org/show_bug.cgi?id=#####

DIPs, however, have it harder. Early DIPs are at:

https://wiki.dlang.org/DIP##

DIPs above 1000 under review can be found like this:

https://github.com/dlang/DIPs/blob/master/DIPs/DIP####.md

But after review, it will be moved to sub folders like 
'accepted', 'rejected' and 'other'. Consequently, links to DIPs 
can easily become dead, for example:

https://github.com/dlang/dmd/pull/10817

You can use a permanent link to a commit on GitHub, but those are 
very long and refer to the DIP at that moment, not the most 
recent version.

Can we create an endpoint on dlang.org that redirects to the 
latest version of a DIP? Something like:

https://dips.dlang.org/####
https://www.dlang.org/dips/####

I would try to implement this myself but I'm not an experienced 
web developer, and as far as I can tell only the static content 
of dlang.org is open source, so help from people in charge of the 
dlang.org domain would be appreciated.
Jun 03 2020
next sibling parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 3 June 2020 at 12:29:09 UTC, Dennis wrote:

 Given an issue number, getting the URL for that issue is not 
 hard. Just put it into this string:
 https://issues.dlang.org/show_bug.cgi?id=#####

 DIPs, however, have it harder. Early DIPs are at:

 https://wiki.dlang.org/DIP##

 DIPs above 1000 under review can be found like this:

 https://github.com/dlang/DIPs/blob/master/DIPs/DIP####.md

 But after review, it will be moved to sub folders like 
 'accepted', 'rejected' and 'other'. Consequently, links to DIPs 
 can easily become dead, for example:

 https://github.com/dlang/dmd/pull/10817

 You can use a permanent link to a commit on GitHub, but those 
 are very long and refer to the DIP at that moment, not the most 
 recent version.

 Can we create an endpoint on dlang.org that redirects to the 
 latest version of a DIP? Something like:

 https://dips.dlang.org/####
 https://www.dlang.org/dips/####

 I would try to implement this myself but I'm not an experienced 
 web developer, and as far as I can tell only the static content 
 of dlang.org is open source, so help from people in charge of 
 the dlang.org domain would be appreciated.


Yeah, I fully agree that DIP links breaking when their status 
changes is a really big problem. I also wrote to Mike about this 
a while ago.

I guess the easiest/quickest option is to add redirects to 
dlang.org.
This can be done by adding "Redirect 301 /dips/1028 
https://github.com/.../"

https://github.com/dlang/dlang.org/blob/master/.htaccess

However, I think this will require quite a bit of manual 
maintenance and is easy to forget, so the best option is to 
render the Markdown into HTML automatically at the DIPs 
repository, which can then continuously be deployed to e.g. 
dips.dlang.org

If someone is interested in doing this little project, it can be 
done entirely with D, e.g.

https://vibed.org/api/vibe.textfilter.markdown/filterMarkdown

For deployment with Netlify + D, a netlify.sh and netlify.toml 
are required. Something like this:

cat > netlify.toml << EOF
[build]
   base    = ""
   publish = "out"
   command = "bash netlify.sh"
EOF

cat > netlify.sh << 'EOF'


DMD_VERSION="2.092.0"
BUILD_DIR="out"

CURL_FLAGS=(-fsSL --retry 10 --retry-delay 30 --retry-max-time 
600 --connect-timeout 5 --speed-time 30 --speed-limit 1024)

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd "${DIR}"

. "$(curl "${CURL_FLAGS[ ]}" https://dlang.org/install.sh | bash 
-s install "dmd-${DMD_VERSION}" --activate)"


dub
EOF
Jun 03 2020
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Wednesday, 3 June 2020 at 13:25:05 UTC, Seb wrote:

 I guess the easiest/quickest option is to add redirects to 
 dlang.org.
 This can be done by adding "Redirect 301 /dips/1028 
 https://github.com/.../"

 https://github.com/dlang/dlang.org/blob/master/.htaccess


I didn't know about that file, that is very useful.
I think manually adding redirects is a great start.


 However, I think this will require quite a bit of manual 
 maintenance and is easy to forget, so the best option is to 
 render the Markdown into HTML automatically at the DIPs 
 repository, which can then continuously be deployed to e.g. 
 dips.dlang.org


It's better than the current situation where every instance of a 
link to a DIP is a maintenance problem. Would it be hard to 
automatically update the redirects file based on the DIP 
respository?


 If someone is interested in doing this little project, it can 
 be done entirely with D, e.g.

 https://vibed.org/api/vibe.textfilter.markdown/filterMarkdown

 For deployment with Netlify + D, a netlify.sh and netlify.toml 
 are required. Something like this:


That sounds a bit more complicated than needed.
DDoc now supports markdown, so if they should be hosted on the 
site, couldn't they simply become .dd files in the dlang.org 
repository?
Jun 03 2020
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 3 June 2020 at 17:42:26 UTC, Dennis wrote:

 On Wednesday, 3 June 2020 at 13:25:05 UTC, Seb wrote:

 I guess the easiest/quickest option is to add redirects to 
 dlang.org.
 This can be done by adding "Redirect 301 /dips/1028 
 https://github.com/.../"

 https://github.com/dlang/dlang.org/blob/master/.htaccess


 I didn't know about that file, that is very useful.
 I think manually adding redirects is a great start.


 However, I think this will require quite a bit of manual 
 maintenance and is easy to forget, so the best option is to 
 render the Markdown into HTML automatically at the DIPs 
 repository, which can then continuously be deployed to e.g. 
 dips.dlang.org


 It's better than the current situation where every instance of 
 a link to a DIP is a maintenance problem. Would it be hard to 
 automatically update the redirects file based on the DIP 
 respository?


It certainly won't be easy as the tech stack of dlang.org is 
rather old. With the current setup it would probably need to be a 
bot that monitors commits to dlang/dips, parses the current DIP 
list, updates the .htaccess based on this and opens this as a PR 
if the diff changed. Not super hard, but certainly not an hour's 
work.
Furthermore, a lot of people won't be aware that the links from 
GitHub can break and might not even know about dips.dlang.org


 If someone is interested in doing this little project, it can 
 be done entirely with D, e.g.

 https://vibed.org/api/vibe.textfilter.markdown/filterMarkdown

 For deployment with Netlify + D, a netlify.sh and netlify.toml 
 are required. Something like this:


 That sounds a bit more complicated than needed.


Not sure, it's not that hard to use:

https://run.dlang.io/is/zWgHH2

All that's missing is to detect the title, loop over all dips and 
generate a ToC.

Vibe.d's Markdown package is also used to render the package 
READMEs on code.dlang.org and while it used to not support some 
features of GitHub-flavored Markdown (GFM) compatibility, this 
compatibility is not a huge issue these days anymore at least 
with a recent Vibe.d (see e.g. the recent 
https://github.com/vibe-d/vibe.d/pull/2446).
Though I do admit that using a Markdown package that provides 
100% well-tested feature parity with GFM might be an alternative 
approach, but it wouldn't be dog-feeding Vibe.d + D.


 DDoc now supports markdown, so if they should be hosted on the 
 site, couldn't they simply become .dd files in the dlang.org 
 repository?


Well, the Markdown supported by Ddoc has been limited to "only 
the useful parts" by Walter. That's why the documentation calls 
it "Markdown inspired", e.g. 
https://dlang.org/changelog/2.087.0.html#ddoc_markdown
I guess it might work with most DIPs, but if one doesn't even try 
to have feature parity with normal Markdown folks are going to be 
very confused (and I'm not even talking about GitHub-flavored 
Markdown).

BTW dlang.org already uses Vibe.d its ddox documentation, so 
there wouldn't be a big "hurdle" of using it to render a few more 
pages.
Jun 03 2020
next sibling parent David Gileadi <gileadisNOSPM gmail.com> writes:
On 6/3/20 11:22 AM, Seb wrote:

 Well, the Markdown supported by Ddoc has been limited to "only the 
 useful parts" by Walter. That's why the documentation calls it "Markdown 
 inspired", e.g. https://dlang.org/changelog/2.087.0.html#ddoc_markdown
 I guess it might work with most DIPs, but if one doesn't even try to 
 have feature parity with normal Markdown folks are going to be very 
 confused (and I'm not even talking about GitHub-flavored Markdown).


To be specific, compared to regular Markdown GitHub-flavored Markdown 
adds support for tables, task list items, strikethrough, bare URLs as 
autolinks and disallows some raw HTML. Of the above, Ddoc "markdown" 
supports tables and bare autolinks but not task list items or 
strikethrough. Compared to regular Markdown, Ddoc doesn't support 
underscores for emphasis (just asterisks), underline style headings, nor 
indented code blocks (just fenced code blocks).
Jun 03 2020
prev sibling parent reply Dennis <dkorpel gmail.com> writes:
On Wednesday, 3 June 2020 at 18:22:34 UTC, Seb wrote:

 With the current setup it would probably need to be a bot that 
 monitors commits to dlang/dips, parses the current DIP list, 
 updates the .htaccess based on this and opens this as a PR if 
 the diff changed. Not super hard, but certainly not an hour's 
 work.


Watching the dlang/dips repository and parsing the DIP list is 
something that needs to happen regardless of whether you want to 
change .htaccess or generate .html pages based on it. Because I 
don't know the underlying machinery, I don't get why the former 
is hard and the latter is easy.


 Furthermore, a lot of people won't be aware that the links from 
 GitHub can break and might not even know about dips.dlang.org


People who do know can safely use the links in e.g. the DMD 
source code.
People who don't know will not use it, regardless of whether it 
redirects to GitHub or renders the DIP on the site.


 Though I do admit that using a Markdown package that provides 
 100% well-tested feature parity with GFM might be an 
 alternative approach, but it wouldn't be dog-feeding Vibe.d + D.


Is the Vibe.d server that runs the dlang website open source?


 I guess it might work with most DIPs, but if one doesn't even 
 try to have feature parity with normal Markdown folks are going 
 to be very confused (and I'm not even talking about 
 GitHub-flavored Markdown).


Fair enough.
Jun 03 2020
parent Seb <seb wilzba.ch> writes:
On Wednesday, 3 June 2020 at 18:57:18 UTC, Dennis wrote:

 On Wednesday, 3 June 2020 at 18:22:34 UTC, Seb wrote:

 With the current setup it would probably need to be a bot that 
 monitors commits to dlang/dips, parses the current DIP list, 
 updates the .htaccess based on this and opens this as a PR if 
 the diff changed. Not super hard, but certainly not an hour's 
 work.


 Watching the dlang/dips repository and parsing the DIP list is 
 something that needs to happen regardless of whether you want 
 to change .htaccess or generate .html pages based on it. 
 Because I don't know the underlying machinery, I don't get why 
 the former is hard and the latter is easy.


HTML generation can happen directly at the dlang/dips repository 
and can easily be pushed to any file storage (like e.g. netlify).
When you want to adjust the .htaccess you can't just update the 
file in-place, but you need to automatically create PRs which (1) 
need to be reviewed by humans every time (there's no direct merge 
into master) and (2) you need to make sure GitHub doesn't block 
your bot account and (3) that there won't ever be a possibility 
for a merge conflict.
Furthermore, it's a lot harder to debug when/why the bot doesn't 
fire compared to just running a HTML build & deploy script which 
almost every CI can do.


 Furthermore, a lot of people won't be aware that the links 
 from GitHub can break and might not even know about 
 dips.dlang.org


 People who do know can safely use the links in e.g. the DMD 
 source code.
 People who don't know will not use it, regardless of whether it 
 redirects to GitHub or renders the DIP on the site.


The rendered site could be promoted as the official way by 
basically moving the ToC to it and let all links point to it.
Then it would be the only way almost all people look at DIPs 
(like Python does with its PEPs).


 Is the Vibe.d server that runs the dlang website open source?


Technically speaking Vibe.d isn't used to run dlang.org. Ddox 
(Vibe.d) like ddoc generate HTML files which are automatically 
uploaded to an Apache2 server.
Jun 03 2020
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-06-03 15:25, Seb wrote:


 Yeah, I fully agree that DIP links breaking when their status changes is 
 a really big problem. I also wrote to Mike about this a while ago.
 
 I guess the easiest/quickest option is to add redirects to dlang.org.
 This can be done by adding "Redirect 301 /dips/1028 
 https://github.com/.../"
 
 https://github.com/dlang/dlang.org/blob/master/.htaccess
 
 However, I think this will require quite a bit of manual maintenance and 
 is easy to forget, so the best option is to render the Markdown into 
 HTML automatically at the DIPs repository, which can then continuously 
 be deployed to e.g. dips.dlang.org


An alternative would be, for a URL like this:

https://dips.dlang.org/1234

Extract everything after the slash, "1234" in this case, and check a 
list of URLs to redirect to:

https://github.com/dlang/DIPs/blob/master/DIPs/archive/DIP1234.md
https://wiki.dlang.org/DIP1234
https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1234.md
https://github.com/dlang/DIPs/blob/master/DIPs/DIP1234.md
... and so on

Redirect to the first one that exists. I don't know if that's possible 
to do directly in Apache, or if an additonal server side process is 
required to handle this.

I remember we did something similar at work for server side caching of 
web pages. But that checked a local file instead of another URL. That 
was purely done inside Apache.

-- 
/Jacob Carlborg
Jun 04 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 5 June 2020 at 06:53:13 UTC, Jacob Carlborg wrote:

 On 2020-06-03 15:25, Seb wrote:


 [...]


 An alternative would be, for a URL like this:

 https://dips.dlang.org/1234

 Extract everything after the slash, "1234" in this case, and 
 check a list of URLs to redirect to:

 https://github.com/dlang/DIPs/blob/master/DIPs/archive/DIP1234.md
 https://wiki.dlang.org/DIP1234
 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1234.md
 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1234.md
 ... and so on

 Redirect to the first one that exists. I don't know if that's 
 possible to do directly in Apache, or if an additonal server 
 side process is required to handle this.

 I remember we did something similar at work for server side 
 caching of web pages. But that checked a local file instead of 
 another URL. That was purely done inside Apache.


It's already done: https://github.com/dlang/dlang.org/pull/2811

For example:
https://dlang.org/dips/1
https://dlang.org/dips/75
https://dlang.org/dips/1018

;)

P.S. Thanks, Dennis!
Jun 05 2020
parent Jacob Carlborg <doob me.com> writes:
On 2020-06-05 12:55, Petar Kirov [ZombineDev] wrote:


 It's already done: https://github.com/dlang/dlang.org/pull/2811


Hmm, I see. I tried to come up with a solution that didn't require 
manual work to keep the list up to date.

I see that all the old ones are redirected to the wiki, although there 
are a couple of ones on GitHub in the "archived" directory.

-- 
/Jacob Carlborg
Jun 05 2020
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 3 June 2020 at 12:29:09 UTC, Dennis wrote:

 Given an issue number, getting the URL for that issue is not 
 hard. Just put it into this string:
 https://issues.dlang.org/show_bug.cgi?id=#####

 DIPs, however, have it harder. Early DIPs are at:

 https://wiki.dlang.org/DIP##

 DIPs above 1000 under review can be found like this:

 https://github.com/dlang/DIPs/blob/master/DIPs/DIP####.md

 But after review, it will be moved to sub folders like 
 'accepted', 'rejected' and 'other'. Consequently, links to DIPs 
 can easily become dead, for example:

 https://github.com/dlang/dmd/pull/10817

 You can use a permanent link to a commit on GitHub, but those 
 are very long and refer to the DIP at that moment, not the most 
 recent version.


Permanent links to the DIPs are all here:

https://github.com/dlang/DIPs/blob/master/DIPs/README.md
Jun 03 2020
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 3 June 2020 at 14:44:18 UTC, Mike Parker wrote:

 On Wednesday, 3 June 2020 at 12:29:09 UTC, Dennis wrote:

 Given an issue number, getting the URL for that issue is not 
 hard. Just put it into this string:
 https://issues.dlang.org/show_bug.cgi?id=#####

 DIPs, however, have it harder. Early DIPs are at:

 https://wiki.dlang.org/DIP##

 DIPs above 1000 under review can be found like this:

 https://github.com/dlang/DIPs/blob/master/DIPs/DIP####.md

 But after review, it will be moved to sub folders like 
 'accepted', 'rejected' and 'other'. Consequently, links to 
 DIPs can easily become dead, for example:

 https://github.com/dlang/dmd/pull/10817

 You can use a permanent link to a commit on GitHub, but those 
 are very long and refer to the DIP at that moment, not the 
 most recent version.


 Permanent links to the DIPs are all here:

 https://github.com/dlang/DIPs/blob/master/DIPs/README.md


Dennis is talking about the fact that if you link to a DIP under 
review the link while cease to work once its status changes as 
the document has been moved to a different folder.
Jun 03 2020
parent Les De Ridder <les lesderid.net> writes:
On Wednesday, 3 June 2020 at 14:48:22 UTC, Seb wrote:

 On Wednesday, 3 June 2020 at 14:44:18 UTC, Mike Parker wrote:


[...]

 Permanent links to the DIPs are all here:

 https://github.com/dlang/DIPs/blob/master/DIPs/README.md


 Dennis is talking about the fact that if you link to a DIP 
 under review the link while cease to work once its status 
 changes as the document has been moved to a different folder.


A simple but imperfect solution would be to have a directory in 
the
DIPs repo with symlinks to all DIPs, that get updated when DIPs 
change
status.
Jun 03 2020