www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Building the docs with own changes

reply "John Colvin" <john.loughran.colvin gmail.com> writes:
Is there any non-insane way of changing some documentation in a 
module and checking that any links/xrefs you may have added 
actually work?

The dlang.org makefile appears to happily erase uncommitted 
changes in ../druntime and ../phobos. I'm really struggling for 
words to describe how infuriating that is. I ran make and it 
deleted my work in a separate folder. Not OK.
Jun 08 2015
next sibling parent reply "anonymous" <anonymous example.com> writes:
On Monday, 8 June 2015 at 11:27:58 UTC, John Colvin wrote:

 Is there any non-insane way of changing some documentation in a 
 module and checking that any links/xrefs you may have added 
 actually work?

 The dlang.org makefile appears to happily erase uncommitted 
 changes in ../druntime and ../phobos. I'm really struggling for 
 words to describe how infuriating that is. I ran make and it 
 deleted my work in a separate folder. Not OK.


This shouldn't happen. Are you sure you worked in the "phobos" 
directory and not in some versioned one like "phobos-2.067.1"? 
That would be a directory created by dlang.org.

Slightly different topic, related to changing documentation 
without losing your sanity: When building dlang.org, changes in 
druntime/phobos are not properly picked up.

You can try deleting a strategic file like 
dlang.org/web/phobos-prerelease/index.html which should trigger a 
rebuild. See posix.mak for what files are dependencies to the 
different targets.

Or just do a fresh build every time: `make -f posix.mak clean; 
make -f posix.mak phobos-prerelease`.

I have an open pull request that's supposed to fix this:
https://github.com/D-Programming-Language/dlang.org/pull/993
Jun 08 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/15 6:33 AM, anonymous wrote:

 Slightly different topic, related to changing documentation without
 losing your sanity: When building dlang.org, changes in druntime/phobos
 are not properly picked up.

 You can try deleting a strategic file like
 dlang.org/web/phobos-prerelease/index.html which should trigger a
 rebuild. See posix.mak for what files are dependencies to the different
 targets.

 Or just do a fresh build every time: `make -f posix.mak clean; make -f
 posix.mak phobos-prerelease`.

 I have an open pull request that's supposed to fix this:
 https://github.com/D-Programming-Language/dlang.org/pull/993


Yah, that's annoying. Thanks for contributing a fix. For now my typical 
approach is:

rm -f web/phobos-prerelease/index.html && make -j6 LATEST=2.067.1 
phobos-prerelease


Andrei
Jun 08 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/8/15 7:27 AM, John Colvin wrote:

 Is there any non-insane way of changing some documentation in a module
 and checking that any links/xrefs you may have added actually work?

 The dlang.org makefile appears to happily erase uncommitted changes in
 ../druntime and ../phobos. I'm really struggling for words to describe
 how infuriating that is. I ran make and it deleted my work in a separate
 folder. Not OK.


If you just want to check that a file's content is going to show up, I use:

dmd -D -o- -c std/somemod.d

The output is not styled properly. But the output should have working links.

FWIW, I avoid building phobos docs. It's too painful. If I change 
dlang.org docs, I will build those, but not phobos.

-Steve
Jun 08 2015
next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Monday, 8 June 2015 at 13:34:14 UTC, Steven Schveighoffer 
wrote:

 If you just want to check that a file's content is going to 
 show up, I use:

 dmd -D -o- -c std/somemod.d

 The output is not styled properly. But the output should have 
 working links.

 FWIW, I avoid building phobos docs. It's too painful. If I 
 change dlang.org docs, I will build those, but not phobos.

 -Steve


What's painful about building phobos docs? It only takes one 
command.
Jun 08 2015
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/8/15 9:36 AM, Meta wrote:

 On Monday, 8 June 2015 at 13:34:14 UTC, Steven Schveighoffer wrote:

 If you just want to check that a file's content is going to show up, I
 use:

 dmd -D -o- -c std/somemod.d

 The output is not styled properly. But the output should have working
 links.

 FWIW, I avoid building phobos docs. It's too painful. If I change
 dlang.org docs, I will build those, but not phobos.


 What's painful about building phobos docs? It only takes one command.


Yes, after you've set up an appropriate directory structure. And that's 
not easy. I've done it before, a long time ago, so maybe it's gotten 
better. Clearly, John can tell you more ;)

-Steve
Jun 08 2015
prev sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 8 June 2015 at 13:36:08 UTC, Meta wrote:

 On Monday, 8 June 2015 at 13:34:14 UTC, Steven Schveighoffer 
 wrote:

 If you just want to check that a file's content is going to 
 show up, I use:

 dmd -D -o- -c std/somemod.d

 The output is not styled properly. But the output should have 
 working links.

 FWIW, I avoid building phobos docs. It's too painful. If I 
 change dlang.org docs, I will build those, but not phobos.

 -Steve


 What's painful about building phobos docs? It only takes one 
 command.


What's _not_ painful?

Atila
Jun 08 2015
next sibling parent "Meta" <jared771 gmail.com> writes:
On Monday, 8 June 2015 at 15:07:11 UTC, Atila Neves wrote:

 What's _not_ painful?

 Atila


It's painful to get set up, but once you do it's smooth sailing. 
It took one small edit to the makefile to allow me to build the 
website and the docs on Windows.
Jun 08 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/15 8:07 AM, Atila Neves wrote:

 On Monday, 8 June 2015 at 13:36:08 UTC, Meta wrote:

 On Monday, 8 June 2015 at 13:34:14 UTC, Steven Schveighoffer wrote:

 If you just want to check that a file's content is going to show up,
 I use:

 dmd -D -o- -c std/somemod.d

 The output is not styled properly. But the output should have working
 links.

 FWIW, I avoid building phobos docs. It's too painful. If I change
 dlang.org docs, I will build those, but not phobos.

 -Steve


 What's painful about building phobos docs? It only takes one command.


 What's _not_ painful?


I build the docs probably a few dozen times every day. I just press up 
arrow to fetch the previous command, and enter. Takes no perceptible 
time. What's the problem? -- Andrei
Jun 08 2015
parent reply "Joseph Rushton Wakeling" <joseph.wakeling webdrake.net> writes:
On Monday, 8 June 2015 at 16:27:49 UTC, Andrei Alexandrescu wrote:

 I build the docs probably a few dozen times every day. I just 
 press up arrow to fetch the previous command, and enter. Takes 
 no perceptible time. What's the problem? -- Andrei


Well, just for one, I tried building phobos docs a couple of days 
ago while disconnected from the internet; the whole thing failed 
because the clones of stable release repos couldn't be done.

No doubt I'm missing a trick, but it didn't seem very obvious, on 
browsing the makefile, what would get me the simple case of 
building the development phobos docs using my development dmd, 
without needing to worry about any of the other options out there.
Jun 08 2015
next sibling parent reply "Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> writes:
On Monday, 8 June 2015 at 18:06:47 UTC, Joseph Rushton Wakeling 
wrote:

 On Monday, 8 June 2015 at 16:27:49 UTC, Andrei Alexandrescu 
 wrote:

 I build the docs probably a few dozen times every day. I just 
 press up arrow to fetch the previous command, and enter. Takes 
 no perceptible time. What's the problem? -- Andrei


 Well, just for one, I tried building phobos docs a couple of 
 days ago while disconnected from the internet; the whole thing 
 failed because the clones of stable release repos couldn't be 
 done.

 No doubt I'm missing a trick, but it didn't seem very obvious, 
 on browsing the makefile, what would get me the simple case of 
 building the development phobos docs using my development dmd, 
 without needing to worry about any of the other options out 
 there.


Yah, you'd need to pass LATEST=2.067.1 in the command line. Then 
the build process won't need to connect to the internet to fetch 
the latest version.

Could you please make a pull request with the explanations you 
wish you had?


Andrei
Jun 08 2015
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 8 June 2015 at 19:51:28 UTC, Andrei Alexandrescu wrote:

 Yah, you'd need to pass LATEST=2.067.1 in the command line. 
 Then the build process won't need to connect to the internet to 
 fetch the latest version.


"LATEST" detection must go. It makes bisecting impossible.

In general, if we want reproducible builds (and this is required 
for reliable bisection of regressions), we can't rely on 
non-immutable online resources. Fetching / building stable 
compiler versions is OK, but detecting which version is latest 
right now is not.
Jun 08 2015
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 8 June 2015 at 21:53:07 UTC, Vladimir Panteleev wrote:

 On Monday, 8 June 2015 at 19:51:28 UTC, Andrei Alexandrescu 
 wrote:

 Yah, you'd need to pass LATEST=2.067.1 in the command line. 
 Then the build process won't need to connect to the internet 
 to fetch the latest version.


 "LATEST" detection must go. It makes bisecting impossible.


Actually, I can work around this fairly easily with Digger by 
calculating the correct LATEST value at the time and passing it 
as a make parameter :)
Jun 08 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/15 3:05 PM, Vladimir Panteleev wrote:

 On Monday, 8 June 2015 at 21:53:07 UTC, Vladimir Panteleev wrote:

 On Monday, 8 June 2015 at 19:51:28 UTC, Andrei Alexandrescu wrote:

 Yah, you'd need to pass LATEST=2.067.1 in the command line. Then the
 build process won't need to connect to the internet to fetch the
 latest version.


 "LATEST" detection must go. It makes bisecting impossible.


 Actually, I can work around this fairly easily with Digger by
 calculating the correct LATEST value at the time and passing it as a
 make parameter :)


... and there's the well-engineered improvement right there. Bravo. -- 
Andrei
Jun 08 2015
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 8 June 2015 at 22:26:23 UTC, Andrei Alexandrescu wrote:

 ... and there's the well-engineered improvement right there. 
 Bravo. -- Andrei


The network activity for the autodetection is still a minor 
nuisance.
Jun 08 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/15 3:29 PM, Vladimir Panteleev wrote:

 On Monday, 8 June 2015 at 22:26:23 UTC, Andrei Alexandrescu wrote:

 ... and there's the well-engineered improvement right there. Bravo. --
 Andrei


 The network activity for the autodetection is still a minor nuisance.


It can be set in the command line, something we should clarify. Actually 
let me work on a PR real quick. -- Andrei
Jun 08 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/15 3:31 PM, Andrei Alexandrescu wrote:

 On 6/8/15 3:29 PM, Vladimir Panteleev wrote:

 On Monday, 8 June 2015 at 22:26:23 UTC, Andrei Alexandrescu wrote:

 ... and there's the well-engineered improvement right there. Bravo. --
 Andrei


 The network activity for the autodetection is still a minor nuisance.


 It can be set in the command line, something we should clarify. Actually
 let me work on a PR real quick. -- Andrei


Destroy: https://github.com/D-Programming-Language/dlang.org/pull/1012

Andrei
Jun 08 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/15 2:53 PM, Vladimir Panteleev wrote:

 On Monday, 8 June 2015 at 19:51:28 UTC, Andrei Alexandrescu wrote:

 Yah, you'd need to pass LATEST=2.067.1 in the command line. Then the
 build process won't need to connect to the internet to fetch the
 latest version.


 "LATEST" detection must go. It makes bisecting impossible.

 In general, if we want reproducible builds (and this is required for
 reliable bisection of regressions), we can't rely on non-immutable
 online resources. Fetching / building stable compiler versions is OK,
 but detecting which version is latest right now is not.


Well I trust you can come up with a well-engineered improvement. -- Andrei
Jun 08 2015
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 8 June 2015 at 22:25:41 UTC, Andrei Alexandrescu wrote:

 Well I trust you can come up with a well-engineered 
 improvement. -- Andrei


1. We can just put it in the makefile (or elsewhere in the 
repository). New releases need website updates anyway (front 
page, changelog...).

Currently, the front page hard-codes the current version number - 
it could use LATEST instead.

2. DMD has a "VERSION" file. It could also have a 
"STABLE_VERSION" file (which is equal to the "VERSION" file only 
for release builds).
Jun 08 2015
prev sibling next sibling parent Joseph Rushton Wakeling via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 08/06/15 19:06, Joseph Rushton Wakeling via Digitalmars-d wrote:

 Well, just for one, I tried building phobos docs a couple of days ago while
 disconnected from the internet; the whole thing failed because the clones of
 stable release repos couldn't be done.

 No doubt I'm missing a trick, but it didn't seem very obvious, on browsing the
 makefile, what would get me the simple case of building the development phobos
 docs using my development dmd, without needing to worry about any of the other
 options out there.


Well, whaddayaknow, make -f posix.mak clean; make -f posix.mak
phobos-prerelease 
just worked.  I'd swear I was trying to build the phobos-prerelease target 
before, but maybe forgetting the 'clean' first had some kind of weird effect on 
how build targets interacted.
Jun 08 2015
prev sibling parent reply Joseph Rushton Wakeling via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 08/06/15 21:53, Joseph Rushton Wakeling via Digitalmars-d wrote:

 On 08/06/15 19:06, Joseph Rushton Wakeling via Digitalmars-d wrote:

 Well, just for one, I tried building phobos docs a couple of days ago while
 disconnected from the internet; the whole thing failed because the clones of
 stable release repos couldn't be done.

 No doubt I'm missing a trick, but it didn't seem very obvious, on browsing the
 makefile, what would get me the simple case of building the development phobos
 docs using my development dmd, without needing to worry about any of the other
 options out there.


 Well, whaddayaknow, make -f posix.mak clean; make -f posix.mak
phobos-prerelease
 just worked.  I'd swear I was trying to build the phobos-prerelease target
 before, but maybe forgetting the 'clean' first had some kind of weird effect on
 how build targets interacted.


Ho hum, spoken too soon.  I don't know what was going on, but I fear I was
using 
some weird checkout of dlang.org at the time I wrote the above.

After upgrading to latest master, I couldn't even issue 'make -f posix.mak 
clean' without generating a network-connectivity-related error:

-----------------
$ make -f posix.mak clean
fatal: unable to access 'https://github.com/D-Programming-Language/dmd/': Could 
not resolve host: github.com
LATEST= <-- place in the command line to skip network traffic.
posix.mak:69: *** Could not fetch latest version, place LATEST=2.xxx.y in the 
command line. Stop.
-----------------

Adding LATEST=2.067.1 to the command indeed allows clean to run, but then -- 
joy! -- I can no longer rebuild anything, because the cached DMD 2.067 is 
deleted and I need internet connectivity to get it back.  It's also a major 
"WTF??!!" that 'make clean' should need network connectivity in any
circumstances.

The bottom line of this for me is that I have dmd, druntime, phobos and tools 
repos all in the "standard" locations relative to my dlang.org repo; all of 
those are built (or I can build them).  It's therefore really baffling to me 
that I should need more than that in order to be able to issue 'make 
phobos-prerelease' and see what my updated docs look like.  Force me to add 
LATEST=dev or something like that to the command if necessary, but _don't_
force 
me to either download or already have cached a separate copy of the compiler.

I cannot speak to how infuriating it was to be on a plane with hours to spare
to 
work on phobos material, only to find myself flying blind unable to validate my 
documentation work, because the doc build system assumes network connectivity 
and has no obvious fallback or workaround.

Sorry if the above sounds abrasive, but take it as a measure of how much I was 
enjoying working on phobos again :-)
Jun 13 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/15 1:10 AM, Joseph Rushton Wakeling via Digitalmars-d wrote:

 Adding LATEST=2.067.1 to the command indeed allows clean to run, but
  then -- joy! -- I can no longer rebuild anything, because the cached
 DMD 2.067 is deleted and I need internet connectivity to get it
 back.


I don't think make clean deletes dmd2.067.1. It does instruct you to do
so if you so want.


 It's also a major "WTF??!!" that 'make clean' should need network
 connectivity in any circumstances.


Yah, supporting 100% disconnected builds was a corner case that's not
properly supported yet. It's just difficult in make to set a variable
efficiently only for some targets.


 The bottom line of this for me is that I have dmd, druntime, phobos
 and tools repos all in the "standard" locations relative to my
 dlang.org repo; all of those are built (or I can build them).  It's
 therefore really baffling to me that I should need more than that in
 order to be able to issue 'make phobos-prerelease' and see what my
 updated docs look like.  Force me to add LATEST=dev or something like
 that to the command if necessary, but _don't_ force me to either
 download or already have cached a separate copy of the compiler.

 I cannot speak to how infuriating it was to be on a plane with hours
 to spare to work on phobos material, only to find myself flying blind
 unable to validate my documentation work, because the doc build
 system assumes network connectivity and has no obvious fallback or
 workaround.

 Sorry if the above sounds abrasive, but take it as a measure of how
 much I was enjoying working on phobos again :)


There's no surprise that things we didn't focus much effort don't work 
well. That said, I'm not sure what's happening there - things should 
have worked; phobos-prerelease doesn't require the legacy stuff. Should 
be reported as a bug. Perhaps it's dub that insists on a specific 
version. Could you please try "make -f posix.mak html" while 
disconnected and see if that goes through?


Andrei
Jun 13 2015
parent Joseph Rushton Wakeling via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13/06/15 17:30, Andrei Alexandrescu via Digitalmars-d wrote:

 On 6/13/15 1:10 AM, Joseph Rushton Wakeling via Digitalmars-d wrote:

 Adding LATEST=2.067.1 to the command indeed allows clean to run, but
  then -- joy! -- I can no longer rebuild anything, because the cached
 DMD 2.067 is deleted and I need internet connectivity to get it
 back.


 I don't think make clean deletes dmd2.067.1. It does instruct you to do
 so if you so want.


Just for clarity, I'm talking about the auto-downloaded executable, not the git 
clones of the stable version to ../dmd-2.067 etc.  In other words right after 
the make command is issued, this gets run:

$ make -j2 -f posix.mak LATEST=2.067.1 phobos-prerelease
mkdir -p .generated/
echo "LATEST=2.067.1" >.generated/2.067.1.ddoc
mkdir -p /tmp/.stable_dmd-2.067.1
TMPFILE=$(mktemp deleteme.XXXXXXXX) && curl -fsSL 
http://downloads.dlang.org/releases/2.x/2.067.1/dmd.2.067.1.linux.zip > 
${TMPFILE}.zip && \
		unzip -qd /tmp/.stable_dmd-2.067.1 ${TMPFILE}.zip && rm ${TMPFILE}.zip

Anyway, I believe you're right that this temporary download doesn't get cleaned 
up by "make clean" but I believe it can get lost through cleanup of the /tmp
dir 
without any user intent.

I'm just trying a general "make -f posix.mak" build (which will recreate the 
clones of stable dmd etc.) and will see if this allows avoiding the curl 
download of prebuilt dmd.


 There's no surprise that things we didn't focus much effort don't work well.
 That said, I'm not sure what's happening there - things should have worked;
 phobos-prerelease doesn't require the legacy stuff. Should be reported as a
bug.
 Perhaps it's dub that insists on a specific version. Could you please try "make
 -f posix.mak html" while disconnected and see if that goes through?


I'll give it a go.  Will report back on results.
Jun 15 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-06-08 15:34, Steven Schveighoffer wrote:


 If you just want to check that a file's content is going to show up, I use:

 dmd -D -o- -c std/somemod.d

 The output is not styled properly. But the output should have working
 links.


What about the custom ddoc macros?

-- 
/Jacob Carlborg
Jun 08 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/8/15 10:07 AM, Jacob Carlborg wrote:

 On 2015-06-08 15:34, Steven Schveighoffer wrote:


 If you just want to check that a file's content is going to show up, I
 use:

 dmd -D -o- -c std/somemod.d

 The output is not styled properly. But the output should have working
 links.


 What about the custom ddoc macros?


Hm... I guess you could include them. I don't even know how to do that.

If the docs don't turn out the way I expect, but I think it would with a 
full build, I usually just stick a warning on my PR that I didn't test it :P

-Steve
Jun 08 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/15 4:27 AM, John Colvin wrote:

 Is there any non-insane way of changing some documentation in a module
 and checking that any links/xrefs you may have added actually work?

 The dlang.org makefile appears to happily erase uncommitted changes in
 ../druntime and ../phobos. I'm really struggling for words to describe
 how infuriating that is. I ran make and it deleted my work in a separate
 folder. Not OK.


Bummer about that. I've never encountered this problem. What command(s) 
did you run? -- Andrei
Jun 08 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 8 June 2015 at 16:14:30 UTC, Andrei Alexandrescu wrote:

 On 6/8/15 4:27 AM, John Colvin wrote:

 Is there any non-insane way of changing some documentation in 
 a module
 and checking that any links/xrefs you may have added actually 
 work?

 The dlang.org makefile appears to happily erase uncommitted 
 changes in
 ../druntime and ../phobos. I'm really struggling for words to 
 describe
 how infuriating that is. I ran make and it deleted my work in 
 a separate
 folder. Not OK.


 Bummer about that. I've never encountered this problem. What 
 command(s) did you run? -- Andrei


with dir structure
.
|- dmd
|- druntime
|- phobos
|- dlang.org

clone phobos, edit
clone druntime, edit
clone dmd, compile
compile druntime,
compile phobos
clone dlang.org, make -f posix.mak

boom, my changes are gone
Jun 08 2015
parent "Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> writes:
On Monday, 8 June 2015 at 16:48:31 UTC, John Colvin wrote:

 with dir structure
 .
 |- dmd
 |- druntime
 |- phobos
 |- dlang.org

 clone phobos, edit
 clone druntime, edit
 clone dmd, compile
 compile druntime,
 compile phobos
 clone dlang.org, make -f posix.mak

 boom, my changes are gone


That shouldn't happen. If you get a repro, I'll look into that. 
-- Andrei
Jun 08 2015