Documentation Generator

The D programming language enables embedding both contracts and test code along side the actual code, which helps to keep them all consistent with each other. One thing lacking is the documentation, as ordinary comments are usually unsuitable for automated extraction and formatting into manual pages. Embedding the user documentation into the source code has important advantages, such as not having to write the documentation twice, and the likelihood of the documentation staying consistent with the code.

Some existing approaches to this are:

Doxygen which already has some support for D
Java's Javadoc, probably the most well-known
C#'s embedded XML
Other documentation tools

D's goals for embedded documentation are:

It looks good as embedded documentation, not just after it is extracted and processed.
It's easy and natural to write, i.e. minimal reliance on <tags> and other clumsy forms one would never see in a finished document.
It does not repeat information that the compiler already knows from parsing the code.
It doesn't rely on embedded HTML, as such will impede extraction and formatting for other purposes.
It's based on existing D comment forms, so it is completely independent of parsers only interested in D code.
It should look and feel different from code, so it won't be visually confused with code.
It should be possible for the user to use Doxygen or other documentation extractor if desired.

Specification

The specification for the form of embedded documentation comments only specifies how information is to be presented to the compiler. It is implementation-defined how that information is used and the form of the final presentation. Whether the final presentation form is an HTML web page, a man page, a PDF file, etc. is not specified as part of the D Programming Language.

Phases of Processing

Embedded documentation comments are processed in a series of phases:

Lexical - documentation comments are identified and attached to tokens.
Parsing - documentation comments are associated with specific declarations and combined.
Sections - each documentation comment is divided up into a sequence of sections.
Special sections are processed.
Highlighting of non-special sections is done.
All sections for the module are combined.
Macro text substitution is performed to produce the final result.

Lexical

Embedded documentation comments are one of the following forms:

/** ... */ The two *'s after the opening /
/++ ... +/ The two +'s after the opening /
/// The three slashes

The following are all embedded documentation comments:

/// This is a one line documentation comment.

/** So is this. */

/++ And this. +/

/**
   This is a brief documentation comment.
 */

/**
 * The leading * on this line is not part of the documentation comment.
 */

/*********************************
   The extra *'s immediately following the /** are not
   part of the documentation comment.
 */

/++
   This is a brief documentation comment.
 +/

/++
 + The leading + on this line is not part of the documentation comment.
 +/

/+++++++++++++++++++++++++++++++++
   The extra +'s immediately following the / ++ are not
   part of the documentation comment.
 +/

/**************** Closing *'s are not part *****************/

The extra *'s and +'s on the comment opening, closing and left margin are ignored and are not part of the embedded documentation. Comments not following one of those forms are not documentation comments.

Parsing

Each documentation comment is associated with a declaration. If the documentation comment is on a line by itself or with only whitespace to the left, it refers to the next declaration. Multiple documentation comments applying to the same declaration are concatenated. Documentation comments not associated with a declaration are ignored. Documentation comments preceding the ModuleDeclaration apply to the entire module. If the documentation comment appears on the same line to the right of a declaration, it applies to that.

If a documentation comment for a declaration consists only of the identifier ditto then the documentation comment for the previous declaration at the same declaration scope is applied to this declaration as well.

If there is no documentation comment for a declaration, that declaration may not appear in the output. To ensure it does appear in the output, put an empty declaration comment for it.

int a;  /// documentation for a; b has no documentation
int b;

/** documentation for c and d */
/** more documentation for c and d */
int c;
/** ditto */
int d;

/** documentation for e and f */ int e;
int f;	/// ditto

/** documentation for g */
int g; /// more documentation for g

/// documentation for C and D
class C {
  int x; /// documentation for C.x

  /** documentation for C.y and C.z */
  int y;
  int z; /// ditto
}

/// ditto
class D { }

Sections

The document comment is a series of Sections. A Section is a name that is the first non-blank character on a line immediately followed by a ':'. This name forms the section name. The section name is not case sensitive.

Summary

The first section is the Summary, and does not have a section name. It is first paragraph, up to a blank line or a section name. While the summary can be any length, try to keep it to one line. The Summary section is optional.

Description

The next unnamed section is the Description. It consists of all the paragraphs following the Summary until a section name is encountered or the end of the comment.

While the Description section is optional, there cannot be a Description without a Summary section.

/***********************************
 * Brief summary of what
 * myfunc does, forming the summary section.
 *
 * First paragraph of synopsis description.
 *
 * Second paragraph of
 * synopsis description.
 */

void myfunc() { }

Named sections follow the Summary and Description unnamed sections.

Standard Sections

For consistency and predictability, there are several standard sections. None of these are required to be present.

Authors:

Lists the author(s) of the declaration.

/**
 * Authors: Melvin D. Nerd, melvin@mailinator.com
 */

Bugs:

Lists any known bugs.

/**
 * Bugs: Doesn't work for negative values.
 */

Date:

Specifies the date of the current revision. The date should be in a form parseable by std.date.

/**
 * Date: March 14, 2003
 */

Deprecated:

Provides an explanation for and corrective action to take if the associated declaration is marked as deprecated.

/**
 * Deprecated: superseded by function bar().
 */

deprecated void foo() { ... }

Examples:

Any usage examples

/**
 * Examples:
 * --------------------
 * writefln("3"); // writes '3' to stdout
 * --------------------
 */

History:

Revision history.

/**
 * History:
 *	V1 is initial version
 *
 *	V2 added feature X
 */

License:

Any license information for copyrighted code.

/**
 * License: use freely for any purpose
 */

void bar() { ... }

Returns:

Explains the return value of the function. If the function returns void, don't redundantly document it.

/**
 * Read the file.
 * Returns: The contents of the file.
 */

void[] readFile(char[] filename) { ... }

See_Also:

List of other symbols and URL's to related items.

/**
 * See_Also:
 *    foo, bar, http://www.digitalmars.com/d/phobos/index.html
 */

Standards:

If this declaration is compliant with any particular standard, the description of it goes here.

/**
 * Standards: Conforms to DSPEC-1234
 */

Throws:

Lists exceptions thrown and under what circumstances they are thrown.

/**
 * Write the file.
 * Throws: WriteException on failure.
 */

void writeFile(char[] filename) { ... }

Version:

Specifies the current version of the declaration.

/**
 * Version: 1.6a
 */

Special Sections

Some sections have specialized meanings and syntax.

Copyright:

This contains the copyright notice. The macro COPYRIGHT is set to the contents of the section when it documents the module declaration. The copyright section only gets this special treatment when it is for the module declaration.

/** Copyright: Public Domain */

module foo;

Params:

Function parameters can be documented by listing them in a params section. Each line that starts with an identifier followed by an '=' starts a new parameter description. A description can span multiple lines.

/***********************************
 * foo does this.
 * Params:
 *	x =	is for this
 *		and not for that
 *	y =	is for that
 */

void foo(int x, int y)
{
}

Macros:

The macros section follows the same syntax as the Params: section. It's a series of NAME=value pairs. The NAME is the macro name, and value is the replacement text.

/**
 * Macros:
 *	FOO =	now is the time for
 *		all good men
 *	BAR =	bar
 *	MAGENTA =   &lt;font color=magenta&gt;&lt;/font&gt;
 */

Highlighting

Embedded Comments

The documentation comments can themselves be commented using the $(DDOC_COMMENT comment text) syntax. These comments do not nest.

Embedded Code

D code can be embedded using lines beginning with at least three hyphens (ignoring whitespace) to delineate the code section:

/++
 + Our function.
 +
 + Example:
 + ---
 + import std.stdio;
 +
 + void foo()
 + {
 +     writefln("foo!");  /* print the string */
 + }
 + ---
 +/

Note that the documentation comment uses the /++ ... +/ form so that /* ... */ can be used inside the code section.

Embedded HTML

HTML can be embedded into the documentation comments, and it will be passed through to the HTML output unchanged. However, since it is not necessarily true that HTML will be the desired output format of the embedded documentation comment extractor, it is best to avoid using it where practical.

/**
 * Example of embedded HTML:
 *
 * <ol>
 *   <li><a href="http://www.digitalmars.com">Digital Mars</a></li>
 *   <li><a href="http://www.classicempire.com">Empire</a></li>
 * </ol>
 */

Emphasis

Identifiers in documentation comments that are function parameters or are names that are in scope at the associated declaration are emphasized in the output. This emphasis can take the form of italics, boldface, a hyperlink, etc. How it is emphasized depends on what it is - a function parameter, type, D keyword, etc. To prevent unintended emphasis of an identifier, it can be preceded by an underscore (_). The underscore will be stripped from the output.

Character Entities

Some characters have special meaning to the documentation processor, to avoid confusion it can be best to replace them with their corresponding character entities:

Characters and Entities
Character	Entity
<	<
>	>
&	&

It is not necessary to do this inside a code section, or if the special character is not immediately followed by a # or a letter.

No Documentation

No documentation is generated for the following constructs, even if they have a documentation comment:

Invariants
Postblits
Destructors
Static constructors and static destructors
Class info, type info, and module info

Macros

The documentation comment processor includes a simple macro text preprocessor. When a $(NAME) appears in section text it is replaced with NAME's corresponding replacement text. The replacement text is then recursively scanned for more macros. If a macro is recursively encountered, with no argument or with the same argument text as the enclosing macro, it is replaced with no text. Macro invocations that cut across replacement text boundaries are not expanded. If the macro name is undefined, the replacement text has no characters in it. If a $(NAME) is desired to exist in the output without being macro expanded, the $ should be replaced with $.

Macros can have arguments. Any text from the end of the identifier to the closing ‘)’ is the $0 argument. A $0 in the replacement text is replaced with the argument text. If there are commas in the argument text, $1 will represent the argument text up to the first comma, $2 from the first comma to the second comma, etc., up to $9. $+ represents the text from the first comma to the closing ‘)’. The argument text can contain nested parentheses, "" or '' strings, comments, or tags. If stray, unnested parentheses are used, they can be replaced with the entity ( for ( and ) for ).

Macro definitions come from the following sources, in the specified order:

Predefined macros.
Definitions from file specified by sc.ini's or dmd.conf DDOCFILE setting.
Definitions from *.ddoc files specified on the command line.
Runtime definitions generated by Ddoc.
Definitions from any Macros: sections.

Macro redefinitions replace previous definitions of the same name. This means that the sequence of macro definitions from the various sources forms a hierarchy.

Macro names beginning with "D_" and "DDOC_" are reserved.

Predefined Macros

These are hardwired into Ddoc, and represent the minimal definitions needed by Ddoc to format and highlight the presentation. The definitions are for simple HTML.

B =     <b>$0</b>
I =     <i>$0</i>
U =     <u>$0</u>
P =     <p>$0</p>
DL =    <dl>$0</dl>
DT =    <dt>$0</dt>
DD =    <dd>$0</dd>
TABLE = <table>$0</table>
TR =    <tr>$0</tr>
TH =    <th>$0</th>
TD =    <td>$0</td>
OL =    <ol>$0</ol>
UL =    <ul>$0</ul>
LI =    <li>$0</li>
BIG =   <big>$0</big>
SMALL = <small>$0</small>
BR =    <br>
LINK =  <a href="$0">$0</a>
LINK2 = <a href="$1">$+</a>
LPAREN= (
RPAREN= )

RED =   <font color=red>$0</font>
BLUE =  <font color=blue>$0</font>
GREEN = <font color=green>$0</font>
YELLOW =<font color=yellow>$0</font>
BLACK = <font color=black>$0</font>
WHITE = <font color=white>$0</font>

D_CODE = <pre class="d_code">$0</pre>
D_COMMENT = $(GREEN $0)
D_STRING  = $(RED $0)
D_KEYWORD = $(BLUE $0)
D_PSYMBOL = $(U $0)
D_PARAM   = $(I $0)

DDOC = <html><head>
       <META http-equiv="content-type" content="text/html; charset=utf-8">
       <title>$(TITLE)</title>
       </head><body>
       <h1>$(TITLE)</h1>
       $(BODY)
       </body></html>

DDOC_COMMENT   = <!-- $0 -->
DDOC_DECL      = $(DT $(BIG $0))
DDOC_DECL_DD   = $(DD $0)
DDOC_DITTO     = $(BR)$0
DDOC_SECTIONS  = $0
DDOC_SUMMARY   = $0$(BR)$(BR)
DDOC_DESCRIPTION = $0$(BR)$(BR)
DDOC_AUTHORS   = $(B Authors:)$(BR)
		$0$(BR)$(BR)
DDOC_BUGS      = $(RED BUGS:)$(BR)
		$0$(BR)$(BR)
DDOC_COPYRIGHT = $(B Copyright:)$(BR)
		$0$(BR)$(BR)
DDOC_DATE      = $(B Date:)$(BR)
		$0$(BR)$(BR)
DDOC_DEPRECATED = $(RED Deprecated:)$(BR)
		$0$(BR)$(BR)
DDOC_EXAMPLES  = $(B Examples:)$(BR)
		$0$(BR)$(BR)
DDOC_HISTORY   = $(B History:)$(BR)
		$0$(BR)$(BR)
DDOC_LICENSE   = $(B License:)$(BR)
		$0$(BR)$(BR)
DDOC_RETURNS   = $(B Returns:)$(BR)
		$0$(BR)$(BR)
DDOC_SEE_ALSO  = $(B See Also:)$(BR)
		$0$(BR)$(BR)
DDOC_STANDARDS = $(B Standards:)$(BR)
		$0$(BR)$(BR)
DDOC_THROWS    = $(B Throws:)$(BR)
		$0$(BR)$(BR)
DDOC_VERSION   = $(B Version:)$(BR)
		$0$(BR)$(BR)
DDOC_SECTION_H = $(B $0)$(BR)$(BR)
DDOC_SECTION   = $0$(BR)$(BR)
DDOC_MEMBERS   = $(DL $0)
DDOC_MODULE_MEMBERS   = $(DDOC_MEMBERS $0)
DDOC_CLASS_MEMBERS    = $(DDOC_MEMBERS $0)
DDOC_STRUCT_MEMBERS   = $(DDOC_MEMBERS $0)
DDOC_ENUM_MEMBERS     = $(DDOC_MEMBERS $0)
DDOC_TEMPLATE_MEMBERS = $(DDOC_MEMBERS $0)
DDOC_PARAMS    = $(B Params:)$(BR)\n$(TABLE $0)$(BR)
DDOC_PARAM_ROW = $(TR $0)
DDOC_PARAM_ID  = $(TD $0)
DDOC_PARAM_DESC  = $(TD $0)
DDOC_BLANKLINE = $(BR)$(BR)

DDOC_ANCHOR  = <a name="$1"></a>
DDOC_PSYMBOL = $(U $0)
DDOC_KEYWORD = $(B $0)
DDOC_PARAM   = $(I $0)

Ddoc does not generate HTML code. It formats into the basic formatting macros, which (in their predefined form) are then expanded into HTML. If output other than HTML is desired, then these macros need to be redefined.

Basic Formatting Macros
B	boldface the argument
I	italicize the argument
U	underline the argument
P	argument is a paragraph
DL	argument is a definition list
DT	argument is a definition in a definition list
DD	argument is a description of a definition
TABLE	argument is a table
TR	argument is a row in a table
TH	argument is a header entry in a row
TD	argument is a data entry in a row
OL	argument is an ordered list
UL	argument is an unordered list
LI	argument is an item in a list
BIG	argument is one font size bigger
SMALL	argument is one font size smaller
BR	start new line
LINK	generate clickable link on argument
LINK2	generate clickable link, first arg is address
RED	argument is set to be red
BLUE	argument is set to be blue
GREEN	argument is set to be green
YELLOW	argument is set to be yellow
BLACK	argument is set to be black
WHITE	argument is set to be white
D_CODE	argument is D code
DDOC	overall template for output

DDOC is special in that it specifies the boilerplate into which the entire generated text is inserted (represented by the Ddoc generated macro BODY). For example, in order to use a style sheet, DDOC would be redefined as:

DDOC =	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
	<html><head>
	<META http-equiv="content-type" content="text/html; charset=utf-8">
	<title>$(TITLE)</title>
	<link rel="stylesheet" type="text/css" href="style.css">
	</head><body>
	<h1>$(TITLE)</h1>
	$(BODY)
	</body></html>

DDOC_COMMENT is used to insert comments into the output file.

Highlighting of D code is performed by the following macros:

D Code Formatting Macros
D_COMMENT	Highlighting of comments
D_STRING	Highlighting of string literals
D_KEYWORD	Highlighting of D keywords
D_PSYMBOL	Highlighting of current declaration name
D_PARAM	Highlighting of current function declaration parameters

The highlighting macros start with DDOC_. They control the formatting of individual parts of the presentation.

Ddoc Section Formatting Macros
DDOC_DECL	Highlighting of the declaration.
DDOC_DECL_DD	Highlighting of the description of a declaration.
DDOC_DITTO	Highlighting of ditto declarations.
DDOC_SECTIONS	Highlighting of all the sections.
DDOC_SUMMARY	Highlighting of the summary section.
DDOC_DESCRIPTION	Highlighting of the description section.
DDOC_AUTHORS .. DDOC_VERSION	Highlighting of the corresponding standard section.
DDOC_SECTION_H	Highlighting of the section name of a non-standard section.
DDOC_SECTION	Highlighting of the contents of a non-standard section.
DDOC_MEMBERS	Default highlighting of all the members of a class, struct, etc.
DDOC_MODULE_MEMBERS	Highlighting of all the members of a module.
DDOC_CLASS_MEMBERS	Highlighting of all the members of a class.
DDOC_STRUCT_MEMBERS	Highlighting of all the members of a struct.
DDOC_ENUM_MEMBERS	Highlighting of all the members of an enum.
DDOC_TEMPLATE_MEMBERS	Highlighting of all the members of a template.
DDOC_PARAMS	Highlighting of a function parameter section.
DDOC_PARAM_ROW	Highlighting of a name=value function parameter.
DDOC_PARAM_ID	Highlighting of the parameter name.
DDOC_PARAM_DESC	Highlighting of the parameter value.
DDOC_ANCHOR	Expands to a named anchor used for hyperlinking to a particular declaration section. Argument $1 expands to the qualified declaration name.
DDOC_PSYMBOL	Highlighting of declaration name to which a particular section is referring.
DDOC_KEYWORD	Highlighting of D keywords.
DDOC_PARAM	Highlighting of function parameters.
DDOC_BLANKLINE	Inserts a blank line.

For example, one could redefine DDOC_SUMMARY:

DDOC_SUMMARY = $(GREEN $0)

And all the summary sections will now be green.

Macro Definitions from `sc.ini`'s DDOCFILE

A text file of macro definitions can be created, and specified in sc.ini:

DDOCFILE=myproject.ddoc

Macro Definitions from .ddoc Files on the Command Line

File names on the DMD command line with the extension .ddoc are text files that are read and processed in order.

Macro Definitions Generated by Ddoc

Generated Macro Definitions
Macro Name	Contents
BODY	Set to the generated document text.
TITLE	Set to the module name.
DATETIME	Set to the current date and time.
YEAR	Set to the current year.
COPYRIGHT	Set to the contents of any Copyright: section that is part of the module comment.
DOCFILENAME	Set to the name of the generated output file.
SRCFILENAME	Set to the name of the source file the documentation is being generated from.

Using Ddoc for other Documentation

Ddoc is primarily designed for use in producing documentation from embedded comments. It can also, however, be used for processing other general documentation. The reason for doing this would be to take advantage of the macro capability of Ddoc and the D code syntax highlighting capability.

If the .d source file starts with the string "Ddoc" then it is treated as general purpose documentation, not as a D code source file. From immediately after the "Ddoc" string to the end of the file or any "Macros:" section forms the document. No automatic highlighting is done to that text, other than highlighting of D code embedded between lines delineated with --- lines. Only macro processing is done.

Much of the D documentation itself is generated this way, including this page. Such documentation is marked at the bottom as being generated by Ddoc.

References

CandyDoc is a very nice example of how one can customize the Ddoc results with macros and style sheets.

Books