digitalmars.D - Calculating file dependancies
- Brad Beveridge (7/7) Aug 10 2004 Hi all, is there a way to calculate what source files should be compiled
- Lars Ivar Igesund (14/22) Aug 10 2004 I've created a dependency checker. The last version is attached. There
- Brad Beveridge (4/31) Aug 10 2004 Thanks Lars, I'll give it a go.
- Lars Ivar Igesund (7/1185) Aug 13 2004 Ok, I've fixed all known (to me) niggles and bugs and uploaded it to
- Matthew (2/1187) Aug 13 2004
- Arcane Jill (20/26) Aug 10 2004 When I want to recompile my project, I click on "Tools/D Build (Debug)" ...
- Billy Zelsnack (5/13) Aug 10 2004 I just compile my entire project every time. It simplifies
Hi all, is there a way to calculate what source files should be compiled when one file has changed? At the moment I am recompiling my whole project, but that is a little bit of overkill. Can the DMD compiler generate dependancies that are suitable for putting in a makefile, or is there some tool out there? Cheers Brad
Aug 10 2004
I've created a dependency checker. The last version is attached. There are some small niggles left, I haven't audited my last changes very thouroughly yet. It seems to work and handles debug and version flags. Compile with whatever flags you want :) Using -debug when compiling will make it useless on stdout, however (write to file should still work (hopefully, haven't tested after writefln came)). When executing ddepcheck, use -m to have it output make compatible dependencies, otherwise it uses a 'proprietary' format. Also, it uses char all around the place, since using dchar leads to lots og conversion all around the place (phobos use char here and dchar there). If you use it, please report any bugs and / or robustness problems. At least it should compile :) Lars Ivar Igesund Brad Beveridge wrote:Hi all, is there a way to calculate what source files should be compiled when one file has changed? At the moment I am recompiling my whole project, but that is a little bit of overkill. Can the DMD compiler generate dependancies that are suitable for putting in a makefile, or is there some tool out there? Cheers Brad
Aug 10 2004
Brad Beveridge <brad.beveridge somewhere.com> Thanks Lars, I'll give it a go. Cheers Brad Lars Ivar Igesund wrote:I've created a dependency checker. The last version is attached. There are some small niggles left, I haven't audited my last changes very thouroughly yet. It seems to work and handles debug and version flags. Compile with whatever flags you want :) Using -debug when compiling will make it useless on stdout, however (write to file should still work (hopefully, haven't tested after writefln came)). When executing ddepcheck, use -m to have it output make compatible dependencies, otherwise it uses a 'proprietary' format. Also, it uses char all around the place, since using dchar leads to lots og conversion all around the place (phobos use char here and dchar there). If you use it, please report any bugs and / or robustness problems. At least it should compile :) Lars Ivar Igesund Brad Beveridge wrote:Hi all, is there a way to calculate what source files should be compiled when one file has changed? At the moment I am recompiling my whole project, but that is a little bit of overkill. Can the DMD compiler generate dependancies that are suitable for putting in a makefile, or is there some tool out there? Cheers Brad
Aug 10 2004
Ok, I've fixed all known (to me) niggles and bugs and uploaded it to http://www.igesund.net/larsivar/ddepcheck.d, calling it version 1.0.0 (wohoo!). Lars Ivar Igesund PS I've noticed that there are some downloads of ddepcheck, but I've received only one comment ever (since end of 2003). Lars Ivar Igesund wrote:I've created a dependency checker. The last version is attached. There are some small niggles left, I haven't audited my last changes very thouroughly yet. It seems to work and handles debug and version flags. Compile with whatever flags you want :) Using -debug when compiling will make it useless on stdout, however (write to file should still work (hopefully, haven't tested after writefln came)). When executing ddepcheck, use -m to have it output make compatible dependencies, otherwise it uses a 'proprietary' format. Also, it uses char all around the place, since using dchar leads to lots og conversion all around the place (phobos use char here and dchar there). If you use it, please report any bugs and / or robustness problems. At least it should compile :) Lars Ivar Igesund Brad Beveridge wrote:Hi all, is there a way to calculate what source files should be compiled when one file has changed? At the moment I am recompiling my whole project, but that is a little bit of overkill. Can the DMD compiler generate dependancies that are suitable for putting in a makefile, or is there some tool out there? Cheers Brad------------------------------------------------------------------------ /* * ddepcheck.d * Version 0.9.9 * Last modified 1st of August 2004 * * Dependency walker for D source files. * Copyright 2003-2004 Lars Ivar Igesund <larsivar'at'igesund.net> * * Permission to use, copy, modify, distribute and sell this software * and its documentation for any purpose is hereby granted without fee, * provided that the above copyright notice appear in all copies and * that both that copyright notice and this permission notice appear * in supporting documentation. It is provided "as is" without express * or implied warranty. */ /* * BUGS: * - (local) dll.d leads to a crash */ /* * TODO: * - Fix importing: blocks of private imports, list of imports * - Audit code, some of it is hairy as hell. * - Exchange ctype with utype when it shows up * - Document better, especially code * - Long term; With multiple input files, cache import results. */ import std.conv; import std.ctype; import std.file; import std.path; import std.stream; import std.string; import std.c.stdio; import std.stdio; import std.utf; alias std.ctype.isdigit isdigit; version (Win32) { import std.c.windows.windows; } bit helpcalled = false; bit versioncalled = false; bit write = false; bit makesyntax = false; bit comment = false; bit runtime = false; bit checkprivate = false; bit fulldebug = false; int curopt = 1; int numpaths = 0; int allpaths = 10; int depthlimit = -1; int depth = 0; int numdeps = 0; int alldeps = 10; int firstfile = 1; char [][] depmods; char [][] depfiles; char [][] depdepths; char [][] paths; char [][] argslist; char [][] dbgidentsstr; char [][] veridentsstr; int dbgidentsnum; int veridentsnum; /* * Add version identifier strings to the list used by ddepcheck based on what * is defined by the compiler. */ static this() { version (DigitalMars) { veridentsstr ~= "DigitalMars"; } version (X86) { veridentsstr ~= "X86"; } version (AMD64) { veridentsstr ~= "AMD64"; } version (Windows) { veridentsstr ~= "Windows"; } version (Win32) { veridentsstr ~= "Win32"; } version (Win64) { veridentsstr ~= "Win64"; } version (linux) { veridentsstr ~= "linux"; } version (LittleEndian) { veridentsstr ~= "LittleEndian"; } version (BigEndian) { veridentsstr ~= "BigEndian"; } version (D_InlineAsm) { veridentsstr ~= "D_InlineAsm"; } } /* * Function called when the "help" option is specified. */ void optHelp() { optVersion(); if (!helpcalled) { writefln(); writefln("Syntax: ddepcheck [options] [src ...]"); writefln(" -I[path] Add a path to be searched."); writefln(" -h/--help Prints this help table."); writefln(" -d=[num]/--depth=[num] Limit the depth searched"); writefln(" for dependencies."); writefln(" -m/--make-syntax Print the dependencies using the make"); writefln(" syntax. \"objectfile : src deps\""); writefln(" -V/--version Prints the version."); writefln(" -w/--writetofile Prints the dependencies to the file"); writefln(" 'depfile' instead of to the console."); writefln(" -l/--checkruntimelib Tries to add runtimelib dependencies."); writefln(" Note that the path must be added "); writefln(" explicitly."); writefln(" -debug=[ident/num] Uses the same switch as the compiler"); writefln(" to remove/include debug codeblocks."); writefln(" -version=[ident/num] Uses the same switch as the compiler"); writefln(" to remove/include version codeblocks."); } helpcalled = true; } /* * Function called when the "checkruntimelib" option is specified. */ void optCheckRuntime() { runtime = true; } /* * Function called when the "checkprivate" option is specified. */ void optCheckPrivate() { checkprivate = true; } /* * Function called if the "version" option is used. */ void optVersion() { if (!versioncalled) { writefln("ddepcheck"); writefln("Dependency walker for D source files."); writefln("Version 0.9.9 Copyright Lars Ivar Igesund 2003 - 2004"); } versioncalled = true; } /* * Function called if the "writetofile" option is used. */ void optWrite() { write = true; } /* * Function called if the "depth" option is used to decide the maximum * depth for recursion. */ void optDepth(int arg, bit doubledash) { if (doubledash) { depthlimit = cast(int)atoi(argslist[arg][8..argslist[arg].length]); } else { depthlimit = cast(int)atoi(argslist[arg][3..argslist[arg].length]); } } /* * Function called for all the "-I" import options used. */ void optImport(int arg) { if (argslist[arg].length == 2) { return; } addPath(argslist[arg][2..argslist[arg].length]); } /* * Function called for all the "-debug" options used. */ void optDebug(int arg) { if (argslist[arg].length == 6) { fulldebug = true; } else { char [] ident = argslist[arg][7..argslist[arg].length]; bit num = isNumber(ident); if (num) { dbgidentsnum = toInt(ident); } else { dbgidentsstr ~= ident; } } } /* * Function called for all the "-version" options used. */ void optVersion(int arg) { if (argslist[arg].length == 2) { return; } char [] ident = argslist[arg][9..argslist[arg].length]; bit num = isNumber(ident); if (num) { veridentsnum = toInt(ident); } else { veridentsstr ~= ident; } } /* * Function called when the "-m"/"--make-syntax" option is used. */ void optMakeSyntax() { makesyntax = true; } /* * Function that checks if an argument is an option. */ bit checkOption(int arg) { switch (argslist[arg]) { case "--version": case "-V": optVersion(); return true; break; case "--help": case "-h": optHelp(); return true; break; case "--writetofile": case "-w": optWrite(); return true; break; case "--make-syntax": case "-m": optMakeSyntax(); return true; break; case "--checkruntimelib": case "-l": optCheckRuntime(); return true; break; case "--checkprivate": case "-p": optCheckPrivate(); return true; break; default: break; } if (cmp(argslist[arg][0..3], "-d=") == 0) { optDepth(arg, false); return true; } else if (argslist[arg].length > 8 && cmp(argslist[arg][0..8], "--depth=") == 0) { optDepth(arg, true); return true; } if (cmp(argslist[arg][0..2], "-I") == 0) { optImport(arg); return true; } if (cmp(argslist[arg][0..6], "-debug") == 0) { optDebug(arg); return true; } if (cmp(argslist[arg][0..8], "-version") == 0) { optVersion(arg); return true; } return false; } /* * Function that starts the dependency walking for base source files. */ void depBase(int arg) { addNewBaseFile(argslist[arg]); depWalk(argslist[arg], "", true); } /* * Function that do the main dependency walking recursively. If it find * files that has been walked before, it skips it to avoid infinite * cyclic dependencies. */ void depWalk(char [] file,char [] mod, bit base) { debug writefln("Filename: ", file); char [] filepath = file; char [] src; int pathnum = 0; int [] importpos; importpos.length = 0; int nextimport = -1; char [][] verspecstrings; char [][] dbgspecstrings; while (pathnum < numpaths && !fileExist(filepath)) { filepath = std.path.join(paths[pathnum], file); pathnum++; } try { src = (new File(filepath)).toString(); } catch (Error e) { return; } if (!base) { if (!addDep(filepath, mod, depth)) { return; } } stripComment(src); bit useVersion(int pos) { int i = pos + 7; int j; while (src[i] != '(') { i++; } j = i + 1; while (src[j] != ')') { j++; } char [] ident = src[i + 1..j]; if (ident.length == 0) { return true; } bit num = true; foreach (char c; ident) { if (!isdigit(c)) { num = false; break; } } if (num) { int identnum = toInt(ident); if (veridentsnum >= identnum) { return true; } } else { foreach (char [] id; veridentsstr) { if (id == ident) { return true; } else if (id == "none") { return false; } } } return false; } void search(int a, int z) { int dbgfound = -1; int verfound = -1; int impfound = -1; int dbgend = -1; int verend = -1; int i = a; bit useDebug(int pos) { int i = pos + 5; int j; if (i >= src.length) { return false; } while (i < (src.length - 1) && isspace(src[i])) { i++; } if (src[i] == '(') { j = i + 1; while (j < (src.length - 1) && src[j] != ')') { j++; } char [] ident = src[i + 1..j]; bit num = true; foreach (char c; ident) { if (!isdigit(c)) { num = false; break; } } if (num) { int identnum = toInt(ident); if (dbgidentsnum <= identnum) { return true; } } else { foreach (char [] id; dbgidentsstr ~ dbgspecstrings) { if (id == ident) { return true; } } } if (fulldebug) { return false; } } else { if (fulldebug) { return true; } } return false; } int findDebugAndVersionEnd(int pos, bit ver, bit other) { int findSpecEnd(int pos) { if (other) { return 0; } int j = pos; if (pos >= src.length) { return 0; } while (j < (src.length - 1) && isspace(src[j])) { j++; } if (src[j] == '=') { int startidx = -1; j++; while (j < (src.length - 1) && src[j] != ';') { if (startidx == -1 && isalnum(src[j])) { startidx = j; } j++; } char [] ident = src[startidx..j]; if (ident.length > 0) { bit num = isNumber(ident); if (ver) { if (num) { try { veridentsnum = toInt(ident); } catch (Exception e) { // Catching top exception here due to strange naming // conventions in Phobos. larsivi 20040724 return j; } } else { verspecstrings ~= ident; } } else { if (num) { try { dbgidentsnum = toInt(ident); } catch (Exception e) { // Catching top exception here due to strange naming // conventions in Phobos. larsivi 20040724 return j; } } else { dbgspecstrings ~= ident; } } } return j; } return 0; } int findBlockEnd(int pos) { debug { if (ver) { writefln("- Finding version block end after pos ", pos); } else { writefln("- Finding debug block end after pos ", pos); } } int j = pos; if (pos >= src.length) { return 0; } while (j < (src.length - 1) && isspace(src[j])) { j++; } int findEnd() { while (isspace(src[j])) { j++; } if (src[j] == '{') { j++; while (src[j] != '}') { j++; } return j; } else { return 0; } } if (src[j] != '(') { return findEnd(); } else { j++; while (src[j] != ')') { j++; } j++; } return findEnd(); } int findAttrEnd(int pos) { int j = pos; if (pos >= src.length) { return 0; } while (j < (src.length - 1) && isspace(src[j])) { j++; } if (src[j] == ':') { while (j < (src.length - 1) && src[j] != '}') { j++; } return j; } else { return 0; } } int findStmntEnd(int pos) { int j = pos; if (pos >= src.length) { return 0; } while (j < (src.length - 1) && src[j] != ';') { j++; } return j; } int i = pos; int res = 0; if (i >= src.length) { return -1; } if (other) { i += 4; } else if (ver) { i += 7; } else { i += 5; } if (cast(bit)(res = findSpecEnd(i))) { return res; } else if (cast(bit)(res = findBlockEnd(i))) { return res; } else if (cast(bit)(res = findAttrEnd(i))) { return res; } else if (cast(bit)(res = findStmntEnd(i))) { return res; } else { return -1; } } int findElse(int pos, bit ver) { int i = pos; while (isspace(src[i++])) { } if (src[i..i + 4] == "else") { i += 4; return findDebugAndVersionEnd(i, ver, true); } else { return -1; } } bit isKeyword(int startpos, int length) { if (startpos < 0) { return false; } if (startpos + length >= src.length) { return false; } if (startpos == 0 || isspace(src[startpos - 1]) || src[startpos] == ';') { if (((startpos + length) == src.length) || isspace(src[startpos + length]) || (src[startpos..startpos + length] == "debug" && src[startpos + length] == ':')) { return true; } } return false; } int checkDebugAndVersion(int pos, bit ver) { int elseend = -1; int end = findDebugAndVersionEnd(pos, ver, false); int found = -1; if (end == -1) { return pos + 1; } elseend = findElse(end, ver); if (ver ? useVersion(pos) : useDebug(pos)) { search(pos, end); if (elseend > -1) { return elseend + 1; } else { return end + 1; } } else if (elseend > -1) { search(end + 1, elseend); return elseend + 1; } return pos + 1; } int handleDebug(int pos) { return checkDebugAndVersion(pos, false); } int handleVersion(int pos) { return checkDebugAndVersion(pos, true); } int handleImport(int pos) { debug writefln("- Handling import at position ", pos); int endpos; int i = pos + 8; while (true) { if (src[i] == ',') { // handle list of imports possibly // FIXME: Implement break; } else if (src[i] == ';') { break; } i++; if (i == src.length) { return i; } } endpos = i; if (!base && !checkprivate) { if (pos > 7) { i = pos - 1; while (isspace(src[i]) && i > 7) { i--; } if (i >= 7 && src[i - 6..i + 1] == "private") { return endpos + 1; } } } char [] mod = strip(src[pos + 6..endpos]); if (!checkIfModule(mod)) { return endpos + 1; } if (!runtime) { if (checkPhobos(mod)) { return endpos + 1; } } char [] fp = createFilePath(mod); depth++; if (depthlimit == -1 || depth <= depthlimit) { depWalk(fp, mod, false); } depth--; return endpos + 1; } int delegate(int) handler; int best; do { handler = null; best = z + 1; uint belowz = z; debug writefln("- Searching for import from ", i, " to ", z); impfound = find(src[i..z], "import"); if (impfound > -1) { impfound += i; belowz = impfound; } debug writefln("- Searching for debug from ", i, " to ", belowz); dbgfound = find(src[i..belowz], "debug"); if (dbgfound > -1) { dbgfound += i; belowz = dbgfound < belowz ? dbgfound : belowz; } debug writefln("- Searching for version from ", i, " to ", belowz); verfound = find(src[i..belowz], "version"); if (verfound > -1) { verfound += i; } if (impfound > -1) { if (isKeyword(impfound + i, 6)) { best = impfound + i; handler = &handleImport; } } if (dbgfound > -1 && (dbgfound + i) < best) { if (isKeyword(dbgfound + i, 5)) { best = dbgfound + i; handler = &handleDebug; } } if (verfound > -1 && (verfound + i) < best) { if (isKeyword(verfound + i, 7)) { best = verfound + i; handler = &handleVersion; } } if (!(handler is null)) { i = handler(best); } else { break; } } while (i < z); } int pos = 0; int found = -1; search(0, src.length); dbgidentsstr ~= dbgspecstrings; veridentsstr ~= verspecstrings; } /* * Function that adds a dependency to the list when it is verified. * A path and the depth where it was found is also added. If the * dependency has been found before, the new depth is added to the * same entry. */ bit addDep(char [] filepath, char [] mod, int depth) { for (int i = 0; i < numdeps; i++) { if (depmods[i] == mod) { depdepths[i] ~= ","; depdepths[i] ~= toString(depth); return false; } } numdeps++; if (numdeps > alldeps) { alldeps *= 2; depmods.length = alldeps; depfiles.length = alldeps; depdepths.length = alldeps; } depmods[numdeps - 1] = mod; depfiles[numdeps - 1] = filepath; depdepths[numdeps - 1] = toString(depth); return true; } /* * Function that adds the filename of the current checked file to * the printout. */ void addNewBaseFile(char [] file) { numdeps += 3; if (numdeps > alldeps) { alldeps *= 2; depmods.length = alldeps; depfiles.length = alldeps; depdepths.length = alldeps; } depmods[numdeps-2] = file; depfiles[numdeps-3] = depfiles[numdeps-2] = depfiles[numdeps-1] = ""; depdepths[numdeps-3] = depdepths[numdeps-2] = depdepths[numdeps-1] = ""; } /*! * Function that takes a module name and creates a real path. */ char [] createFilePath(char [] mod) { char [] tempmod; tempmod = replace(mod, ".", sep); tempmod ~= ".d"; return tempmod; } /*! * Checks if the found dependency is a part of phobos in which case * it is ignored. */ bit checkPhobos(char [] mod) { if (mod == "Object") { return true; } else if (mod.length < 4) { return false; } else if (cmp(mod[0..4], "std.") == 0) { return true; } else { return false; } } /*! * Checks if the found module really can be a module since the parsing * is a bit hackish. */ bit checkIfModule(char [] mod) { if (!isalpha(mod[0])) { return false; } foreach (char c; mod[1..mod.length]) { if (!(isalnum(c) || c == '.')) { return false; } } return true; } /* * Adds a path given with "-I" to the path list. */ void addPath(char [] path) { numpaths++; if (numpaths > allpaths) { allpaths *= 2; paths.length = allpaths; } paths[numpaths - 1] = path; } /*! * Checks if a file exist. */ bit fileExist(char [] file) { version (Win32) { return (GetFileAttributesA(toStringz(file)) != 0xFFFFFFFF); } version (Linux) { // FIXME: I guess there is a better way to do this. larsivi 18042004 bit result = true; try { read(file); } catch (FileException e) { result = false; } return result; } } /*! * Checks if the string is a number. Returns true if it is. */ bit isNumber(char [] ident) { foreach (char c; ident) { if (!isdigit(c)) { return false; } } return true; } /*! * Convert a substring in src to whitespace. */ void toWhiteSpace(inout char [] src, int start, int end) in { assert (end > start); assert (start >= 0); assert (end <= src.length); } body { src[start..end] = ' '; } /*! * Strip off the comments. */ void stripComment(inout char[] src) { int len = src.length; int doubleslash = -1; int star = -1; int plus = 0; int plusstart = -1; for (int i = 0; i < len; i++) { if (doubleslash > -1) { if (src[i] == '\\') { if (src[i + 1] == 'n') { toWhiteSpace(src, doubleslash, i + 2); i++; doubleslash = -1; continue; } } continue; } else if (star > -1) { if (src[i] == '*') { if (src[i + 1] == '/') { toWhiteSpace(src, star, i + 2); i++; star = -1; continue; } } continue; } else if (plus) { if (src[i] == '+') { if (src[i + 1] == '/') { i++; plus--; if (!plus) { toWhiteSpace(src, plusstart, i + 1); plusstart = -1; } continue; } } } if (src[i] == '/') { i++; if (src[i] == '/') { doubleslash = i - 1; } else if (src[i] == '*') { star = i - 1; } else if (src[i] == '+') { if (!plus) { plusstart = i - 1; } plus++; } } } } /* * Writes dependencies to file. */ void printToFile() { File depfile = new File("depfile", FileMode.Out); for (int i = 0; i < numdeps; i++) { depfile.printf(depmods[i] ~ " " ~ depfiles[i] ~ " " ~ depdepths[i] ~ "\n"); } depfile.close(); } /* * Prints dependencies to stdout. */ void printToStdout() { // FIXME: Use writef(ln) // larsivi 20040715 if (makesyntax) { char [] objsuf = ""; version (Win32) { objsuf = ".obj"; } version (linux) { objsuf = ".o"; } writef(replace(argslist[firstfile], ".d", objsuf), " : "); writef(argslist[firstfile]); if (numdeps > 3) { writef(" "); for (int i = 3; i < numdeps; i++) { writef(depfiles[i], " "); } } } else { for (int i = 0; i < numdeps; i++) { writefln(depmods[i], " ", depfiles[i], " ", depdepths[i]); } } } int main(char[][] args) { if (args.length == 1) { optHelp(); return 0; } depmods.length = 10; depfiles.length = 10; depdepths.length = 10; paths.length = 10; argslist = args; while (curopt < args.length && checkOption(curopt)) { curopt++; } if (curopt < args.length) { firstfile = curopt; for (int i = curopt; i < args.length; i++) { // FIXME: Handle multiple input files better // larsivi 20040715 depBase(i); } } if (write) { printToFile(); } else { printToStdout(); } return 0; }
Aug 13 2004
FWIW, I am looking forward to using it, at some indeterminate time in the future. :) "Lars Ivar Igesund" <larsivar igesund.net> wrote in message news:cfiuhi$tr6$1 digitaldaemon.com...Ok, I've fixed all known (to me) niggles and bugs and uploaded it to http://www.igesund.net/larsivar/ddepcheck.d, calling it version 1.0.0 (wohoo!). Lars Ivar Igesund PS I've noticed that there are some downloads of ddepcheck, but I've received only one comment ever (since end of 2003). Lars Ivar Igesund wrote:I've created a dependency checker. The last version is attached. There are some small niggles left, I haven't audited my last changes very thouroughly yet. It seems to work and handles debug and version flags. Compile with whatever flags you want :) Using -debug when compiling will make it useless on stdout, however (write to file should still work (hopefully, haven't tested after writefln came)). When executing ddepcheck, use -m to have it output make compatible dependencies, otherwise it uses a 'proprietary' format. Also, it uses char all around the place, since using dchar leads to lots og conversion all around the place (phobos use char here and dchar there). If you use it, please report any bugs and / or robustness problems. At least it should compile :) Lars Ivar Igesund Brad Beveridge wrote:Hi all, is there a way to calculate what source files should be compiled when one file has changed? At the moment I am recompiling my whole project, but that is a little bit of overkill. Can the DMD compiler generate dependancies that are suitable for putting in a makefile, or is there some tool out there? Cheers Brad------------------------------------------------------------------------ /* * ddepcheck.d * Version 0.9.9 * Last modified 1st of August 2004 * * Dependency walker for D source files. * Copyright 2003-2004 Lars Ivar Igesund <larsivar'at'igesund.net> * * Permission to use, copy, modify, distribute and sell this software * and its documentation for any purpose is hereby granted without fee, * provided that the above copyright notice appear in all copies and * that both that copyright notice and this permission notice appear * in supporting documentation. It is provided "as is" without express * or implied warranty. */ /* * BUGS: * - (local) dll.d leads to a crash */ /* * TODO: * - Fix importing: blocks of private imports, list of imports * - Audit code, some of it is hairy as hell. * - Exchange ctype with utype when it shows up * - Document better, especially code * - Long term; With multiple input files, cache import results. */ import std.conv; import std.ctype; import std.file; import std.path; import std.stream; import std.string; import std.c.stdio; import std.stdio; import std.utf; alias std.ctype.isdigit isdigit; version (Win32) { import std.c.windows.windows; } bit helpcalled = false; bit versioncalled = false; bit write = false; bit makesyntax = false; bit comment = false; bit runtime = false; bit checkprivate = false; bit fulldebug = false; int curopt = 1; int numpaths = 0; int allpaths = 10; int depthlimit = -1; int depth = 0; int numdeps = 0; int alldeps = 10; int firstfile = 1; char [][] depmods; char [][] depfiles; char [][] depdepths; char [][] paths; char [][] argslist; char [][] dbgidentsstr; char [][] veridentsstr; int dbgidentsnum; int veridentsnum; /* * Add version identifier strings to the list used by ddepcheck based on what * is defined by the compiler. */ static this() { version (DigitalMars) { veridentsstr ~= "DigitalMars"; } version (X86) { veridentsstr ~= "X86"; } version (AMD64) { veridentsstr ~= "AMD64"; } version (Windows) { veridentsstr ~= "Windows"; } version (Win32) { veridentsstr ~= "Win32"; } version (Win64) { veridentsstr ~= "Win64"; } version (linux) { veridentsstr ~= "linux"; } version (LittleEndian) { veridentsstr ~= "LittleEndian"; } version (BigEndian) { veridentsstr ~= "BigEndian"; } version (D_InlineAsm) { veridentsstr ~= "D_InlineAsm"; } } /* * Function called when the "help" option is specified. */ void optHelp() { optVersion(); if (!helpcalled) { writefln(); writefln("Syntax: ddepcheck [options] [src ...]"); writefln(" -I[path] Add a path to be searched."); writefln(" -h/--help Prints this help table."); writefln(" -d=[num]/--depth=[num] Limit the depth searched"); writefln(" for dependencies."); writefln(" -m/--make-syntax Print the dependencies using the make"); writefln(" syntax. \"objectfile : src deps\""); writefln(" -V/--version Prints the version."); writefln(" -w/--writetofile Prints the dependencies to the file"); writefln(" 'depfile' instead of to the console."); writefln(" -l/--checkruntimelib Tries to add runtimelib dependencies."); writefln(" Note that the path must be added "); writefln(" explicitly."); writefln(" -debug=[ident/num] Uses the same switch as the compiler"); writefln(" to remove/include debug codeblocks."); writefln(" -version=[ident/num] Uses the same switch as the compiler"); writefln(" to remove/include version codeblocks."); } helpcalled = true; } /* * Function called when the "checkruntimelib" option is specified. */ void optCheckRuntime() { runtime = true; } /* * Function called when the "checkprivate" option is specified. */ void optCheckPrivate() { checkprivate = true; } /* * Function called if the "version" option is used. */ void optVersion() { if (!versioncalled) { writefln("ddepcheck"); writefln("Dependency walker for D source files."); writefln("Version 0.9.9 Copyright Lars Ivar Igesund 2003 - 2004"); } versioncalled = true; } /* * Function called if the "writetofile" option is used. */ void optWrite() { write = true; } /* * Function called if the "depth" option is used to decide the maximum * depth for recursion. */ void optDepth(int arg, bit doubledash) { if (doubledash) { depthlimit = cast(int)atoi(argslist[arg][8..argslist[arg].length]); } else { depthlimit = cast(int)atoi(argslist[arg][3..argslist[arg].length]); } } /* * Function called for all the "-I" import options used. */ void optImport(int arg) { if (argslist[arg].length == 2) { return; } addPath(argslist[arg][2..argslist[arg].length]); } /* * Function called for all the "-debug" options used. */ void optDebug(int arg) { if (argslist[arg].length == 6) { fulldebug = true; } else { char [] ident = argslist[arg][7..argslist[arg].length]; bit num = isNumber(ident); if (num) { dbgidentsnum = toInt(ident); } else { dbgidentsstr ~= ident; } } } /* * Function called for all the "-version" options used. */ void optVersion(int arg) { if (argslist[arg].length == 2) { return; } char [] ident = argslist[arg][9..argslist[arg].length]; bit num = isNumber(ident); if (num) { veridentsnum = toInt(ident); } else { veridentsstr ~= ident; } } /* * Function called when the "-m"/"--make-syntax" option is used. */ void optMakeSyntax() { makesyntax = true; } /* * Function that checks if an argument is an option. */ bit checkOption(int arg) { switch (argslist[arg]) { case "--version": case "-V": optVersion(); return true; break; case "--help": case "-h": optHelp(); return true; break; case "--writetofile": case "-w": optWrite(); return true; break; case "--make-syntax": case "-m": optMakeSyntax(); return true; break; case "--checkruntimelib": case "-l": optCheckRuntime(); return true; break; case "--checkprivate": case "-p": optCheckPrivate(); return true; break; default: break; } if (cmp(argslist[arg][0..3], "-d=") == 0) { optDepth(arg, false); return true; } else if (argslist[arg].length > 8 && cmp(argslist[arg][0..8], "--depth=") == 0) { optDepth(arg, true); return true; } if (cmp(argslist[arg][0..2], "-I") == 0) { optImport(arg); return true; } if (cmp(argslist[arg][0..6], "-debug") == 0) { optDebug(arg); return true; } if (cmp(argslist[arg][0..8], "-version") == 0) { optVersion(arg); return true; } return false; } /* * Function that starts the dependency walking for base source files. */ void depBase(int arg) { addNewBaseFile(argslist[arg]); depWalk(argslist[arg], "", true); } /* * Function that do the main dependency walking recursively. If it find * files that has been walked before, it skips it to avoid infinite * cyclic dependencies. */ void depWalk(char [] file,char [] mod, bit base) { debug writefln("Filename: ", file); char [] filepath = file; char [] src; int pathnum = 0; int [] importpos; importpos.length = 0; int nextimport = -1; char [][] verspecstrings; char [][] dbgspecstrings; while (pathnum < numpaths && !fileExist(filepath)) { filepath = std.path.join(paths[pathnum], file); pathnum++; } try { src = (new File(filepath)).toString(); } catch (Error e) { return; } if (!base) { if (!addDep(filepath, mod, depth)) { return; } } stripComment(src); bit useVersion(int pos) { int i = pos + 7; int j; while (src[i] != '(') { i++; } j = i + 1; while (src[j] != ')') { j++; } char [] ident = src[i + 1..j]; if (ident.length == 0) { return true; } bit num = true; foreach (char c; ident) { if (!isdigit(c)) { num = false; break; } } if (num) { int identnum = toInt(ident); if (veridentsnum >= identnum) { return true; } } else { foreach (char [] id; veridentsstr) { if (id == ident) { return true; } else if (id == "none") { return false; } } } return false; } void search(int a, int z) { int dbgfound = -1; int verfound = -1; int impfound = -1; int dbgend = -1; int verend = -1; int i = a; bit useDebug(int pos) { int i = pos + 5; int j; if (i >= src.length) { return false; } while (i < (src.length - 1) && isspace(src[i])) { i++; } if (src[i] == '(') { j = i + 1; while (j < (src.length - 1) && src[j] != ')') { j++; } char [] ident = src[i + 1..j]; bit num = true; foreach (char c; ident) { if (!isdigit(c)) { num = false; break; } } if (num) { int identnum = toInt(ident); if (dbgidentsnum <= identnum) { return true; } } else { foreach (char [] id; dbgidentsstr ~ dbgspecstrings) { if (id == ident) { return true; } } } if (fulldebug) { return false; } } else { if (fulldebug) { return true; } } return false; } int findDebugAndVersionEnd(int pos, bit ver, bit other) { int findSpecEnd(int pos) { if (other) { return 0; } int j = pos; if (pos >= src.length) { return 0; } while (j < (src.length - 1) && isspace(src[j])) { j++; } if (src[j] == '=') { int startidx = -1; j++; while (j < (src.length - 1) && src[j] != ';') { if (startidx == -1 && isalnum(src[j])) { startidx = j; } j++; } char [] ident = src[startidx..j]; if (ident.length > 0) { bit num = isNumber(ident); if (ver) { if (num) { try { veridentsnum = toInt(ident); } catch (Exception e) { // Catching top exception here due to strange naming // conventions in Phobos. larsivi 20040724 return j; } } else { verspecstrings ~= ident; } } else { if (num) { try { dbgidentsnum = toInt(ident); } catch (Exception e) { // Catching top exception here due to strange naming // conventions in Phobos. larsivi 20040724 return j; } } else { dbgspecstrings ~= ident; } } } return j; } return 0; } int findBlockEnd(int pos) { debug { if (ver) { writefln("- Finding version block end after pos ", pos); } else { writefln("- Finding debug block end after pos ", pos); } } int j = pos; if (pos >= src.length) { return 0; } while (j < (src.length - 1) && isspace(src[j])) { j++; } int findEnd() { while (isspace(src[j])) { j++; } if (src[j] == '{') { j++; while (src[j] != '}') { j++; } return j; } else { return 0; } } if (src[j] != '(') { return findEnd(); } else { j++; while (src[j] != ')') { j++; } j++; } return findEnd(); } int findAttrEnd(int pos) { int j = pos; if (pos >= src.length) { return 0; } while (j < (src.length - 1) && isspace(src[j])) { j++; } if (src[j] == ':') { while (j < (src.length - 1) && src[j] != '}') { j++; } return j; } else { return 0; } } int findStmntEnd(int pos) { int j = pos; if (pos >= src.length) { return 0; } while (j < (src.length - 1) && src[j] != ';') { j++; } return j; } int i = pos; int res = 0; if (i >= src.length) { return -1; } if (other) { i += 4; } else if (ver) { i += 7; } else { i += 5; } if (cast(bit)(res = findSpecEnd(i))) { return res; } else if (cast(bit)(res = findBlockEnd(i))) { return res; } else if (cast(bit)(res = findAttrEnd(i))) { return res; } else if (cast(bit)(res = findStmntEnd(i))) { return res; } else { return -1; } } int findElse(int pos, bit ver) { int i = pos; while (isspace(src[i++])) { } if (src[i..i + 4] == "else") { i += 4; return findDebugAndVersionEnd(i, ver, true); } else { return -1; } } bit isKeyword(int startpos, int length) { if (startpos < 0) { return false; } if (startpos + length >= src.length) { return false; } if (startpos == 0 || isspace(src[startpos - 1]) || src[startpos] == ';') { if (((startpos + length) == src.length) || isspace(src[startpos + length]) || (src[startpos..startpos + length] == "debug" && src[startpos + length] == ':')) { return true; } } return false; } int checkDebugAndVersion(int pos, bit ver) { int elseend = -1; int end = findDebugAndVersionEnd(pos, ver, false); int found = -1; if (end == -1) { return pos + 1; } elseend = findElse(end, ver); if (ver ? useVersion(pos) : useDebug(pos)) { search(pos, end); if (elseend > -1) { return elseend + 1; } else { return end + 1; } } else if (elseend > -1) { search(end + 1, elseend); return elseend + 1; } return pos + 1; } int handleDebug(int pos) { return checkDebugAndVersion(pos, false); } int handleVersion(int pos) { return checkDebugAndVersion(pos, true); } int handleImport(int pos) { debug writefln("- Handling import at position ", pos); int endpos; int i = pos + 8; while (true) { if (src[i] == ',') { // handle list of imports possibly // FIXME: Implement break; } else if (src[i] == ';') { break; } i++; if (i == src.length) { return i; } } endpos = i; if (!base && !checkprivate) { if (pos > 7) { i = pos - 1; while (isspace(src[i]) && i > 7) { i--; } if (i >= 7 && src[i - 6..i + 1] == "private") { return endpos + 1; } } } char [] mod = strip(src[pos + 6..endpos]); if (!checkIfModule(mod)) { return endpos + 1; } if (!runtime) { if (checkPhobos(mod)) { return endpos + 1; } } char [] fp = createFilePath(mod); depth++; if (depthlimit == -1 || depth <= depthlimit) { depWalk(fp, mod, false); } depth--; return endpos + 1; } int delegate(int) handler; int best; do { handler = null; best = z + 1; uint belowz = z; debug writefln("- Searching for import from ", i, " to ", z); impfound = find(src[i..z], "import"); if (impfound > -1) { impfound += i; belowz = impfound; } debug writefln("- Searching for debug from ", i, " to ", belowz); dbgfound = find(src[i..belowz], "debug"); if (dbgfound > -1) { dbgfound += i; belowz = dbgfound < belowz ? dbgfound : belowz; } debug writefln("- Searching for version from ", i, " to ", belowz); verfound = find(src[i..belowz], "version"); if (verfound > -1) { verfound += i; } if (impfound > -1) { if (isKeyword(impfound + i, 6)) { best = impfound + i; handler = &handleImport; } } if (dbgfound > -1 && (dbgfound + i) < best) { if (isKeyword(dbgfound + i, 5)) { best = dbgfound + i; handler = &handleDebug; } } if (verfound > -1 && (verfound + i) < best) { if (isKeyword(verfound + i, 7)) { best = verfound + i; handler = &handleVersion; } } if (!(handler is null)) { i = handler(best); } else { break; } } while (i < z); } int pos = 0; int found = -1; search(0, src.length); dbgidentsstr ~= dbgspecstrings; veridentsstr ~= verspecstrings; } /* * Function that adds a dependency to the list when it is verified. * A path and the depth where it was found is also added. If the * dependency has been found before, the new depth is added to the * same entry. */ bit addDep(char [] filepath, char [] mod, int depth) { for (int i = 0; i < numdeps; i++) { if (depmods[i] == mod) { depdepths[i] ~= ","; depdepths[i] ~= toString(depth); return false; } } numdeps++; if (numdeps > alldeps) { alldeps *= 2; depmods.length = alldeps; depfiles.length = alldeps; depdepths.length = alldeps; } depmods[numdeps - 1] = mod; depfiles[numdeps - 1] = filepath; depdepths[numdeps - 1] = toString(depth); return true; } /* * Function that adds the filename of the current checked file to * the printout. */ void addNewBaseFile(char [] file) { numdeps += 3; if (numdeps > alldeps) { alldeps *= 2; depmods.length = alldeps; depfiles.length = alldeps; depdepths.length = alldeps; } depmods[numdeps-2] = file; depfiles[numdeps-3] = depfiles[numdeps-2] = depfiles[numdeps-1] = ""; depdepths[numdeps-3] = depdepths[numdeps-2] = depdepths[numdeps-1] = ""; } /*! * Function that takes a module name and creates a real path. */ char [] createFilePath(char [] mod) { char [] tempmod; tempmod = replace(mod, ".", sep); tempmod ~= ".d"; return tempmod; } /*! * Checks if the found dependency is a part of phobos in which case * it is ignored. */ bit checkPhobos(char [] mod) { if (mod == "Object") { return true; } else if (mod.length < 4) { return false; } else if (cmp(mod[0..4], "std.") == 0) { return true; } else { return false; } } /*! * Checks if the found module really can be a module since the parsing * is a bit hackish. */ bit checkIfModule(char [] mod) { if (!isalpha(mod[0])) { return false; } foreach (char c; mod[1..mod.length]) { if (!(isalnum(c) || c == '.')) { return false; } } return true; } /* * Adds a path given with "-I" to the path list. */ void addPath(char [] path) { numpaths++; if (numpaths > allpaths) { allpaths *= 2; paths.length = allpaths; } paths[numpaths - 1] = path; } /*! * Checks if a file exist. */ bit fileExist(char [] file) { version (Win32) { return (GetFileAttributesA(toStringz(file)) != 0xFFFFFFFF); } version (Linux) { // FIXME: I guess there is a better way to do this. larsivi 18042004 bit result = true; try { read(file); } catch (FileException e) { result = false; } return result; } } /*! * Checks if the string is a number. Returns true if it is. */ bit isNumber(char [] ident) { foreach (char c; ident) { if (!isdigit(c)) { return false; } } return true; } /*! * Convert a substring in src to whitespace. */ void toWhiteSpace(inout char [] src, int start, int end) in { assert (end > start); assert (start >= 0); assert (end <= src.length); } body { src[start..end] = ' '; } /*! * Strip off the comments. */ void stripComment(inout char[] src) { int len = src.length; int doubleslash = -1; int star = -1; int plus = 0; int plusstart = -1; for (int i = 0; i < len; i++) { if (doubleslash > -1) { if (src[i] == '\\') { if (src[i + 1] == 'n') { toWhiteSpace(src, doubleslash, i + 2); i++; doubleslash = -1; continue; } } continue; } else if (star > -1) { if (src[i] == '*') { if (src[i + 1] == '/') { toWhiteSpace(src, star, i + 2); i++; star = -1; continue; } } continue; } else if (plus) { if (src[i] == '+') { if (src[i + 1] == '/') { i++; plus--; if (!plus) { toWhiteSpace(src, plusstart, i + 1); plusstart = -1; } continue; } } } if (src[i] == '/') { i++; if (src[i] == '/') { doubleslash = i - 1; } else if (src[i] == '*') { star = i - 1; } else if (src[i] == '+') { if (!plus) { plusstart = i - 1; } plus++; } } } } /* * Writes dependencies to file. */ void printToFile() { File depfile = new File("depfile", FileMode.Out); for (int i = 0; i < numdeps; i++) { depfile.printf(depmods[i] ~ " " ~ depfiles[i] ~ " " ~ depdepths[i] ~ "\n"); } depfile.close(); } /* * Prints dependencies to stdout. */ void printToStdout() { // FIXME: Use writef(ln) // larsivi 20040715 if (makesyntax) { char [] objsuf = ""; version (Win32) { objsuf = ".obj"; } version (linux) { objsuf = ".o"; } writef(replace(argslist[firstfile], ".d", objsuf), " : "); writef(argslist[firstfile]); if (numdeps > 3) { writef(" "); for (int i = 3; i < numdeps; i++) { writef(depfiles[i], " "); } } } else { for (int i = 0; i < numdeps; i++) { writefln(depmods[i], " ", depfiles[i], " ", depdepths[i]); } } } int main(char[][] args) { if (args.length == 1) { optHelp(); return 0; } depmods.length = 10; depfiles.length = 10; depdepths.length = 10; paths.length = 10; argslist = args; while (curopt < args.length && checkOption(curopt)) { curopt++; } if (curopt < args.length) { firstfile = curopt; for (int i = curopt; i < args.length; i++) { // FIXME: Handle multiple input files better // larsivi 20040715 depBase(i); } } if (write) { printToFile(); } else { printToStdout(); } return 0; }
Aug 13 2004
In article <cfa1vf$1g6u$1 digitaldaemon.com>, Brad Beveridge says...Hi all, is there a way to calculate what source files should be compiled when one file has changed?YesAt the moment I am recompiling my whole project, but that is a little bit of overkill. Can the DMD compiler generate dependancies that are suitable for putting in a makefile, or is there some tool out there?When I want to recompile my project, I click on "Tools/D Build (Debug)" or "Tools/D Build (Release") from within my text editor (TextPad). When I do this, only those files which are out of date, or are (recursively) dependent upon changed files, are recompiled/relinked. There is no makefile - only a simple (optional) config file which allows me to build with different version() names defined. The dependency tree is figured out just by parsing import statements. The compiler output appears in a TextPad window. In the event of a compile error, I can double-click on the error line and TextPad jumps directly to the source file line containing the error. To make this work, you need TextPad (or some other suitably powerful text editor) and a build tool which does the job. I wrote my own build tool - I wrote it in PHP. You're welcome to it, but it won't work for you unless you also have PHP installed. (Also, because of the way I wrote it, my tool expects the file containing "main" to be called "main.d". You could change that by editing the source, or just live with it, because it's a nice convention anyway). Let me know if you want it. Arcane Jill
Aug 10 2004
I just compile my entire project every time. It simplifies things and compiles just as fast anyway. The D compiler is very, very fast. Less than a second for 700k of source on a PentiumM 1.6 is pretty damn good. Brad Beveridge wrote:Hi all, is there a way to calculate what source files should be compiled when one file has changed? At the moment I am recompiling my whole project, but that is a little bit of overkill. Can the DMD compiler generate dependancies that are suitable for putting in a makefile, or is there some tool out there? Cheers Brad
Aug 10 2004