www.digitalmars.com         C & C++   DMDScript  

D - Compiling Phobos

reply Burton Radons <loth users.sourceforge.net> writes:
I'm working on listdir and stat functions for the file module in
Phobos, which naturally requires compiling Phobos once I'm done.  The
makefile is out of date, however, and the directories are all wrong.

There's more.  array, memset, obj, and switcherr are not included in
the distribution but symbols from them are used.  interface.c,
intrinsic.d, iunknown.d, and time.d are not touched by the makefile.
None of them provide the necessary symbols, so I'm guessing there's
files missing.

Finally, sc compiles the .asm files incorrectly (it adds a semicolon
to the filename, bizarrely enough), and the version of masm and link I
got don't like the options used.  It'd be nice if the obj file were
just stuck in the distribution.

Now, onto listdir, statdir, and stat.  This code is in the public
domain blah blah blah and it follows.  I tried to follow Walter's
coding style, except for the no-indent-for-trivial-blocks, which I've
never seen before and hopefully will never see again.

This requires "ERROR_NO_MORE_FILES = 18," in the appropriate place in
windows.d and FileTimeToSystemTime using "export BOOL
FileTimeToSystemTime(FILETIME* lpFileTime, SYSTEMTIME*
lpSystemTime);".

-- /* For the string module */ --

/***************************************************
 * Determine whether the string ends with another.
 */

bit endswith(char[] string, char[] match)
{
    if (string.length < match.length)
        return 0;
    if (memcmp(&string [string.length - match.length], &match [0],
match.length))
        return 0;
    return 1;
}

/***************************************************
 * Determine whether the string begins with another.
 */

bit startswith(char[] string, char[] match)
{
    if (string.length < match.length)
        return 0;
    if (memcmp(&string [0], &match [0], match.length))
        return 0;
    return 1;
}

-- /* For the file module */ --

import date;
import path;

/***************************************************
 * Structure of a file stat.
 */

class Stat
{
    char[] filename; /* Filename plus full path */
    char[] filebase; /* Filename without path */
    char[] alternate; /* Alternate (8.3) filename, no path */
    d_time creation; /* When this file was created */
    d_time access; /* When this file was last used (this will probably
be writeTime in Linux) */
    d_time write; /* When this file was last written */
    long filesize; /* File size in bytes */

    bit archive; /* Is an archive file */
    bit directory; /* Is a directory */
    bit hidden; /* Is hidden (Linux should probably flag this if the
file starts with .) */
    bit readonly; /* Is read-only (Linux should set this if the
process can't write this file) */
    bit system; /* Is a system file */

    this (char[] path, WIN32_FIND_DATA finder)
    {
        SYSTEMTIME systemtime;
        FILETIME filetime;

        /* Get the filename and file size */
        filebase = (char []) finder.cFileName;
        filename = path ~ filebase;
        alternate = ((char []) finder.cAlternateFileName).dup;
        filesize = (long) finder.nFileSizeHigh * (long) 0x100000000 +
finder.nFileSizeLow;

        /* Get the attributes */
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_ARCHIVE)
            archive = 1;
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)
            directory = 1;
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_HIDDEN)
            hidden = 1;
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_READONLY)
            readonly = 1;
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_SYSTEM)
            system = 1;

        /* Get the times */

        filetime = finder.ftCreationTime;
        if (filetime.dwLowDateTime == 0 && filetime.dwHighDateTime ==
0)
            filetime = finder.ftLastWriteTime;
        if (!FileTimeToSystemTime(&filetime, &systemtime))
            throw new FileError(filename, GetLastError());
        creation = SYSTEMTIME2d_time(&systemtime, 0);

        filetime = finder.ftLastAccessTime;
        if (filetime.dwLowDateTime == 0 && filetime.dwHighDateTime ==
0)
            filetime = finder.ftLastWriteTime;
        if (!FileTimeToSystemTime(&filetime, &systemtime))
            throw new FileError(filename, GetLastError());
        access = SYSTEMTIME2d_time(&systemtime, 0);

        filetime = finder.ftLastWriteTime;
        if (!FileTimeToSystemTime(&filetime, &systemtime))
            throw new FileError(filename, GetLastError());
        write = SYSTEMTIME2d_time(&systemtime, 0);
    }
}

/***************************************************
 * Collate all files in a directory, returning an
 * array of strings.  The strings do not contain the
 * path.  Throws FileError if the directory does not
 * exist, if there was some file problem during the
 * collation, or if the path is malformed (generally
 * trying to use nonportable features).
 */

char[][] listdir(char[] path)
{
    WIN32_FIND_DATA finder;
    HANDLE handle;
    char[][] list;
    bit done = 0;

    /* Since we want to flatten out platform differences,
       first we cry if the user tried to use Windows. */

    if (find (path, (char) "*") >= 0 || find (path, (char) "?") >= 0)
        throw new FileError(path, "Path contains invalid characters");

    /* Form the search path for Windows */
    if (endswith(path, "/") || endswith(path, "\\"))
        path ~= "*";
    else
        path ~= "/*";

    /* Start the file search */
    handle = FindFirstFileA(toStringz(path ~ "*"), &finder);
    if (handle == (HANDLE)-1)
        throw new FileError(path, GetLastError());
    
    while (!done)
    {
        char[] filename;

        /* Handle this file in our own inimitable fashion */
        filename = (char[]) finder.cFileName;
        list ~= filename.dup;

        /* Go to the next file or bust */
        if (!FindNextFileA(handle, &finder))
        {
            if (GetLastError() == ERROR_NO_MORE_FILES)
                done = 1;
            else
            {
                FindClose(handle);
                throw new FileError(path, GetLastError());
            }
        }
    }

    /* Finished, clean up */
    if (!FindClose(handle))
        throw new FileError(path, GetLastError());

    return list;
}

/***************************************************
 * Collate all files in a directory, returning an
 * array of stats.  Throws FileError if the
 * directory does not exist, if there was some file
 * problem during the collation, or if the path is
 * malformed (generally trying to use nonportable
 * features).
 */

Stat[] statdir(char[] path)
{
    WIN32_FIND_DATA finder;
    HANDLE handle;
    Stat[] list;
    bit done = 0;

    /* Since we want to flatten out platform differences,
       first we cry if the user tried to use Windows. */

    if (find (path, (char) "*") >= 0 || find (path, (char) "?") >= 0)
        throw new FileError(path, "Path contains invalid characters");

    /* Form the search path for Windows */
    if (!endswith(path, "/") && !endswith(path, "\\"))
        path ~= "/";

    /* Start the file search */
    handle = FindFirstFileA(toStringz(path ~ "*"), &finder);
    if (handle == (HANDLE)-1)
        throw new FileError(path, GetLastError());
    
    while (!done)
    {
        /* Handle this file in our own inimitable fashion */
        list ~= new Stat (path, finder);

        /* Go to the next file or bust */
        if (!FindNextFileA(handle, &finder))
        {
            if (GetLastError() == ERROR_NO_MORE_FILES)
                done = 1;
            else
            {
                FindClose(handle);
                throw new FileError(path, GetLastError());
            }
        }
    }

    /* Finished, clean up */
    if (!FindClose(handle))
        throw new FileError(path, GetLastError());

    return list;
}

/***************************************************
 * Attempt to stat a single file, returning the
 * file details.  This will throw FileError when the
 * file cannot be found or when the path is badly
 * formed.
 */
 
Stat stat(char[] filename)
{
    WIN32_FIND_DATA finder;
    HANDLE handle;
    char[] path;
    Stat stat;

    /* Since we want to flatten out platform differences,
       first we cry if the user tried to use Windows. */

    if (find (filename, (char) "*") >= 0 || find (filename, (char)
"?") >= 0)
        throw new FileError(filename, "Path contains invalid
characters");

    /* Start the file search */
    handle = FindFirstFileA(toStringz(filename), &finder);
    if (handle == (HANDLE)-1)
        throw new FileError(filename, GetLastError());

    /* Construct the path */
    if (filename [0] != "/" && filename [0] != "\\")
        path = curdir;
    else
        path = getDirName(filename);

    /* Statify the sucker */
    stat = new Stat(path, finder);

    /* Finish up */
    if (!FindClose(handle))
        throw new FileError(path ~ pathsep, GetLastError());

    return stat;
}
Apr 26 2002
next sibling parent reply Burton Radons <loth users.sourceforge.net> writes:
Not really on the same subject, but I've put up a pre-alpha version of
a quick program maker for D; most Deed files (as the program is
called) are a single line, but have inherited options, dependencies
checking, and multitargets.  You can grab it at:

http://prdownloads.sourceforge.net/willow/deed.zip

The only extra symbol I needed was system:

"extern (C) { export int system(char* string); }
int system(char[] string) { return system ((char *) &(string ~ "\0")
[0]); }"

There apparently aren't many solid programs using D, so here's my
observations.

It's about as easy to prototype in D as it is in Python, if a little
easier (and that's saying something)... particularly now that I have
Deed, of course.

Array slicing should use default arguments; [ .. ] refers to the full
array, [x .. ] refers to x to the end of the array, [ .. x] refers to
the start of the array to x.  It's convenient.  I also like Python's
negative array indexing, but I can see many reasons to oppose it.

I want operator overloading; something like:

operator a + b (qbert a, int b) { qbert (a.x + b); }

Clean and unambiguous; C++'s method is way too confusing.

I want control of the parser and compiler at runtime.  I also want to
be able to take a class instance and reconstruct its class definition
perfectly.  I then want to be able to pick up dynamic modules and link
them into the running program myself.  This is all for my never-ending
quest for a scripting language that can act as a system language.

Having runtime compiler control is not only useful for modularization
and scripting, but also for dynamic code generation of OpenGL shaders
and utility parsers that always work with the language.

Otherwise things are either working good or are on a clear path to
getting good.  Nothing has made me feel that such is the wrong way to
go, although I'm opposed to templates.
Apr 26 2002
parent reply "Pavel Minayev" <evilone omen.ru> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:4kekcu4p1pbtp790rvugkbatvln5b3ra2l 4ax.com...


 It's about as easy to prototype in D as it is in Python, if a little
 easier (and that's saying something)... particularly now that I have
 Deed, of course.


Agreed. D is definitely much better in this area than C or C++,
and this is a great achievement.


 Array slicing should use default arguments; [ .. ] refers to the full
 array, [x .. ] refers to x to the end of the array, [ .. x] refers to
 the start of the array to x.  It's convenient.  I also like Python's
 negative array indexing, but I can see many reasons to oppose it.


Default arguments for slicing would be very nice as well. It's nothing
more than syntactic sugar, but it's convenient, why not add it?


 I want control of the parser and compiler at runtime.  I also want to
 be able to take a class instance and reconstruct its class definition
 perfectly.  I then want to be able to pick up dynamic modules and link
 them into the running program myself.  This is all for my never-ending
 quest for a scripting language that can act as a system language.


Wouldn't it bloat the resulting programs too much? I can understand the
need for RTTI and dynamic construction of objects of unknown classes.
But how are you going to control PARSER at run-time??? And linking
modules dynamically... I think it's a bit too much... DLLs are there,
isn't it enough?
Apr 27 2002
parent reply Burton Radons <loth users.sourceforge.net> writes:
On Sat, 27 Apr 2002 11:06:00 +0400, "Pavel Minayev" <evilone omen.ru>
wrote:


"Burton Radons" <loth users.sourceforge.net> wrote in message
news:4kekcu4p1pbtp790rvugkbatvln5b3ra2l 4ax.com...


 It's about as easy to prototype in D as it is in Python, if a little
 easier (and that's saying something)... particularly now that I have
 Deed, of course.


Agreed. D is definitely much better in this area than C or C++,
and this is a great achievement.


 Array slicing should use default arguments; [ .. ] refers to the full
 array, [x .. ] refers to x to the end of the array, [ .. x] refers to
 the start of the array to x.  It's convenient.  I also like Python's
 negative array indexing, but I can see many reasons to oppose it.


Default arguments for slicing would be very nice as well. It's nothing
more than syntactic sugar, but it's convenient, why not add it?


Sugar makes string processing go round.  Out of 14 string slices in
Deed, 13 use either 0 or the string length on the appropriate side,
pretty typical.


 I want control of the parser and compiler at runtime.  I also want to
 be able to take a class instance and reconstruct its class definition
 perfectly.  I then want to be able to pick up dynamic modules and link
 them into the running program myself.  This is all for my never-ending
 quest for a scripting language that can act as a system language.


Wouldn't it bloat the resulting programs too much? I can understand the
need for RTTI and dynamic construction of objects of unknown classes.
But how are you going to control PARSER at run-time??? And linking
modules dynamically... I think it's a bit too much... DLLs are there,
isn't it enough?


What do you mean by bloat?  If the parser's not linked in, it takes no
space.  The parser part of the compiler shouldn't take much space at
all even when it is linked in.

A parser class would mostly create tokens from an input stream, as
well as have functions for reading in declarations and expressions and
whatnot, plus a list of keywords.  It doesn't have to be a
particularly complex interface, just consistent with the language.  So
I could quickly pass through expressions and find any import
statements no matter what future changes occur.  Good tools to parse
source code are awful hard to make correct; it'd be nice if the
language itself standardized this.

The compiler would produce Module, Class, or Function objects from a
stream.  The common expression compile should be done by building
functions.  It's for scripting.  I'm sick and/or tired of having to
use two languages in a project; the reason I'm here is as part of my
exploration to get out of the paradigm.  C/Python was the best I've
ever tried, but getting the two languages to work together was a lot
of work, and I had to put a lot of Python-applicable code in C merely
because Python was so slow.

Now that I've thought through it some more, I can see a way to make D
behave for the most part, so dynamic linking control is not necessary.

The only thing I need from here is a compiler and parser, actually,
for in-game entities with code.  But I'll let it simmer until I'm
actually at the point that I could use it.

Oh, Walter: Trying to compile mfile.d in Deed using -O causes an
"Internal error: ..\ztc\gloop.c 1299".  It appears to be in the
listdir function, specifically the line "filename = (char[])
finder.cFileName;"  Commenting this out removes the internal error,
although a doppleganger of this line appears earlier in Stat's
constructor.
Apr 27 2002
next sibling parent reply "Pavel Minayev" <evilone omen.ru> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg 4ax.com...


 What do you mean by bloat?  If the parser's not linked in, it takes no
 space.  The parser part of the compiler shouldn't take much space at
 all even when it is linked in.


Parser of what? The program is already compiled to machine code!


 A parser class would mostly create tokens from an input stream, as
 well as have functions for reading in declarations and expressions and
 whatnot, plus a list of keywords.  It doesn't have to be a
 particularly complex interface, just consistent with the language.  So
 I could quickly pass through expressions and find any import
 statements no matter what future changes occur.  Good tools to parse
 source code are awful hard to make correct; it'd be nice if the
 language itself standardized this.


So, you propose to integrate a D interpreter into each program?


 The compiler would produce Module, Class, or Function objects from a
 stream.  The common expression compile should be done by building
 functions.  It's for scripting.  I'm sick and/or tired of having to
 use two languages in a project; the reason I'm here is as part of my
 exploration to get out of the paradigm.  C/Python was the best I've
 ever tried, but getting the two languages to work together was a lot
 of work, and I had to put a lot of Python-applicable code in C merely
 because Python was so slow.


I guess it's better to have an external D interpreter module (DScript
or something), available as a part of Phobos. Not something built
into the language itself though.


 The only thing I need from here is a compiler and parser, actually,
 for in-game entities with code.  But I'll let it simmer until I'm
 actually at the point that I could use it.


Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed
a standard interpreter module could be done. Seems like a worthy
idea.

... wanna try?
Apr 27 2002
next sibling parent reply Burton Radons <loth users.sourceforge.net> writes:
On Sat, 27 Apr 2002 20:58:32 +0400, "Pavel Minayev" <evilone omen.ru>
wrote:


"Burton Radons" <loth users.sourceforge.net> wrote in message
news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg 4ax.com...


 What do you mean by bloat?  If the parser's not linked in, it takes no
 space.  The parser part of the compiler shouldn't take much space at
 all even when it is linked in.


Parser of what? The program is already compiled to machine code!


Parser of D.  You're missing what this would provide, simply: put the
parser and compiler in a library and provide access to it.  That's it,
nothing weird about the language or anything, just a refactoring and
an interface on the D side.

[snip]

 The only thing I need from here is a compiler and parser, actually,
 for in-game entities with code.  But I'll let it simmer until I'm
 actually at the point that I could use it.


Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed
a standard interpreter module could be done. Seems like a worthy
idea.


Well, let's see.  Of the 1930kb of C code in my current project,
1230kb is actual implementation and 700.39kb is devoted to nothing
more than interfacing Python; that's 36% of the body.  The Python code
itself sums up to only 375kb.  The balance is all wacky, definitely
not acceptable, but very typical.

With runtime access to the D compiler and full introspection, on the
other hand, getting the compiler to interact is pretty trivial, and
anything I do will come with the guarantee that it will run as fast as
my machine and compiler can make it.


... wanna try?


Been there, done that, have many, many T-shirts.  If I were to do this
I would write a compiler to machine code; apart from floating point
they're not particularly hard to do dumbly and are easier to interact
with the rest of the system.  But then there's a lot of duplication of
effort; it'd be like using a Tonka truck to dig a swimming pool when
you have the keys to the backhoe in your pocket.
Apr 29 2002
next sibling parent "Pavel Minayev" <evilone omen.ru> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com...


Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed
a standard interpreter module could be done. Seems like a worthy
idea.


 Well, let's see.  Of the 1930kb of C code in my current project,
 1230kb is actual implementation and 700.39kb is devoted to nothing
 more than interfacing Python; that's 36% of the body.  The Python code
 itself sums up to only 375kb.  The balance is all wacky, definitely
 not acceptable, but very typical.

 With runtime access to the D compiler and full introspection, on the
 other hand, getting the compiler to interact is pretty trivial, and
 anything I do will come with the guarantee that it will run as fast as
 my machine and compiler can make it.


This sounds like a special kind of computer language, quite different
by the very concept from traditional all-purpose languages. Personally,
I wouldn't like D to turn into something like that. It might be a
useful tool, but not for everybody; not for me, at least.
Apr 29 2002
prev sibling next sibling parent reply "Richard Krehbiel" <rich kastle.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com...

 On Sat, 27 Apr 2002 20:58:32 +0400, "Pavel Minayev" <evilone omen.ru>
 wrote:


Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed
a standard interpreter module could be done. Seems like a worthy
idea.


 With runtime access to the D compiler and full introspection, on the
 other hand, getting the compiler to interact is pretty trivial, and
 anything I do will come with the guarantee that it will run as fast as
 my machine and compiler can make it.


Script languages usually have a set of problems to solve that differ from
the problems of compiled languages.  For in-game code, there is an
additional difficulty: keeping the game player safe.

Quake C and UnrealScript solve this problem.  It's not possible to harm the
gamer's computer with these languages.  He/she can download mods (or have
them automatically downloaded during on-line play) with a high level of
comfort.  If in-game code is in a native-compiled language, then you've got
an opportunity to spread worms and viruses and cause real harm.

Do you *really* want to subject your audience to malware like that?  Just so
that you have the convenience of coding the core game engine and logic in
the same language?

And I might even suggest another alternative: Java.  The Java sandbox works,
and JIT compilation works.  You can use a Java native-code compiler for game
engine performance, and load Java .class files for mods, which are then
JIT-compiled.

(Having said that, I know id Software experimented using Java for Quake 3
game logic, and performance considerations caused them to shelve it.
Instead, Quake 3 builds in a C compiler; game mods are programmed in a safe
C subset, which is compiled at load time to actual native machine code.)

--
Richard Krehbiel, Arlington, VA, USA
rich kastle.com (work) or krehbiel3 comcast.net  (personal)
Apr 29 2002
parent reply "Walter" <walter digitalmars.com> writes:
"Richard Krehbiel" <rich kastle.com> wrote in message
news:aajmkt$qtu$1 digitaldaemon.com...

 (Having said that, I know id Software experimented using Java for Quake 3
 game logic, and performance considerations caused them to shelve it.
 Instead, Quake 3 builds in a C compiler; game mods are programmed in a


safe

 C subset, which is compiled at load time to actual native machine code.)


That's an interesting idea. Does this C subset not allow pointers?
Apr 29 2002
parent reply Russell Borogove <kaleja estarcion.com> writes:
Walter wrote:

 "Richard Krehbiel" <rich kastle.com> wrote in message
 news:aajmkt$qtu$1 digitaldaemon.com...
 

(Having said that, I know id Software experimented using Java for Quake 3
game logic, and performance considerations caused them to shelve it.
Instead, Quake 3 builds in a C compiler; game mods are programmed in a


 
 safe
 

C subset, which is compiled at load time to actual native machine code.)


 
 
 That's an interesting idea. Does this C subset not allow pointers?


Quake 1 used QuakeC, which didn't allow pointers (and
actually didn't allow a lot of things -- I wouldn't
call it a C subset in fact; it was a vaguely C looking
scripting language). If I remember rightly it was
compiled to a tokenized or "bytecode"[1] form at game
build time (or maybe at load time, it's been a while).

Quake 2 used native DLLs.

According to 5 minutes of Google research, it looks to
me like Quake 3 also uses native DLLs:

http://www.planetquake.com/code3arena/tutorials/tutorial2.shtml

I believe they decided to leave security as a "caveat
player" thing in favor of performance and the ready
availability of tools.

-Russell B

[1] Anyone else hate this term as much as I do?
Apr 29 2002
next sibling parent reply "Richard Krehbiel" <rich kastle.com> writes:
"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...

 Walter wrote:

 "Richard Krehbiel" <rich kastle.com> wrote in message
 news:aajmkt$qtu$1 digitaldaemon.com...


(Having said that, I know id Software experimented using Java for Quake






3

game logic, and performance considerations caused them to shelve it.
Instead, Quake 3 builds in a C compiler; game mods are programmed in a


 safe


C subset, which is compiled at load time to actual native machine code.)



 That's an interesting idea. Does this C subset not allow pointers?


 Quake 1 used QuakeC, which didn't allow pointers (and
 actually didn't allow a lot of things -- I wouldn't
 call it a C subset in fact; it was a vaguely C looking
 scripting language). If I remember rightly it was
 compiled to a tokenized or "bytecode"[1] form at game
 build time (or maybe at load time, it's been a while).

 Quake 2 used native DLLs.

 According to 5 minutes of Google research, it looks to
 me like Quake 3 also uses native DLLs:

 http://www.planetquake.com/code3arena/tutorials/tutorial2.shtml


Um, I had forgotten that Quake 3 also allows game mods using .DLLs.

But read the very bottom where it talks about QVMs.


 I believe they decided to leave security as a "caveat
 player" thing in favor of performance and the ready
 availability of tools.


Quake 2 was definitely programmed that way; game modes *are* done in
native-code .DLLs.

Here's a link that goes into more detail on Quake 3's mod system:

http://firingsquad.gamers.com/hardware/q3adrivers/default.asp

(I dimly recall John Carmack describing his QVM to native code JIT-compiler,
but I can't find any links...)

--
Richard Krehbiel, Arlington, VA, USA
rich kastle.com (work) or krehbiel3 comcast.net  (personal)
Apr 29 2002
parent Burton Radons <loth users.sourceforge.net> writes:
On Mon, 29 Apr 2002 15:03:15 -0400, "Richard Krehbiel"
<rich kastle.com> wrote:


"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...


[snip]

 I believe they decided to leave security as a "caveat
 player" thing in favor of performance and the ready
 availability of tools.


Quake 2 was definitely programmed that way; game modes *are* done in
native-code .DLLs.

Here's a link that goes into more detail on Quake 3's mod system:

http://firingsquad.gamers.com/hardware/q3adrivers/default.asp

(I dimly recall John Carmack describing his QVM to native code JIT-compiler,
but I can't find any links...)


It's in his .plan records, try (finger.quakeworld.com); I can't recall
the date.  However, it is entirely optional to do this.  You can go
through the full machine code route purely at your own option.

Accountability and peer review pretty much eliminates the danger
there.  Can anyone think of one instance where this hole was exploited
by a cracker?  One?  Otherwise understand if I'm dubious about this
being a real problem... or whether it is relevant to us in any case.

We have the compiler; we can define restricted compilation; it can be
as secure as restricted execution in interpreted languages; it is no
more difficult to make it secure than it is to make an interpreted
language secure.

If restricted compilation is not enough, uh, use an interpreted
language.  Better yet, convert your simplified scripting language to D
and compile that.  I haven't claimed a panacea here.
Apr 29 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...

 scripting language). If I remember rightly it was
 compiled to a tokenized or "bytecode"[1] form at game
 build time (or maybe at load time, it's been a while).
 [1] Anyone else hate this term as much as I do?



LOL. I've written interpreters, and while bytecode minimizes object code
size, it impairs runtime performance. Better to make your interpreter
instructions word size and align them.
Apr 29 2002
parent reply Burton Radons <loth users.sourceforge.net> writes:
On Mon, 29 Apr 2002 14:54:06 -0700, "Walter" <walter digitalmars.com>
wrote:


"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...

 scripting language). If I remember rightly it was
 compiled to a tokenized or "bytecode"[1] form at game
 build time (or maybe at load time, it's been a while).
 [1] Anyone else hate this term as much as I do?


LOL. I've written interpreters, and while bytecode minimizes object code
size, it impairs runtime performance. Better to make your interpreter
instructions word size and align them.


The FORTH way is to make these function pointers to call, then in the
functions tail into the next instruction using jmp (gforth does this).
Very fast stuff and smaller than native code, but bound to be eclipsed
in the future by optimising native code compilers (bigforth does
this).
Apr 29 2002
parent "Walter" <walter digitalmars.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:bisrcu4ng7qcqartp3cec51mb851bqviep 4ax.com...

 On Mon, 29 Apr 2002 14:54:06 -0700, "Walter" <walter digitalmars.com>
 wrote:

"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...

 scripting language). If I remember rightly it was
 compiled to a tokenized or "bytecode"[1] form at game
 build time (or maybe at load time, it's been a while).
 [1] Anyone else hate this term as much as I do?


LOL. I've written interpreters, and while bytecode minimizes object code
size, it impairs runtime performance. Better to make your interpreter
instructions word size and align them.


 The FORTH way is to make these function pointers to call, then in the
 functions tail into the next instruction using jmp (gforth does this).
 Very fast stuff and smaller than native code, but bound to be eclipsed
 in the future by optimising native code compilers (bigforth does
 this).


Ah yes, threaded code.
Apr 30 2002
prev sibling parent reply Patrick Down <pat codemoon.com> writes:
Burton Radons <loth users.sourceforge.net> wrote in 
news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:

[snip]

 Parser of D.  You're missing what this would provide, simply: put the
 parser and compiler in a library and provide access to it.  That's it,
 nothing weird about the language or anything, just a refactoring and
 an interface on the D side.


You could use this to build a scripting engine into you program and
it would be very fast.  However there is a security issue with it
depending on what you want to use it for.  For example with games
you may want to auto download game mods much like Unreal Tournament 
does.  How do you prevent the mod writer from doing something malicious?
You can certainly filter the functions that downloaded script can call
but the main issue is that D still allows pointer manipulation. Given
a pointer a clever coder can gain access to system level functions.

So you need a more sandbox friendly version of D which ends up being
a different language anyway.

I do like the idea of having the compiler modules as libraries.  I talked 
about something similar in a different thread here but I had different 
reasons for wanting it.  Building the compiler in a modular way allows 
user defined processing to done between the compilation stages.

[snip]
 

 Well, let's see.  Of the 1930kb of C code in my current project,
 1230kb is actual implementation and 700.39kb is devoted to nothing
 more than interfacing Python; that's 36% of the body.  The Python code
 itself sums up to only 375kb.  The balance is all wacky, definitely
 not acceptable, but very typical.


I've done a lot of C++/Python integration too.  It's true that code to 
hook the two together is a pain.  I eventually invented my own IDL 
language.
I would run it through a script that would auto generate 99.9% of the 
hook 
code for me.  Of course the runtime bulk is still there.


 Been there, done that, have many, many T-shirts.  If I were to do this
 I would write a compiler to machine code; apart from floating point
 they're not particularly hard to do dumbly and are easier to interact
 with the rest of the system.  But then there's a lot of duplication of
 effort; it'd be like using a Tonka truck to dig a swimming pool when
 you have the keys to the backhoe in your pocket.


Still I think a separate system is needed for this type of application.
What I've really looking for is an embedable JIT compiler that also takes 
care of sandbox issues.
Apr 29 2002
next sibling parent reply Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Patrick Down wrote:


 Burton Radons <loth users.sourceforge.net> wrote in
 news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:

 [snip]

 Parser of D.  You're missing what this would provide, simply: put the
 parser and compiler in a library and provide access to it.  That's it,
 nothing weird about the language or anything, just a refactoring and
 an interface on the D side.


 You could use this to build a scripting engine into you program and
 it would be very fast.  However there is a security issue with it
 depending on what you want to use it for.  For example with games
 you may want to auto download game mods much like Unreal Tournament
 does.  How do you prevent the mod writer from doing something malicious?
 You can certainly filter the functions that downloaded script can call
 but the main issue is that D still allows pointer manipulation. Given
 a pointer a clever coder can gain access to system level functions.

 So you need a more sandbox friendly version of D which ends up being
 a different language anyway.


Hey, it can be D--   :)

D, without pointers, without asm, with only access to a very carefully
limited subset of the library, could potentially be a very good safe
scripting language.  I proposed (something very like) this back on 8/20/01 in
"A Java-style sandbox without a Virtual Machine?", but there was a lot of
confusion on what I meant by it :(

I did get a pointer to Bochs from that thread, which was a real win :)



Even if Walter doesn't feel like supporting it...we could do it.

--
The Villagers are Online! villagersonline.com

.[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ]
.[ (a version.of(English).(precise.more)) is(possible) ]
?[ you want.to(help(develop(it))) ]
Apr 29 2002
next sibling parent reply Patrick Down <pat codemoon.com> writes:
Russ Lewis <spamhole-2001-07-16 deming-os.org> wrote in
news:3CCD9200.64D5035D deming-os.org: 


 Patrick Down wrote:
 

 Burton Radons <loth users.sourceforge.net> wrote in
 news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:

 [snip]

 Parser of D.  You're missing what this would provide, simply: put
 the parser and compiler in a library and provide access to it. 
 That's it, nothing weird about the language or anything, just a
 refactoring and an interface on the D side.


 You could use this to build a scripting engine into you program and
 it would be very fast.  However there is a security issue with it
 depending on what you want to use it for.  For example with games
 you may want to auto download game mods much like Unreal Tournament
 does.  How do you prevent the mod writer from doing something
 malicious? You can certainly filter the functions that downloaded
 script can call but the main issue is that D still allows pointer
 manipulation. Given a pointer a clever coder can gain access to
 system level functions. 

 So you need a more sandbox friendly version of D which ends up being
 a different language anyway.


 
 Hey, it can be D--   :)
 
 D, without pointers, without asm, with only access to a very carefully
 limited subset of the library, could potentially be a very good safe
 scripting language.  I proposed (something very like) this back on
 8/20/01 in "A Java-style sandbox without a Virtual Machine?", but
 there was a lot of confusion on what I meant by it :(
 
 I did get a pointer to Bochs from that thread, which was a real win :)
 
 
 
 Even if Walter doesn't feel like supporting it...we could do it.
 


I would be cool to do this with D but it would require some
restrictions.

1. Heavy control over imports
2. No pointers
3. No asm
4. Array bounds checking is always turned on.
Apr 29 2002
parent C.R.Chafer (DrWhat?) <blackmarlin nospam.asean-mail.com> writes:
Patrick Down wrote:


 
 I would be cool to do this with D but it would require some
 restrictions.
 
 1. Heavy control over imports
 2. No pointers
 3. No asm
 4. Array bounds checking is always turned on.
 


For a more theoritical discriptions of restrictions required you could try 
www.tunes.org .  I believe one of their goals is the production of a HLL 
which could be checked at compile time disallowing access to privledged 
code - their site has been updated since last time I looked so I no longer 
have the link - it probably can be found under subprojects / HLL.

C 2002/4/29
Apr 29 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Russ Lewis" <spamhole-2001-07-16 deming-os.org> wrote in message
news:3CCD9200.64D5035D deming-os.org...

 Even if Walter doesn't feel like supporting it...we could do it.


It's not a matter of feeling like it, it's just that I'm supporting 4
compilers simultaneously, and I get stretched a little thin. People ask me
why I don't play chess, etc. After working on compilers all night, I prefer
mindless entertainment <g>.

Anyhow, source to the D front end is now part of the distribution, and if
anyone wants to morph that into an interpretted version, that would be a
cool addition to D.

Also, any gcc experts out there want to create a gcc version of the D
compiler?
May 01 2002
parent reply Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Walter wrote:


 "Russ Lewis" <spamhole-2001-07-16 deming-os.org> wrote in message
 news:3CCD9200.64D5035D deming-os.org...

 Even if Walter doesn't feel like supporting it...we could do it.


 It's not a matter of feeling like it, it's just that I'm supporting 4
 compilers simultaneously, and I get stretched a little thin. People ask me
 why I don't play chess, etc. After working on compilers all night, I prefer
 mindless entertainment <g>.


Sorry, I very definitely did NOT mean that to sound like you were lazy, random,
or arbitrary.  I am VERY happy that you put all this work into D.  Like I said
when I first arrived at this newsgroup...I'd been working on a spec very much
like your D spec for a year or more, but I had exactly 0 lines of code actually
written for it.

(gets up on soapbox)

Hear ye!  Hear ye!  Walter Bright is a good, hard working man!  Let all gather
together with me and express their great appreciation for his long hours and
his willingness to give (at least some of it) away for free!  Hear ye!  Hear
ye!

(gets down, laughs at self)


 Anyhow, source to the D front end is now part of the distribution, and if
 anyone wants to morph that into an interpretted version, that would be a
 cool addition to D.


I'm going to have to look at the code you've written for that...is it
architected where somebody could remove your code and drop in an alternative
version, or would it be a pretty involved port?

In a perfect world, there would be a parser module that turns D code into a
recursive set of structures that could then be passed to any arbitrary backend:
an interpreter, a GCC frontend, a dfront translator, or even a D-- compiler.

--
The Villagers are Online! villagersonline.com

.[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ]
.[ (a version.of(English).(precise.more)) is(possible) ]
?[ you want.to(help(develop(it))) ]
May 01 2002
next sibling parent "Walter" <walter digitalmars.com> writes:
"Russ Lewis" <spamhole-2001-07-16 deming-os.org> wrote in message
news:3CCFF42C.A7C4906E deming-os.org...

 Sorry, I very definitely did NOT mean that to sound like you were lazy,


random,

 or arbitrary.  I am VERY happy that you put all this work into D.  Like I


said

 when I first arrived at this newsgroup...I'd been working on a spec very


much

 like your D spec for a year or more, but I had exactly 0 lines of code


actually

 written for it.


Ok!


 (gets up on soapbox)

 Hear ye!  Hear ye!  Walter Bright is a good, hard working man!  Let all


gather

 together with me and express their great appreciation for his long hours


and

 his willingness to give (at least some of it) away for free!  Hear ye!


Hear

 ye!

 (gets down, laughs at self)


Let's not get carried away here!


 I'm going to have to look at the code you've written for that...is it
 architected where somebody could remove your code and drop in an


alternative

 version, or would it be a pretty involved port?


The source is for the front end only, so an alternative wouldn't make much
sense.


 In a perfect world, there would be a parser module that turns D code into


a

 recursive set of structures that could then be passed to any arbitrary


backend:

 an interpreter, a GCC frontend, a dfront translator, or even a D--


compiler.
May 01 2002
prev sibling parent Burton Radons <loth users.sourceforge.net> writes:
On Wed, 01 May 2002 06:57:01 -0700, Russ Lewis
<spamhole-2001-07-16 deming-os.org> wrote:


Walter wrote:


[snip]

 Anyhow, source to the D front end is now part of the distribution, and if
 anyone wants to morph that into an interpretted version, that would be a
 cool addition to D.


I'm going to have to look at the code you've written for that...is it
architected where somebody could remove your code and drop in an alternative
version, or would it be a pretty involved port?

In a perfect world, there would be a parser module that turns D code into a
recursive set of structures that could then be passed to any arbitrary backend:
an interpreter, a GCC frontend, a dfront translator, or even a D-- compiler.


Another speaks of the devil.  Working on it now; lexer is done,
expression parsing appears done, statement parsing is halfway done,
still have to do class/struct/union/function parsing, then cleaning,
then testing it to hell, then documentation.  Few days perhaps for a
beta?

D in, uh, restricted compilation mode, doesn't even warrant a variant
from what's been discussed.  It could be done by making a kind of lint
that confirms a file's validity under the stricter rules, then tells
the caller whether it's all right to compile the code.
May 01 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Patrick Down" <pat codemoon.com> wrote in message
news:Xns91FF8288ECF6Fpatcodemooncom 63.105.9.61...

 Burton Radons <loth users.sourceforge.net> wrote in
 news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:
 I've done a lot of C++/Python integration too.  It's true that code to
 hook the two together is a pain.  I eventually invented my own IDL
 language.
 I would run it through a script that would auto generate 99.9% of the
 hook
 code for me.  Of course the runtime bulk is still there.


What would it take to implement in D:

    extern (Python) int some_python_function(int foo);

basically, make D really easy to interface to Python?
May 01 2002
next sibling parent reply Burton Radons <loth users.sourceforge.net> writes:
On Wed, 1 May 2002 03:42:54 -0700, "Walter" <walter digitalmars.com>
wrote:


"Patrick Down" <pat codemoon.com> wrote in message
news:Xns91FF8288ECF6Fpatcodemooncom 63.105.9.61...

 Burton Radons <loth users.sourceforge.net> wrote in
 news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:
 I've done a lot of C++/Python integration too.  It's true that code to
 hook the two together is a pain.  I eventually invented my own IDL
 language.
 I would run it through a script that would auto generate 99.9% of the
 hook
 code for me.  Of course the runtime bulk is still there.


What would it take to implement in D:

    extern (Python) int some_python_function(int foo);

basically, make D really easy to interface to Python?


The more common direction, by a gigantic majority, is calling X from
Python rather than Python from X.  In my code the number of such cases
can be counted on a hand that has fought with a wood chipper: three
times, out of perhaps a thousand functions and methods.

For all three times I don't see any way that they could be supported
automatically; each has really specialized circumstances.

Similarly classes are far more common than functions (although that
will vary with the project, of course).  Until recently you couldn't
subclass builtin types in Python, and that would be easier to support.
You could support Python subclassed types, however, by preprocessing a
proxy class, and that would even allow calling Python-overloaded
methods from D.

So I think this should be alotted to a preprocessor that takes a D
file and constructs an interface to Python, like SWIG, but more
automatic.  There's no magic bullet, that's for sure.  D does help
make it easier, but it's a lot of work in any case.
May 01 2002
parent Keith Ray <k1e2i3t4h5r6a7y 1m2a3c4.5c6o7m> writes:
In article <7oovcukrmtcvnvvlbuhh9s0vup7b9mk9nd 4ax.com>,
 Burton Radons <loth users.sourceforge.net> wrote:
[...]

 The more common direction, by a gigantic majority, is calling X from
 Python rather than Python from X.  In my code the number of such cases
 can be counted on a hand that has fought with a wood chipper: three
 times, out of perhaps a thousand functions and methods.
 
 For all three times I don't see any way that they could be supported
 automatically; each has really specialized circumstances.
 
 Similarly classes are far more common than functions (although that
 will vary with the project, of course).  Until recently you couldn't
 subclass builtin types in Python, and that would be easier to support.
 You could support Python subclassed types, however, by preprocessing a
 proxy class, and that would even allow calling Python-overloaded
 methods from D.
 
 So I think this should be alotted to a preprocessor that takes a D
 file and constructs an interface to Python, like SWIG, but more
 automatic.  There's no magic bullet, that's for sure.  D does help
 make it easier, but it's a lot of work in any case.


Interesting thing about Objective-C is how well it can work with Java, 
Ruby, and Python without needing pre-processing of code... The thing 
those four languages have in common is good reflection/introspection. 

For calling any of those languages from ObjectiveC, you can take 
advantage of the fact that polymorphism in ObjectiveC does NOT require 
inheritance. 

You can call any method on (that is, "send any message to") any object. 
If that object doesn't understand that method/message, then the run-time 
code will invoke the method "forwardInvocation:" on the object (if it 
exists), passing in as the argument information about the originally 
intended message and arguments. In that object's implementation of 
"forwardInvocation:", it can use functions in Java (via JNI), Python, 
Ruby, etc., to forward that call and arguments to that other language. 
The same technique is used for 'distributed objects' -- forwarding calls 
through a network to a remote computer.

More information about this is in:

<http://developer.apple.com/techpubs/macosx/Cocoa/ObjectiveC/5RunTime/For
warding.html>

<http://developer.apple.com/techpubs/macosx/Cocoa/ObjectiveC/4MoreObjC/iR
emote_Messaging.html>

You might ask, if Objective C is so wonderful, why don't more people use 
it. I think three reasons: 

One, early implementations of Objective C were proprietary and 
expensive, so it lost mindshare to C++, which was being given away (in 
the form of cfront) to universities by AT&T. 

Two, the 'object-oriented' portions of Objective C use a syntax similar 
to Smalltalk, which is looks alien to C programmers. 

Three, the current maintainer of the Objective C FAQ wrote his own 
'non-standard' Objective C implementation, and hates Apple/Next. Since 
most questions on the objective C newsgroup are asked by people using 
Apple/Cocoa or NextStep, those message threads turn into flames-fests 
regularly. (Is there any other newsgroup where the FAQ maintainer is 
also the resident troll?)

Today, Objective C is available in gcc, so the only thing preventing 
people from using it is their lack of familiarity (few environments 
require Objective C, so few people take the time to learn it) and the 
lack of good objective-c run-time library on the user's platform. 

For example, on MacOS 9, I'm not aware of any useful implementation of 
the Objective-c run-time library. On other platforms, the run-time 
library is there (provided by gnu), but it may not have the full breadth 
of capabilities of the one NeXT developed and that is now in Apple's 
MacOS X -- for example, the gnu implementation of objective-c run-time 
library may not have distributed (over networks) objects support.

Even though the Objective C syntax is 'alien', it is simple to learn -- 
much simpler than C++. Of course, it doesn't have templates and 
operator-overloading, but that is what makes it easy to learn. People 
who rely on Objective C love the syntax, since it basically names each 
argument.

   [anInvocation invokeWithTarget:someOtherObject];
is equivalent to C++
   anInvocation->invokeWithTarget( someOtherObject );

Now imagine that there are more arguments...

   [anObject setWidth: 12.0 height: 13.0 colorMode: 'gray'  
       taxes: 'irs' filmMaker: 'woo' star: 'sirius' ];
is equivalent to 
   anObject->setLotsOfStuff(12.0, 13.0, 'gray', 'irs', 'sirius', 'woo');

(or was that supposed to be:
   anObject->setLotsOfStuff(12.0, 13.0, 'gray', 'irs', 'woo', 'sirius');
?)

Of course, many IDEs these days let you look up the calling docs for a 
function or method easily, so this advantage is not as big a deal as it 
used to be.

To get back to my point. I think the ideal language is simplified C++ 
with Objective C's runtime support, including the ability to "send a 
message" to any object and allow a class to implement 
'forwardInvocation:' in order to catch unimplemented method-calls and 
handle them appropriately.

This ability to catch unimplemented method-calls makes interfacing with 
dynamic languages like Python, Ruby, Java easy.
-- 
C. Keith Ray

<http://homepage.mac.com/keithray/xpminifaq.html>
May 01 2002
prev sibling parent reply Patrick Down <pat codemoon.com> writes:
"Walter" <walter digitalmars.com> wrote in news:aaogn4$2pau$1
 digitaldaemon.com:

 
 What would it take to implement in D:
 
     extern (Python) int some_python_function(int foo);
 
 basically, make D really easy to interface to Python?
 


First off you need classes too.

extern(Python) class some_python_class
{
  long some_python_method(long foo);
}

The way I usually handle this is to generate a helper
class like the following.  

class some_python_class : public CPyObj
{
  public:
  some_python_class() : CPyObj() {}
  
  some_python_class(some_python_class& rO)
  : CPyObj(rO) {}
    
  int some_python_method(long p1)
  {
    CPyObj args;
    args.CreateTuple(1);
    args.SetTupleItem(1,PyInt_FromLong(p1));
    CPyObj meth = GetAttr("some_python_method");
    if(!meth.IsValid())
    { 
      MethodNotFoundError("some_python_method");
      return 0;
    }
    CPyObj rtn = meth.Call(args);
    return PyInt_AsLong(rtn);
  }
};

CPyObj contains the handle to the real Python object and 
automatically deals with reference counting and other 
utility functions.

However this is all trivial compared to calling C code from
Python.  You need to build tables mapping string names to
function pointers.  You need to fill out other data structures that 
define things like constructors, destructors, and where to fine 
attribute and method tables.  You need code to type check and unpack
parameters from Python's variant type and call the C function with
them.  Then type need to pack up the return value into a
python object and return it.  Are you letting Python control
the lifetime of the object?  Then you need code to track the 
reference count and delete it.  "Programming Python" is a good 
reference it has a couple of chapters devoted to Python/C interfacing.
I will append an example to the end of this post.

In D you would need the equivalent reverse syntax.  Something 
this might cover it.

export(Python) int Func(int a, char[] b)
{
  return a;
}

export(Python) class CFoo
{
  export(Python) int Method(int a)
  {
    return a;
  }
}


However I will still go with my opinion that a tool
outside the compiler is a better option.  But I will say
supporting such tools with some user defined attribute

attributes are actually classes derived from System.Attribute
that can be discovered via reflection.  But perhaps we don't 
need anything as complicated.

defattr scriptable;

scriptable
int Func(int a,char[] b)
{
  return a;
}

scriptable("constructor=no")
class Foo
{
}

The D front end parser would know how to parse the attribute
syntax but wouldn't do anything with the information.  A 
third party tool could use the D front end to create a code
generator for interfacing the D module to a scripting engine, 
Python or anything else.



Excerpts from a Python to C example.  The example shown
connects to a C++ class called CListControl derived from 
CBaseControl.

class CListControl
{
  public:
  virtual long AddString(const char*);

  virtual long InsertString(long,const char*);

  virtual long AddStringData(const char*,CPyObj);

  // Cut

  virtual long Get_SelectedIndex();
  virtual void Set_SelectedIndex(long);

  virtual long Get_ListSize();
  virtual void Set_ListSize(long);

  virtual long Get_Enabled();
  virtual void Set_Enabled(long);

  // Cut
};


This is rather old stuff from Python 2.0 days. 2.2 is a little bit 
easier to handle.  The real cpp file is over 4000 lines long so I've
cut out sections.  Forgive me if it's rather ugly.  :-)



// GUIFramePy.h generated by pyintgen.py
// Wed Sep 19 17:02:33 2001

// Cut some stuff here

#include <PyObj.h>

// Cut some stuff here

struct CListControl_PyObj
{
  PyObject_HEAD
  CListControl* m_pObj;
};

// Cut some stuff here



// GUIFramePy.cpp generated by pyintgen.py
// Wed Sep 19 17:02:33 2001


// Cut some stuff here

#include "listctrl.h"  
 
  
#include "GUIFramePy.h"

static PyObject* pyErrorObj;

// Cut some stuff here


PyObject* CListControl_AddString(CListControl_PyObj* pObj, PyObject* 
pArgs)
{
  const char* p1;
  if(!PyArg_ParseTuple(pArgs,"z" , &p1)) return 0;
  long rtn =  pObj->m_pObj->AddString(p1);
  return PyInt_FromLong(rtn);
}

PyObject* CListControl_InsertString(CListControl_PyObj* pObj, PyObject* 
pArgs)
{
  long p1;
  const char* p2;
  if(!PyArg_ParseTuple(pArgs,"iz" , &p1, &p2)) return 0;
  long rtn =  pObj->m_pObj->InsertString(p1,p2);
  return PyInt_FromLong(rtn);
}

PyObject* CListControl_AddStringData(CListControl_PyObj* pObj, PyObject* 
pArgs)
{
  const char* p1;
  CPyObj p2;
  if(!PyArg_ParseTuple(pArgs,"zO&" , &p1,Conv_PyObj, &p2)) return 0;
  long rtn =  pObj->m_pObj->AddStringData(p1,p2);
  return PyInt_FromLong(rtn);
}

// Cut some stuff here

PyMethodDef CListControl_methods[] = {
  { "AddString", (PyCFunction)CListControl_AddString, 1 },
  { "InsertString", (PyCFunction)CListControl_InsertString, 1 },
  { "AddStringData", (PyCFunction)CListControl_AddStringData, 1 },
  { "InsertStringData", (PyCFunction)CListControl_InsertStringData, 1 },
  { "DeleteItem", (PyCFunction)CListControl_DeleteItem, 1 },
  { "DeleteAll", (PyCFunction)CListControl_DeleteAll, 1 },
  { "SetData", (PyCFunction)CListControl_SetData, 1 },
  { "GetData", (PyCFunction)CListControl_GetData, 1 },
  { "GetText", (PyCFunction)CListControl_GetText, 1 },
  { "LoadDirectory", (PyCFunction)CListControl_LoadDirectory, 1 },
  { 0, 0, 0 }
};

PyObject* CListControl_ObjAttr(CListControl_PyObj* self, char *pAttrName)
{
  if(strcmp(pAttrName,"SelectedIndex") == 0)
    return PyInt_FromLong(self->m_pObj->Get_SelectedIndex());
  else if(strcmp(pAttrName,"ListSize") == 0)
    return PyInt_FromLong(self->m_pObj->Get_ListSize());
  else if(strcmp(pAttrName,"Enabled") == 0)
    return PyInt_FromLong(self->m_pObj->Get_Enabled());
  else if(strcmp(pAttrName,"LD_READWRITE") == 0)
    return PyInt_FromLong(DDL_READWRITE);
  else if(strcmp(pAttrName,"LD_READONLY") == 0)
    return PyInt_FromLong(DDL_READONLY);
  else if(strcmp(pAttrName,"LD_HIDDEN") == 0)
    return PyInt_FromLong(DDL_HIDDEN);
  else if(strcmp(pAttrName,"LD_SYSTEM") == 0)
    return PyInt_FromLong(DDL_SYSTEM);
  else if(strcmp(pAttrName,"LD_DIRECTORY") == 0)
    return PyInt_FromLong(DDL_DIRECTORY);
  else if(strcmp(pAttrName,"LD_ARCHIVE") == 0)
    return PyInt_FromLong(DDL_ARCHIVE);
  else if(strcmp(pAttrName,"LD_DRIVES") == 0)
    return PyInt_FromLong(DDL_DRIVES);
  else if(strcmp(pAttrName,"LD_EXCLUSIVE") == 0)
    return PyInt_FromLong(DDL_EXCLUSIVE);

  return 0;
}

PyMethodChain CListControl_chain = { CListControl_methods, 
&CBaseControl_chain };

PyObject* CListControl_getattr(CListControl_PyObj* self, char *pAttrName)
{
  PyObject* pyObjRtn = CListControl_ObjAttr(self,pAttrName);
  if(pyObjRtn)
    return pyObjRtn;
      
  // Really nasty solution to a multiple inheritance problem
  // Fix this someday
  CListControl* pTmp = self->m_pObj;                                           
    
  ((CBaseControl_PyObj*)self)->m_pObj = (CBaseControl*)self->m_pObj;
    
  pyObjRtn = CBaseControl_ObjAttr((CBaseControl_PyObj*)self,pAttrName);
  
  self->m_pObj = pTmp;
  
  if(pyObjRtn)
    return pyObjRtn; 
    
  return Py_FindMethodInChain(&CListControl_chain,(PyObject*)
self,pAttrName);
}


int CListControl_setattr(CListControl_PyObj* self, char *pAttrName, 
PyObject* pyValue)
{
  if(strcmp(pAttrName,"SelectedIndex") == 0)
  {
    self->m_pObj->Set_SelectedIndex(static_cast<long>(PyInt_AsLong
(pyValue)));
    return 0;
  }
  else if(strcmp(pAttrName,"Enabled") == 0)
  {
    self->m_pObj->Set_Enabled(static_cast<long>(PyInt_AsLong(pyValue)));
    return 0;
  }

  return -1;
}

// Cut some stuff here




void CListControl_dealloc(CListControl_PyObj* self)
{
  MODTRACE("CListControl(%p) dereference\n",self);

  if(self->m_pObj)
    self->m_pObj->ResetPyObject();
#if 0   
  delete self->m_pObj;
#endif

  PyMem_DEL(self);
}


PyTypeObject CListControl_type = {
  PyObject_HEAD_INIT(&PyType_Type)
  0,
  "CListControl",
  sizeof(CListControl_type),
  0,

  (destructor)	CListControl_dealloc,
  0,
  (getattrfunc) CListControl_getattr,
  (setattrfunc) CListControl_setattr,
  0,
  0,

  0,
  0,
  0,

  0,
  0,
  0
};

// Cut some stuff here

int Conv_CListControl(PyObject* pyObj,CListControl** pVar)
{
  if(pyObj->ob_type == &CListControl_type)
  {
    *pVar = ((CListControl_PyObj*)pyObj)->m_pObj;
    return 1;
  }
  else if(pyObj == Py_None)
  {
    *pVar = 0;
    return 1;
  }
  
  PyErr_SetString(pyErrorObj,"Function requires a CListControl");
  
  return 0;
}

// Cut some stuff here

// Module level functions
PyMethodDef AppFrame_methods[] = {
  { "RunApp", (PyCFunction)RunApp_PyFunc, 1 },
  { "ShowDialog", (PyCFunction)ShowDialog_PyFunc, 1 },
  { "FileBrowse", (PyCFunction)FileBrowse_PyFunc, 1 },
  { "ShellExec", (PyCFunction)ShellExec_PyFunc, 1 },
  {0, 0, 0}
};


void initAppFrame()
{
  PyObject* pMod;
  PyObject* pDict;
  
  pMod = Py_InitModule("AppFrame", AppFrame_methods);
  pDict = PyModule_GetDict(pMod);
  
  pyErrorObj = Py_BuildValue("s","AppFrame.error");
  
  PyDict_SetItemString(pDict,"error",pyErrorObj);
  
  if(PyErr_Occurred())
    Py_FatalError("Can't initialize module AppFrame");
  else
    InstallDataStruct();
}
May 01 2002
parent "Walter" <walter digitalmars.com> writes:
"Patrick Down" <pat codemoon.com> wrote in message
news:Xns9201B2282EE82patcodemooncom 63.105.9.61...

 However this is all trivial compared to calling C code from
 Python.  You need to build tables mapping string names to
 function pointers.  You need to fill out other data structures that
 define things like constructors, destructors, and where to fine
 attribute and method tables.  You need code to type check and unpack
 parameters from Python's variant type and call the C function with
 them.  Then type need to pack up the return value into a
 python object and return it.  Are you letting Python control
 the lifetime of the object?  Then you need code to track the
 reference count and delete it.  "Programming Python" is a good
 reference it has a couple of chapters devoted to Python/C interfacing.
 I will append an example to the end of this post.


Oh darn. Sounds like a project for another time.
May 02 2002
prev sibling parent reply Patrick Down <pat codemoon.com> writes:
"Pavel Minayev" <evilone omen.ru> wrote in news:aael3n$18ql$1
 digitaldaemon.com:


 
 Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed
 a standard interpreter module could be done. Seems like a worthy
 idea.
 
 ... wanna try?


I like the idea of a standard interpreter module although as I said
in a different thread here what I really want is a embeddable JIT
compiler module.  But in the mean time I'll take a regular old script 
interpreter.  I've thought about doing this for D if I have some
spare time.

I've had a lot of success building applications that mix C++ and Python.
I usually let the Python code handle the GUI stuff like dialogs and it
also acts as a glue layer between the C++ modules.

There are a couple of problems. 

1) The amount of interface code that is needed to tie the interpreter to 
the C++ code.  I have scripts that will automatically generate a lot of 
the code.  But the interface code is still bulky.  

2) Python's objects are garbage collected via reference counting.  Any 
C++ code that references one of these objects had to manually deal with 
the reference count.  And C++ object that is reference from Python has 
to supply a reference count for the interpreter.

3) This is less of a problem and more of a personal preference on my 
part. 
I like Python's power I still don't like it's syntax.  I just can't
handle using indention as the only indicator of code blocks. :-)

Some of these problems could be solved with a scripting language
interpreter written in D that uses extension modules written in D.

Number 2 is easy because the interpreter will use D's garbage 
collector.  All the memory management problems go away because the
interpreter and D will use common memory management.

Number 1 is not quite so easy to solve.  Having the memory management
issue solved does away with some of the interface code but there are
still issues with marshalling function parameters between the interpreter
and D code.  There are two sides to this.  Calling D functions from the
script and calling script functions from D.

Having a script language construct like the "native" directive in Unreal 
Script would allow the interpreter to automatically generate stack frames 
for calls from script into D.  The interpreter could also use reflection 
to find the function pointers in a module.  However, there are issues 
with 
controlling which functions you want exported to the script interpreter.  
If D allowed user defined attribute tagging of classes and functions like 

to things in D that you wanted to export to the interpreter.  Another way
is just to hand a list of exported functions and classes to the 
interpreter
when it is initialized.

Calling script functions from D is still messy but manageable with some
helper functions.
Apr 29 2002
next sibling parent Nuno Nunes <nachn altagis.biz> writes:
 

 I like Python's power I still don't like it's syntax.  I just can't
 handle using indention as the only indicator of code blocks. :-)
 


I think Ruby tries to cover this bad feature (at least to me), and
as a better integration with native code (C).
But it as few (far few) support from libraries
Apr 29 2002
prev sibling next sibling parent reply Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Patrick Down wrote:


 Number 1 is not quite so easy to solve.  Having the memory management
 issue solved does away with some of the interface code but there are
 still issues with marshalling function parameters between the interpreter
 and D code.  There are two sides to this.  Calling D functions from the
 script and calling script functions from D.


I don't know Python, so this is just a shot in the dark, but...could you
define a "Python" calling convention?  D already supports multiple calling
conventions, and this could be another:

    extern(Python) void MyFunc(...);

Thus, you can take functions exported from Python and call them directly from
D with no wrapper code (the compiler generates the wrapper); similarly, you
can export function TO Python and have Python call your D code.

--
The Villagers are Online! villagersonline.com

.[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ]
.[ (a version.of(English).(precise.more)) is(possible) ]
?[ you want.to(help(develop(it))) ]
Apr 29 2002
next sibling parent Nuno Nunes <nachn altagis.biz> writes:
 

 I don't know Python, so this is just a shot in the dark, but...could you
 define a "Python" calling convention?  D already supports multiple calling
 conventions, and this could be another:
 
     extern(Python) void MyFunc(...);
 


this was like a dream come true
Apr 29 2002
prev sibling parent reply Patrick Down <pat codemoon.com> writes:
Russ Lewis <spamhole-2001-07-16 deming-os.org> wrote in
news:3CCDB223.17897881 deming-os.org: 


 Patrick Down wrote:
  
 I don't know Python, so this is just a shot in the dark, but...could
 you define a "Python" calling convention?  D already supports multiple
 calling conventions, and this could be another:
 
     extern(Python) void MyFunc(...);


Yes but there are a lot of scripting languages out there. I wouldn't
ask Walter to support all of them, even Python. :-)  It's a task
better left to an third party tool.
Apr 29 2002
next sibling parent Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Patrick Down wrote:


 Russ Lewis <spamhole-2001-07-16 deming-os.org> wrote in
 news:3CCDB223.17897881 deming-os.org:


 Patrick Down wrote:

 I don't know Python, so this is just a shot in the dark, but...could
 you define a "Python" calling convention?  D already supports multiple
 calling conventions, and this could be another:

     extern(Python) void MyFunc(...);


 Yes but there are a lot of scripting languages out there. I wouldn't
 ask Walter to support all of them, even Python. :-)  It's a task
 better left to an third party tool.


I'm not sure there isn't a way to integrate 3rd party plug-ins into a
compiler...what if you could just download a plugin that would add support
for the Python calling convention to your D compiler?

--
The Villagers are Online! http://villagersonline.com

.[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ]
.[ (a version.of(English).(precise.more)) is(possible) ]
?[ you want.to(help(develop(it))) ]
Apr 29 2002
prev sibling parent "Walter" <walter digitalmars.com> writes:
"Patrick Down" <pat codemoon.com> wrote in message
news:Xns91FFAE8EF54EFpatcodemooncom 63.105.9.61...

 Russ Lewis <spamhole-2001-07-16 deming-os.org> wrote in
 news:3CCDB223.17897881 deming-os.org:


 Patrick Down wrote:

 I don't know Python, so this is just a shot in the dark, but...could
 you define a "Python" calling convention?  D already supports multiple
 calling conventions, and this could be another:

     extern(Python) void MyFunc(...);


 Yes but there are a lot of scripting languages out there. I wouldn't
 ask Walter to support all of them, even Python. :-)  It's a task
 better left to an third party tool.


I've received many requests for supporting an interface to Python.
Interestingly, people who find D appealing also find Python appealing for
their scripting needs.
May 01 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Patrick Down" <pat codemoon.com> wrote in message
news:Xns91FF9D202AFD6patcodemooncom 63.105.9.61...

 I like the idea of a standard interpreter module although as I said
 in a different thread here what I really want is a embeddable JIT
 compiler module.  But in the mean time I'll take a regular old script
 interpreter.  I've thought about doing this for D if I have some
 spare time.


What do people think of adding an ECMAScript interpreter as a Phobos library
routine? (ECMAScript is the official name for javascript, jscript, etc.)
May 01 2002
parent reply "Pavel Minayev" <evilone omen.ru> writes:
"Walter" <walter digitalmars.com> wrote in message
news:aaoh8b$2qe7$2 digitaldaemon.com...


 What do people think of adding an ECMAScript interpreter as a Phobos


library

 routine? (ECMAScript is the official name for javascript, jscript, etc.)


You mean, that one you've written?
Anyhow, YES. Some standartized way to do scripting would be just great.
It's nice to have such a handy tool as a part of distribution: no need
to search for 3rd-party libs, try to compile them, then find out they
crash every second call...
May 01 2002
parent reply "Walter" <walter digitalmars.com> writes:
"Pavel Minayev" <evilone omen.ru> wrote in message
news:aaovcd$t5r$1 digitaldaemon.com...

 "Walter" <walter digitalmars.com> wrote in message
 news:aaoh8b$2qe7$2 digitaldaemon.com...

 What do people think of adding an ECMAScript interpreter as a Phobos


 library

 routine? (ECMAScript is the official name for javascript, jscript, etc.)


 You mean, that one you've written?


Yes. Gotta find some use for it <g>.


 Anyhow, YES. Some standartized way to do scripting would be just great.
 It's nice to have such a handy tool as a part of distribution: no need
 to search for 3rd-party libs, try to compile them, then find out they
 crash every second call...


The interpreter has been well tested, and is pretty solid.
May 01 2002
next sibling parent reply "Pavel Minayev" <evilone omen.ru> writes:
"Walter" <walter digitalmars.com> wrote in message
news:aapcpr$1mt7$1 digitaldaemon.com...


 routine? (ECMAScript is the official name for javascript, jscript,






etc.)

 You mean, that one you've written?


 Yes. Gotta find some use for it <g>.


Well, you have to port it to D first... =)


 Anyhow, YES. Some standartized way to do scripting would be just great.
 It's nice to have such a handy tool as a part of distribution: no need
 to search for 3rd-party libs, try to compile them, then find out they
 crash every second call...


 The interpreter has been well tested, and is pretty solid.


That's what I mean. Just "import script.ecma" - nothing
more needed. I like it! =)
May 01 2002
parent "Walter" <walter digitalmars.com> writes:
"Pavel Minayev" <evilone omen.ru> wrote in message
news:aapeo8$1pes$1 digitaldaemon.com...

 "Walter" <walter digitalmars.com> wrote in message
 news:aapcpr$1mt7$1 digitaldaemon.com...


 routine? (ECMAScript is the official name for javascript, jscript,






 etc.)

 You mean, that one you've written?


 Yes. Gotta find some use for it <g>.


 Well, you have to port it to D first... =)


It would be some work to interface it to D, but I am just not up to
rewriting it.
May 02 2002
prev sibling parent Patrick Down <pat codemoon.com> writes:
"Walter" <walter digitalmars.com> wrote in
news:aapcpr$1mt7$1 digitaldaemon.com: 


 
 "Pavel Minayev" <evilone omen.ru> wrote in message
 news:aaovcd$t5r$1 digitaldaemon.com...

 "Walter" <walter digitalmars.com> wrote in message
 news:aaoh8b$2qe7$2 digitaldaemon.com...

 What do people think of adding an ECMAScript interpreter as a
 Phobos 


 library

 routine? (ECMAScript is the official name for javascript, jscript,
 etc.) 


 You mean, that one you've written?


 
 Yes. Gotta find some use for it <g>.


This would be really cool.
May 01 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg 4ax.com...

 On Sat, 27 Apr 2002 11:06:00 +0400, "Pavel Minayev" <evilone omen.ru>
 wrote:

"Burton Radons" <loth users.sourceforge.net> wrote in message
news:4kekcu4p1pbtp790rvugkbatvln5b3ra2l 4ax.com...

 Array slicing should use default arguments; [ .. ] refers to the full
 array, [x .. ] refers to x to the end of the array, [ .. x] refers to
 the start of the array to x.  It's convenient.  I also like Python's
 negative array indexing, but I can see many reasons to oppose it.


Default arguments for slicing would be very nice as well. It's nothing
more than syntactic sugar, but it's convenient, why not add it?


 Sugar makes string processing go round.  Out of 14 string slices in
 Deed, 13 use either 0 or the string length on the appropriate side,
 pretty typical.


This is a good idea.



 What do you mean by bloat?  If the parser's not linked in, it takes no
 space.  The parser part of the compiler shouldn't take much space at
 all even when it is linked in.


Are you thinking of something like javascript's exec function?


 Oh, Walter: Trying to compile mfile.d in Deed using -O causes an
 "Internal error: ..\ztc\gloop.c 1299".  It appears to be in the
 listdir function, specifically the line "filename = (char[])
 finder.cFileName;"  Commenting this out removes the internal error,
 although a doppleganger of this line appears earlier in Stat's
 constructor.


Ok, could you email me please a reproducible example?
Apr 27 2002
parent reply Burton Radons <loth users.sourceforge.net> writes:
On Sat, 27 Apr 2002 09:50:25 -0700, "Walter" <walter digitalmars.com>
wrote:


"Burton Radons" <loth users.sourceforge.net> wrote in message
news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg 4ax.com...


[snip]

 What do you mean by bloat?  If the parser's not linked in, it takes no
 space.  The parser part of the compiler shouldn't take much space at
 all even when it is linked in.


Are you thinking of something like javascript's exec function?


You mean eval?  No - t'would be a pain to get the local scope into the
dynamic code (and not really be desirable) and the type system would
interfere.  Better would be to construct a function in text, compile
that, then execute it.

For the parser, I just mean the ability to get tokens and parse
declarations, nothing more.  Just a few judicious links into the
frontend as it stands, with the frontend refactored into a library.
So, for a minimal example:

class DLexer
{
    this (wchar[] text);
    int LineNo ();
    DToken Token ();
}

enum DTokenType { ... }

class DToken
{
    DTokenType type;
    DIdentifier ident;
}

class DIdentifier
{
    int value;
    wchar[] string;
}

With additional features added later.  I'm not proposing suddenly
jumping into a full-fledged feature-complete parser, but just a start
that I can add things to.  The only part of the work I can't do is
putting the frontend into a library that the backend uses.  I'm
volunteering my time for the rest.


 Oh, Walter: Trying to compile mfile.d in Deed using -O causes an
 "Internal error: ..\ztc\gloop.c 1299".  It appears to be in the
 listdir function, specifically the line "filename = (char[])
 finder.cFileName;"  Commenting this out removes the internal error,
 although a doppleganger of this line appears earlier in Stat's
 constructor.


Ok, could you email me please a reproducible example?


This is the smallest fragment I could get it into:

void listdir()
{
    char[14] foo;
    char[] bar;

    while (1)
        bar = foo;
}

Using "\dmd\bin\dmd -I/dmd/src/phobos file.d -O" to compile, version
0.28.
Apr 27 2002
parent "Walter" <walter digitalmars.com> writes:
Thanks, I can reproduce the error. I'll see about a fix.
Apr 27 2002
prev sibling next sibling parent "Walter" <walter digitalmars.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:tg9jcu418udak1jp1g1d98g82fqjfk9or3 4ax.com...

 There's more.  array, memset, obj, and switcherr are not included in
 the distribution but symbols from them are used.


That's a mistake. I'll put them in.


  interface.c,


Obsolete, I'll remove it.


 intrinsic.d,


Just declarations. No need to compile it.


 iunknown.d,


Needs to be added to the library, I'll fix it.


 and time.d




Just declarations, no need to compile it.


 are not touched by the makefile.

 Finally, sc compiles the .asm files incorrectly (it adds a semicolon
 to the filename, bizarrely enough), and the version of masm and link I
 got don't like the options used.


Unfortunately, Microsoft keeps changing how MASM works. Older versions did
require the ;. It's a constant problem, and one reason why D has its own
inline assembler.


 It'd be nice if the obj file were
 just stuck in the distribution.


Good idea. I'll do it.
Apr 27 2002
prev sibling parent "Walter" <walter digitalmars.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:tg9jcu418udak1jp1g1d98g82fqjfk9or3 4ax.com...

 This requires "ERROR_NO_MORE_FILES = 18," in the appropriate place in
 windows.d and FileTimeToSystemTime using "export BOOL
 FileTimeToSystemTime(FILETIME* lpFileTime, SYSTEMTIME*
 lpSystemTime);".


Done.
Apr 27 2002