www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Indigo library 0.93

reply "Uwe Salomon" <post uwesalomon.de> writes:
Indigo is a project to provide a platform-independent non-GUI library that  
accomplishes some of the tasks of the Qt library for C++. Everything is  
designed to behave as equal to its Qt equivalent as possible, but also to  
take advantage of D specialties. Everything should be fast and simple.  
Currently it includes almost all Qt containers, a clean signals&slots  
implementation, I/O with IODevice, File and DataStream, and  
internationalisation basics with fast UTF conversion and message  
translation (including 2 helper programs imupdate and imrelease).

The new message translation module is very easy to use, and the helper  
program imupdate produces TS files like lupdate from Qt. Use linguist or  
KBabel to translate, and run imrelease (equivalent to lrelease)  
afterwards. Message translation is heavily optimized, and imupdate has a  
sophisticated algorithm to find similar older translations for new/changed  
strings. I would be happy to receive comments on the API and the programs.

You reach the docs under:
   http://www.uwesalomon.de/code/indigo/index.html
And the source under:
   http://www.uwesalomon.de/code/indigo/indigo.tar.gz

Ciao
uwe
Jul 10 2005
next sibling parent reply "Uwe Salomon" <post uwesalomon.de> writes:
Ah, and i forgot: The whole library is now LGPL-licensed. Thanks to  
Stewart for pointing this out to me. License is included as well, and the  
zip file now also contains the HTML docs.

Ciao
uwe
Jul 10 2005
parent reply license d.com writes:
In article <op.stpzurmn6yjbe6 sandmann.maerchenwald.net>, Uwe Salomon says...

Ah, and i forgot: The whole library is now LGPL-licensed. 


How about LGPL with exception allow static linking?

Like the wxWidgets Licence:

http://www.wxwidgets.org/newlicen.htm
http://www.wxwidgets.org/licence3.txt
Jul 18 2005
parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 How about LGPL with exception allow static linking?


Arrggh this licensing stuff eats more time than programming. :)  Well, of  
couse you can link the library to your program statically. There is no way  
beyond it anyways, as all the templates must be instantiated "freshly".  
I'll add a notice to the license.

Thanks & ciao
uwe
Jul 18 2005
parent "Uwe Salomon" <post uwesalomon.de> writes:
 Arrggh this licensing stuff eats more time than programming. :)  Well,  
 of couse you can link the library to your program statically. There is  
 no way beyond it anyways, as all the templates must be instantiated  
 "freshly". I'll add a notice to the license.


Well, i changed my mind. I will leave the license unchanged. There are  
enough options specified in section 6 of the LGPL that permit binary  
distributions of works using the library. If someone needs an exception  
for his commercial application, he can still contact me.

Ciao
uwe
Jul 18 2005
prev sibling next sibling parent reply =?utf-8?B?RGF3aWQgQ2nEmcW8YXJraWV3aWN6?= <arael asn.pl> writes:
On Mon, 11 Jul 2005 01:05:05 +0200, Uwe Salomon <post uwesalomon.de> wrote:

 You reach the docs under:
    http://www.uwesalomon.de/code/indigo/index.html
 And the source under:
    http://www.uwesalomon.de/code/indigo/indigo.tar.gz


Are there any screenshots available? I'm very interested in a matter of  
"look & feel" of indigo. Does it use (will use?) pure XServer / WindowsAPI  
or some frontend (gtk or smth).
-- 
Dawid Ciężarkiewicz
Jul 10 2005
parent =?utf-8?B?RGF3aWQgQ2nEmcW8YXJraWV3aWN6?= <arael asn.pl> writes:
On Mon, 11 Jul 2005 01:50:05 +0200, Dawid Ciężarkiewicz <arael asn.pl>  
wrote:


 On Mon, 11 Jul 2005 01:05:05 +0200, Uwe Salomon <post uwesalomon.de>  
 wrote:

 You reach the docs under:
    http://www.uwesalomon.de/code/indigo/index.html
 And the source under:
    http://www.uwesalomon.de/code/indigo/indigo.tar.gz


 Are there any screenshots available? I'm very interested in a matter of  
 "look & feel" of indigo. Does it use (will use?) pure XServer /  
 WindowsAPI or some frontend (gtk or smth).


Sorry. Now I see _non_ -GUI word. Late here - I need a sleep ... dreaming  
about the library with Qt frontend. :)
-- 
Dawid Ciężarkiewicz
Jul 10 2005
prev sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Uwe Salomon wrote:
<snip>

 You reach the docs under:
   http://www.uwesalomon.de/code/indigo/index.html
 And the source under:
   http://www.uwesalomon.de/code/indigo/indigo.tar.gz


I can compile the modules into a library using a home-made makefile. 
However, when I try to use it, the linker complains that the symbol 
_lstat is undefined.

Is there some undocumented library requirement?  Or has it just not been 
tested under Windows in a while?

Stewart.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS/M d- s:- a->--- UB  P+ L E  W++  N+++ o K- w++  O? M V? PS- PE- Y? 
PGP- t- 5? X? R b DI? D G e++>++++ h-- r-- !y
------END GEEK CODE BLOCK------

My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
Jul 14 2005
parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 I can compile the modules into a library using a home-made makefile.  
 However, when I try to use it, the linker complains that the symbol  
 _lstat is undefined.

 Is there some undocumented library requirement?  Or has it just not been  
 tested under Windows in a while?


The I/O modules have been tested under Windows as well, but obviously not  
after adding the lstat() function (needed by File.exists). I have fixed  
the error, and uploaded a new indigo.tar.gz.

Thanks for the report.

Ciao
uwe
Jul 14 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Uwe Salomon wrote:

 I can compile the modules into a library using a home-made makefile.  
 However, when I try to use it, the linker complains that the symbol 
 _lstat is undefined.

 Is there some undocumented library requirement?  Or has it just not 
 been tested under Windows in a while?


 
 The I/O modules have been tested under Windows as well, but obviously 
 not after adding the lstat() function (needed by File.exists). I have 
 fixed the error, and uploaded a new indigo.tar.gz.


I've now got it installed and running.  Just a few issues drop it short
of perfect.  For example, it's a step behind std.stream in that it only 
detects EOF after trying to read past the end.  My thought is that an 
I/O library should be able to detect EOF when it gets there, and that 
one should also be able to rely on exceptions to catch a premature end 
of file.

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/4085
http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/24322

This would be simple to implement for File.  It might take a bit of work 
for other IODevice types but it would be worth it IMO.

But I believe a TextStream is a feasible addition to this library, with 
the help of a few more IODevice members to help with UTF detection and 
the like.  I'll see what I can do....

Stewart.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS/M d- s:- a->--- UB  P+ L E  W++  N+++ o K- w++  O? M V? PS- PE- Y?
PGP- t- 5? X? R b DI? D G e++>++++ h-- r-- !y
------END GEEK CODE BLOCK------

My e-mail is valid but not my primary mailbox.  Please keep replies on
the 'group where everyone may benefit.
Jul 19 2005
parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 I've now got it installed and running.  Just a few issues drop it short
 of perfect.  For example, it's a step behind std.stream in that it only  
 detects EOF after trying to read past the end.


Hmm, i simply used feof() for determining the File end. If it does not  
support detecting EOF before actually hitting the end, it would have to be  
replaced by another function, which in turn would lead to the replacement  
of all the fileengine functions (fopen, fread, fwrite, etc.). I am not  
sure if this is worth this improvement? As is stated in the File  
documentation, i intentionally used the C streaming API because it is  
portable (look at the amount of code required to get stat() working on  
three platforms to get an idea of this advantage), and it already includes  
buffering and all that jazz which is needed for higher-level devices like  
DataStream and TextStream (which often read only a few bytes at a time).  
The alternative is to use the open, read, write family of functions, and  
implement buffering ourselves. Another 1000 lines of code (minimum).


 My thought is that an I/O library should be able to detect EOF when it  
 gets there, and that one should also be able to rely on exceptions to  
 catch a premature end of file.


Well, the IODevices are not meant to be used directly. I know, in my  
helper program imupdate and others i use them directly, but that is only  
because there is no TextStream yet. Look at DataStream, how it handles  
premature EOF. This is the interface the user will see: simple shifting of  
values into / out of the stream, and if something fails an exception is  
thrown. TextStream should be implemented in a similar manner.  
IODevice/File should only be used directly by the user if he really wants  
to, normally he will create a stream for every kind of IODevice (perhaps  
there will be a socket implementation in the future, or some kind of  
memory-based file).


 But I believe a TextStream is a feasible addition to this library, with  
 the help of a few more IODevice members to help with UTF detection and  
 the like.


UTF detection is also something that should be taken care of in a  
higher-level protocol, i.e. TextStream. IODevices encapsulate  
reading/writing raw data to/from an abstract kind of device, like a file,  
socket, memory. Interpretation of this data is done by DataStream (look  
into its implementation, it takes care of the number of bytes for every  
data type, the storing format used for arrays and the like, and the byte  
order of the machine) or TextStream.

By the way, i am currently writing Unicode character properties and  
message formatting. The character properties are already finished. I am  
not sure if they are needed for TextStream implementation (isSpace()  
perhaps?). Anyways, i uploaded the current Indigo 0.94 to my homepage,  
thus you can look at the changes.

Thanks
uwe
Jul 19 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Uwe Salomon wrote:

 I've now got it installed and running.  Just a few issues drop it short 
 of perfect.  For example, it's a step behind std.stream in that it 
 only detects EOF after trying to read past the end.


 
 Hmm, i simply used feof() for determining the File end. If it does not 
 support detecting EOF before actually hitting the end, it would have to 
 be  replaced by another function, which in turn would lead to the 
 replacement of all the fileengine functions (fopen, fread, fwrite, 
 etc.). I am not sure if this is worth this improvement?


Rewriting one function doesn't have to mean changing which API the whole 
library wraps.  Especially when this single rewrite doesn't rely on 
access to another API at all.

The solution for files is very simple: just test pos() == size().

For sequential streams such as stdin, process I/O streams and (?) 
sockets, it isn't quite so simple, but it isn't complicated either.  To 
test for EOF, read in the next byte.  If it returns EOF, return true. 
Otherwise, add it to the unread buffer or something.  (Is the unread 
buffer supposed to be internal?  Or are the wrappers in DataStream 
and/or TextStream waiting to be written?)

<snip>

 My thought is that an I/O library should be able to detect EOF when 
 it gets there, and that one should also be able to rely on exceptions 
 to catch a premature end of file.


 
 Well, the IODevices are not meant to be used directly.


But testing for EOF should certainly be part of the API that one is 
meant to use.  I refer you back to the concept of expected versus 
unexpected EOF.


 I know, in my helper program imupdate and others i use them directly, 
 but that is only because there is no TextStream yet. Look at 
 DataStream, how it handles premature EOF. This is the interface the 
 user will see: simple shifting of values into / out of the stream, 
 and if something fails an exception is thrown.


It didn't throw an exception when I tried it.  At least as far as an 
infinite loop of DataStream >> char is indeed infinite.  Has this changed?

<snip>

 UTF detection is also something that should be taken care of in a 
 higher-level protocol, i.e. TextStream. IODevices encapsulate 
 reading/writing raw data to/from an abstract kind of device, like a 
 file, socket, memory. Interpretation of this data is done by DataStream 
 (look into its implementation, it takes care of the number of bytes for 
 every data type, the storing format used for arrays and the like, and 
 the byte order of the machine) or TextStream.


The way encoding detection would work depends on the kind of IODevice. 
So you're saying TextStream should enumerate the possibilities and take 
appropriate action?

(For that matter, what is OS API support like for detecting what 
encoding the console is using, for stdin/out/err stuff?)


 By the way, i am currently writing Unicode character properties and 
 message formatting. The character properties are already finished. I am 
 not sure if they are needed for TextStream implementation (isSpace() 
 perhaps?). Anyways, i uploaded the current Indigo 0.94 to my homepage, 
 thus you can look at the changes.


I'll check it out.

Stewart.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS/M d- s:- a->--- UB  P+ L E  W++  N+++ o K- w++  O? M V? PS- PE- Y? 
PGP- t- 5? X? R b DI? D G e++>++++ h-- r-- !y
------END GEEK CODE BLOCK------

My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
Jul 19 2005
next sibling parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 I know, in my helper program imupdate and others i use them directly,  
 but that is only because there is no TextStream yet. Look at  
 DataStream, how it handles premature EOF. This is the interface the  
 user will see: simple shifting of values into / out of the stream, and  
 if something fails an exception is thrown.


 It didn't throw an exception when I tried it.  At least as far as an  
 infinite loop of DataStream >> char is indeed infinite.  Has this  
 changed?


Huh? I could not believe that, but i tried myself and it is indeed  
infinite. %)  It is funny that message catalogues are working perfect,  
despite their usage of DataStream. Always interesting how bugs can hide.  
Well, this is not intended behaviour, and i'll fix it as fast as possible.


 UTF detection is also something that should be taken care of in a  
 higher-level protocol, i.e. TextStream. IODevices encapsulate  
 reading/writing raw data to/from an abstract kind of device, like a  
 file, socket, memory. Interpretation of this data is done by DataStream  
 (look into its implementation, it takes care of the number of bytes for  
 every data type, the storing format used for arrays and the like, and  
 the byte order of the machine) or TextStream.


 The way encoding detection would work depends on the kind of IODevice.  
 So you're saying TextStream should enumerate the possibilities and take  
 appropriate action?

 (For that matter, what is OS API support like for detecting what  
 encoding the console is using, for stdin/out/err stuff?)


Hm, perhaps you're right. I thought the TextStream could read the first  
bytes and determine if they are a byte order mark. But that does only make  
sense if it is instantly plugged to a File. Thus the encoding information  
should be moved into the IODevice, yes. I'll take a look at that.

Ciao
uwe
Jul 19 2005
parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 UTF detection is also something that should be taken care of in a  
 higher-level protocol, i.e. TextStream. IODevices encapsulate  
 reading/writing raw data to/from an abstract kind of device, like a  
 file, socket, memory. Interpretation of this data is done by  
 DataStream (look into its implementation, it takes care of the number  
 of bytes for every data type, the storing format used for arrays and  
 the like, and the byte order of the machine) or TextStream.


 The way encoding detection would work depends on the kind of IODevice.  
 So you're saying TextStream should enumerate the possibilities and take  
 appropriate action?

 (For that matter, what is OS API support like for detecting what  
 encoding the console is using, for stdin/out/err stuff?)


 Hm, perhaps you're right. I thought the TextStream could read the first  
 bytes and determine if they are a byte order mark. But that does only  
 make sense if it is instantly plugged to a File. Thus the encoding  
 information should be moved into the IODevice, yes. I'll take a look at  
 that.


Well, in Qt the TextStream assumes that the device is in the local 8-bit  
encoding, but autodetects UTF encodings if the first thing it reads is a  
BOM. There are functions to turn autodetection off, and there are  
functions to set the encoding. Problems are:

(1) As you pointed out, this is not always the best solution. But it does  
its job, as terminals and stuff are local 8-bit encoded, and sockets  
should be set by the user anyways.

(2) We have no codecs currently. Just some (pretty fast) functions in  
indigo.i18n.conversion for the different UTF flavours. I wanted to add a  
toAscii function there. But codecs for more local 8-bits? Some chinese  
multibyte encodings??? This is a little bit too ambitious, i think.  
Anyways, the question remains if we provide the "Codec" base class, and  
some derived classes for the UTFs, ASCII and ISO-8859-1 to be prepared for  
later expansion.

By the way, should i talk with Brad about a forum at dsource? If you  
really plan to contribute to Indigo, this would be a little more  
convenient perhaps?

Ciao
uwe
Jul 20 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Uwe Salomon wrote:
<snip>

 Well, in Qt the TextStream assumes that the device is in the local 
 8-bit encoding, but autodetects UTF encodings if the first thing it 
 reads is a BOM.


I see.  But what about heuristic detection?

http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/7102

though it's practically useless on certain kinds of sequential streams. 
  Maybe throw in the heuristic fallback only if the stream is seekable?


 There are functions to turn autodetection off, and 
 there are functions to set the encoding. Problems are:
 
 (1) As you pointed out, this is not always the best solution. But it 
 does its job, as terminals and stuff are local 8-bit encoded, and 
 sockets should be set by the user anyways.


But how can we determine the local 8-bit encoding, in the cases where it 
isn't a constant of the platform?


 (2) We have no codecs currently. Just some (pretty fast) functions in  
 indigo.i18n.conversion for the different UTF flavours. I wanted to add 
 a toAscii function there. But codecs for more local 8-bits? Some 
 chinese multibyte encodings??? This is a little bit too ambitious, i 
 think.  Anyways, the question remains if we provide the "Codec" base 
 class, and some derived classes for the UTFs, ASCII and ISO-8859-1 to 
 be prepared for later expansion.


I think that would be a good idea.


 By the way, should i talk with Brad about a forum at dsource? If you  
 really plan to contribute to Indigo, this would be a little more  
 convenient perhaps?


The only trouble is that for some strange reason the dsource svn server 
seems to be incompatible with my Internet connection.

Stewart.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS/M d- s:- a->--- UB  P+ L E  W++  N+++ o K- w++  O? M V? PS- PE- Y? 
PGP- t- 5? X? R b DI? D G e++>++++ h-- r-- !y
------END GEEK CODE BLOCK------

My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
Jul 20 2005
parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 Well, in Qt the TextStream assumes that the device is in the local  
 8-bit encoding, but autodetects UTF encodings if the first thing it  
 reads is a BOM.


 I see.  But what about heuristic detection?


Nice thing. We'll add that later if we feel like it. Maybe as a member  
function of the codecs, for example like this:

int Codec.suitability(void[] someData);

The higher the return value, the more appropiate the codec is for the  
contents of someData. Then we register all existing codecs in a list, and  
test them. But only if the user requests that (IODevice.determineEncoding  
or similar).

And you are right, i think the best solution is to code this into the  
IODevice, i.e. the different devices have functions for receiving their  
encoding, and TextStream only calls them after creation. The IODevice base  
class implements common functionality like this heuristic or BOM detection.


 But how can we determine the local 8-bit encoding, in the cases where it  
 isn't a constant of the platform?


Huh, i'm not even sure how to detect the local 8 bit on the different  
platforms. In Linux its an appendix to the LANG environment variable, and  
Windows will have some function, i guess. Mac OS too. And if there's  
nothing, we fall back to ASCII :)


 (2) We have no codecs currently. Just some (pretty fast) functions in   
 indigo.i18n.conversion for the different UTF flavours. I wanted to add  
 a toAscii function there. But codecs for more local 8-bits? Some  
 chinese multibyte encodings??? This is a little bit too ambitious, i  
 think.  Anyways, the question remains if we provide the "Codec" base  
 class, and some derived classes for the UTFs, ASCII and ISO-8859-1 to  
 be prepared for later expansion.


 I think that would be a good idea.


All of them should be easy to write, cause the functions already exist or  
are trivial (ASCII and ISO-8859-1). Well, the class should be "TextCodec",  
and it should work roughly like  
http://doc.trolltech.com/4.0/qtextcodec.html


 By the way, should i talk with Brad about a forum at dsource? If you   
 really plan to contribute to Indigo, this would be a little more   
 convenient perhaps?


 The only trouble is that for some strange reason the dsource svn server  
 seems to be incompatible with my Internet connection.


I thought of it mostly because of the chattering here in the announce NG...

Ciao
uwe
Jul 20 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Uwe Salomon wrote:

 Well, in Qt the TextStream assumes that the device is in the local  
 8-bit encoding, but autodetects UTF encodings if the first thing it  
 reads is a BOM.


 I see.  But what about heuristic detection?


 
 Nice thing. We'll add that later if we feel like it. Maybe as a member  
 function of the codecs, for example like this:
 
 int Codec.suitability(void[] someData);


Good idea.


 The higher the return value, the more appropiate the codec is for the  
 contents of someData. Then we register all existing codecs in a list, 
 and  test them. But only if the user requests that 
 (IODevice.determineEncoding  or similar).
 
 And you are right, i think the best solution is to code this into the  
 IODevice, i.e. the different devices have functions for receiving their  
 encoding, and TextStream only calls them after creation. The IODevice 
 base  class implements common functionality like this heuristic or BOM 
 detection.


Not sure.  Now that I come to think about it, the only thing that's
really dependent on the kind of IODevice is whether it makes sense to
apply heuristics.  And thinking about it now, if we're going to rely on 
the class user to request heuristic detection anyway, I guess we can go 
with the first quoted paragraph and don't really need to add this stuff 
to IODevice.


 But how can we determine the local 8-bit encoding, in the cases where 
 it  isn't a constant of the platform?


 
 Huh, i'm not even sure how to detect the local 8 bit on the different  
 platforms. In Linux its an appendix to the LANG environment variable, 
 and  Windows will have some function, i guess. Mac OS too. And if 
 there's  nothing, we fall back to ASCII :)


<snip>

Here's how I'm thinking of implementing it now.

The only thing that needs to be added to IODevice is a read-only codec 
property.  This would return null by default, and implement the
platform-dependent logic to detect the local codec for stdin/out/err.
TextStream would retrieve this from the IODevice on construction, or on
setting the device if no codec is already set.

When the time comes to read some data from the TextStream, if the codec
is null then it'll do the BOM detection, and if no BOM is present, fall
back to UTF-8.  We could also have a detectBOM method, which will look
for a BOM at the current point and set the codec as appropriate if one
is present, otherwise leave the codec unchanged.  This'll enable stuff like

     stream.codec = ISO_8859_1;  /* or whatever naming convention we
                                    decide on/copy */
     stream.detectBOM();

meaning "if there's a BOM, honour it, otherwise treat it as ISO-8859-1". 
  Except that QTextStream has an autoDetectUnicode property - where 
would this fit into the equation?

I'll try and get somewhere with coding up TextStream over the weekend.
Probably supporting only UTF-8 at first, and then improving it to
support codecs once these are implemented.

Stewart.

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS/M d- s:- a->--- UB  P+ L E  W++  N+++ o K- w++  O? M V? PS- PE- Y?
PGP- t- 5? X? R b DI? D G e++>++++ h-- r-- !y
------END GEEK CODE BLOCK------

My e-mail is valid but not my primary mailbox.  Please keep replies on
the 'group where everyone may benefit.
Jul 22 2005
parent "Uwe Salomon" <post uwesalomon.de> writes:
We have a forum now :)  thanks to Brad. Just go to

http://www.dsource.org/forums/viewforum.php?f=67

There are the posts we've made so far, and my answer to your post.

Ciao
uwe
Jul 22 2005
prev sibling parent "Uwe Salomon" <post uwesalomon.de> writes:
 It didn't throw an exception when I tried it.  At least as far as an  
 infinite loop of DataStream >> char is indeed infinite.  Has this  
 changed?


This has been fixed, and i uploaded the changed version. Interestingly, it  
only happened for 1-byte data types: char, byte, ubyte.

Thanks
uwe
Jul 19 2005