www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - SQLite 3.6.23.1 wrapper + connector

reply awishformore <awishformore gmail.com> writes:
Hello there.

I've converted the .h file of the latest SQLite version to .d and I 
thought I'd let the world know (as suggested on IRC ;). Maybe it will 
save someone some work.

I've also written a nice connector class that wraps all the C functions 
for convenient use in your application. It's very basic, but I will 
probably add more features (like automatic preparation of statements and 
automatic caching of several prepared statements) later.

For the time being both files are included in the download: 
http://nexunity.com/sqlite4d.rar

I'm pretty new to this kind of stuff, so what I did to get it to work 
was compiling the latest SQLite dll from source with dmc and then link 
the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

I'm sure there is a better way like compiling it as static lib (dmc 
complained about no entry point) or having some kind of other file to 
link into your app in order for it to compile and then use the dll. I 
however couldn't figure it out and it works for me. Don't hesitate to 
teach me nonetheless.

Any kind of feedback is always appreciated.

Greetings, Max.
Jul 18 2010
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from awishformore (awishformore gmail.com)'s article

 Hello there.
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.
 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar
 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).
 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.
 Any kind of feedback is always appreciated.
 Greetings, Max.


Awesome.  D1, D2 or both?  I've wanted a simple database for some time now, but
been too lazy to write bindings or find one one myself, which encourages me to
roll my own ad-hoc formats instead.  Given that sqlite is in the public domain,
maybe Phobos should eventually include SQLite + a nice D-ish wrapper for it, so
that people can use it w/o creating dependency hell in their projects.
Jul 18 2010
next sibling parent reply BLS <windevguy hotmail.de> writes:
On 19/07/2010 00:57, dsimcha wrote:

   Given that sqlite is in the public domain,
 maybe Phobos should eventually include SQLite + a nice D-ish wrapper for it, so
 that people can use it w/o creating dependency hell in their projects.


Agreed, even a very popular and commercial multi OS RAD Tool called 
"Real Basic" comes bundled with SQLite.  Let's use this excellent tool 
instead of the thick openrj stuff, we had before.
-bjoern
Jul 18 2010
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/18/2010 06:22 PM, BLS wrote:

 On 19/07/2010 00:57, dsimcha wrote:

 Given that sqlite is in the public domain,
 maybe Phobos should eventually include SQLite + a nice D-ish wrapper
 for it, so
 that people can use it w/o creating dependency hell in their projects.


 Agreed, even a very popular and commercial multi OS RAD Tool called
 "Real Basic" comes bundled with SQLite. Let's use this excellent tool
 instead of the thick openrj stuff, we had before.
 -bjoern


I thought Qt does, too.

Python definitely does, and I've actually been using it a lot lately. 
One thing I noticed when switching my code from postgresql to sqlite is 
that their respective python libraries use different parameter styles. 
Otherwise, it was completely painless.

I'd love to see sqlite distributed with D, but not before a good DB API 
specification (not to disparage awishformore's work, of course).
Jul 18 2010
prev sibling next sibling parent awishformore <awishformore gmail.com> writes:
On 19/07/2010 00:57, dsimcha wrote:

 == Quote from awishformore (awishformore gmail.com)'s article

 Hello there.
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.
 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar
 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).
 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.
 Any kind of feedback is always appreciated.
 Greetings, Max.


 Awesome.  D1, D2 or both?  I've wanted a simple database for some time now, but
 been too lazy to write bindings or find one one myself, which encourages me to
 roll my own ad-hoc formats instead.  Given that sqlite is in the public domain,
 maybe Phobos should eventually include SQLite + a nice D-ish wrapper for it, so
 that people can use it w/o creating dependency hell in their projects.


Hello.

Ah yeah, I guess I should have mentioned. I'm only working with D2 at 
the moment, so it is D2 right now (because of the use of string). 
However, it should be straight forward to replace all string occurrences 
with char[] to get it to work on D1. I'm pretty sure that's the only 
issue stopping it from working on D1.

Also, when using my wrapper, please note a few specifics with the bind 
method:

- to bind a zeroblob to a prepared statement, pass an int array with a 
single element with the size of the blob as value. Couldn't come up with 
a better way to make that function available through the same method.
- to bind NULL just pass null ;)
- currently, it lets you pass anything and will try to treat it as 
sqlite3_value struct reference, so be careful

Greetings, Max.
Jul 18 2010
prev sibling parent Charles Hixson <charleshixsn earthlink.net> writes:
On 07/18/2010 03:57 PM, dsimcha wrote:

 == Quote from awishformore (awishformore gmail.com)'s article

 Hello there.
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.


...  for me. Don't hesitate to

 teach me nonetheless.
 Any kind of feedback is always appreciated.
 Greetings, Max.


 Awesome.  D1, D2 or both?  I've wanted a simple database for some time now, but
 been too lazy to write bindings or find one one myself, which encourages me to
 roll my own ad-hoc formats instead.  Given that sqlite is in the public domain,
 maybe Phobos should eventually include SQLite + a nice D-ish wrapper for it, so
 that people can use it w/o creating dependency hell in their projects.


Personally, I think it would be wonderful if Phobos included an SQLite 
class, or even just a B+Tree class.  (Often I don't want the entire 
weight of SQL, and just use it because it's so difficult to get access 
to a B+Tree.)
Jul 19 2010
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/18/2010 05:44 PM, awishformore wrote:

 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar


btw, would it be possible to distribute as something other than a rar? I 
wanted to have a peek at what you've done, but apparently nothing on my 
[linux] system can read that file.
Jul 18 2010
parent reply Jonathan M Davis <jmdavisprog gmail.com> writes:
On Sunday 18 July 2010 19:10:35 Ellery Newcomer wrote:

 On 07/18/2010 05:44 PM, awishformore wrote:

 Hello there.
 
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
 
 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.
 
 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar


 
 btw, would it be possible to distribute as something other than a rar? I
 wanted to have a peek at what you've done, but apparently nothing on my
 [linux] system can read that file.


While I hate rar, every distribution than I have used has had unrar. There's a 
decent chance that it isn't installed by default though.

- Jonathan M Davis
Jul 18 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/18/2010 09:20 PM, Jonathan M Davis wrote:

 On Sunday 18 July 2010 19:10:35 Ellery Newcomer wrote:

 btw, would it be possible to distribute as something other than a rar? I
 wanted to have a peek at what you've done, but apparently nothing on my
 [linux] system can read that file.


 While I hate rar, every distribution than I have used has had unrar. There's a
 decent chance that it isn't installed by default though.

 - Jonathan M Davis


Hmm. I suppose I could have spent more than 2 seconds trying to open it. 
You're right on both counts.

As for the code, I'm looking at SQLiteConnector.ptrtostr. Does reserve 
really work that way? Just curious - I wouldn't know one way or the 
other, although I would probably use something more like

string ptrtostr(char* ptr, int size = 0){
  return ptr ? ptr[0 .. size].idup : null;
}
Jul 18 2010
parent reply awishformore <awishformore gmail.com> writes:
On 19/07/2010 04:59, Ellery Newcomer wrote:

 On 07/18/2010 09:20 PM, Jonathan M Davis wrote:

 On Sunday 18 July 2010 19:10:35 Ellery Newcomer wrote:

 btw, would it be possible to distribute as something other than a rar? I
 wanted to have a peek at what you've done, but apparently nothing on my
 [linux] system can read that file.


 While I hate rar, every distribution than I have used has had unrar.
 There's a
 decent chance that it isn't installed by default though.

 - Jonathan M Davis


 Hmm. I suppose I could have spent more than 2 seconds trying to open it.
 You're right on both counts.

 As for the code, I'm looking at SQLiteConnector.ptrtostr. Does reserve
 really work that way? Just curious - I wouldn't know one way or the
 other, although I would probably use something more like

 string ptrtostr(char* ptr, int size = 0){
 return ptr ? ptr[0 .. size].idup : null;
 }


I think an empty array/string is safer to return than null. And yes, 
reserve works that way, it will reserve the space needed for the 
(character) array to expand.

The strings are null terminated in SQLite and you won't always know the 
size; I had previously used the same method in another place of my app. 
I however also wasn't aware that you could still access pointers like 
arrays in D, so that would indeed be a better way to do it when you 
always know the size.

You could then even go as far as not iduping, but instead casting it to 
a string to avoid unnecessary allocations.

Greetings, Max.
Jul 18 2010
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/18/2010 11:20 PM, awishformore wrote:

 I think an empty array/string is safer to return than null.


It doesn't matter. D treats 'null' arrays exactly the same as empty 
arrays. Just think of a 'null' array as {length=0, ptr=null}
Jul 18 2010
prev sibling next sibling parent reply Johannes Pfau <spam example.com> writes:
On 19.07.2010 00:44, awishformore wrote:

 Hello there.
 
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
 
 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.
 
 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar
 
 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).
 
 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.
 
 Any kind of feedback is always appreciated.
 
 Greetings, Max.


Nice work!

I'm by no means an expert in d, but I wonder whether the following in
the connector could cause troubles:
--------------------------------------
int open(string db_name)
{
    return sqlite3_open_v2(cast(char*)db_name,
--------------------------------------

I always thought just casting to the pointer type isn't really safe. I
guess you should use std.strings toStringz.

(Off topic: std.strings toStringz uses the GC to store the c string. Now
if I passed the returned pointer to a C function, but didn't store a
reference in the D program, couldn't that c string get collected while
the c library would expect it to be still there?)
-- 
Johannes Pfau
Jul 19 2010
parent reply awishformore <awishformore gmail.com> writes:
Am 19.07.2010 17:50, schrieb Johannes Pfau:

 On 19.07.2010 00:44, awishformore wrote:

 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.


 Nice work!

 I'm by no means an expert in d, but I wonder whether the following in
 the connector could cause troubles:
 --------------------------------------
 int open(string db_name)
 {
      return sqlite3_open_v2(cast(char*)db_name,
 --------------------------------------

 I always thought just casting to the pointer type isn't really safe. I
 guess you should use std.strings toStringz.

 (Off topic: std.strings toStringz uses the GC to store the c string. Now
 if I passed the returned pointer to a C function, but didn't store a
 reference in the D program, couldn't that c string get collected while
 the c library would expect it to be still there?)


Hello there.

I'm not an expert for anything related to programming, but I'm pretty 
sure the string/char[]/char* isn't needed by SQLite after the function 
returns, as you will always be using the sqlite4 object pointer.

As for the issue about null-terminated C strings, I was initially 
expecting this to not work, but tried anyway. I guess this has to do 
with the fact that SQLite interprets the char* as UTF-8, which is what D 
uses. With only a few lines of code, UTF-16 would also work with the 
wrapper by the way, but I won't ever use it, so I didn't bother.

I added quite a few unit tests to the file in order to verify stuff like 
this and all of them seem to pass without the slightest issue and there 
are no memory leaks either, so I'm assuming that at least the basic 
functionality of my connector class is absolutely safe to use.

The only thing I changed except for the unit tests is the 
ptrtostr/ptrtoblob to access the pointer like an array instead of a 
while loop and the else -> else static if(T==sqlite3_value*) to ensure a 
valid parameter is always passed.

Greetings, Max.
Jul 19 2010
parent reply "Rory McGuire" <rmcguire neonova.co.za> writes:
On Mon, 19 Jul 2010 20:39:35 +0200, awishformore <awishformore gmail.com>  
wrote:


 Am 19.07.2010 17:50, schrieb Johannes Pfau:

 On 19.07.2010 00:44, awishformore wrote:

 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements  
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.


 Nice work!

 I'm by no means an expert in d, but I wonder whether the following in
 the connector could cause troubles:
 --------------------------------------
 int open(string db_name)
 {
      return sqlite3_open_v2(cast(char*)db_name,
 --------------------------------------

 I always thought just casting to the pointer type isn't really safe. I
 guess you should use std.strings toStringz.

 (Off topic: std.strings toStringz uses the GC to store the c string. Now
 if I passed the returned pointer to a C function, but didn't store a
 reference in the D program, couldn't that c string get collected while
 the c library would expect it to be still there?)


 Hello there.

 I'm not an expert for anything related to programming, but I'm pretty  
 sure the string/char[]/char* isn't needed by SQLite after the function  
 returns, as you will always be using the sqlite4 object pointer.

 As for the issue about null-terminated C strings, I was initially  
 expecting this to not work, but tried anyway. I guess this has to do  
 with the fact that SQLite interprets the char* as UTF-8, which is what D  
 uses. With only a few lines of code, UTF-16 would also work with the  
 wrapper by the way, but I won't ever use it, so I didn't bother.

 I added quite a few unit tests to the file in order to verify stuff like  
 this and all of them seem to pass without the slightest issue and there  
 are no memory leaks either, so I'm assuming that at least the basic  
 functionality of my connector class is absolutely safe to use.

 The only thing I changed except for the unit tests is the  
 ptrtostr/ptrtoblob to access the pointer like an array instead of a  
 while loop and the else -> else static if(T==sqlite3_value*) to ensure a  
 valid parameter is always passed.

 Greetings, Max.


Did you test with a string that was not in the code itself, e.g. from a  
config file?
String literals are null terminated so you wouldn't have had an issue if  
all your strings were literals.
Utf8 doesn't contain the string length, so you will run in to problems  
eventually.

You have to use toStringz or your own null terminator. Unless of course  
you know that the function will always be
taking string literals. But even then leaving something like that up to  
the programmer to remember is not exactly
fool proof.

Enjoy.
~Rory
Jul 19 2010
parent reply awishformore <awishformore gmail.com> writes:
Am 19.07.2010 21:34, schrieb Rory McGuire:

 On Mon, 19 Jul 2010 20:39:35 +0200, awishformore
 <awishformore gmail.com> wrote:


 Am 19.07.2010 17:50, schrieb Johannes Pfau:

 On 19.07.2010 00:44, awishformore wrote:

 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.


 Nice work!

 I'm by no means an expert in d, but I wonder whether the following in
 the connector could cause troubles:
 --------------------------------------
 int open(string db_name)
 {
 return sqlite3_open_v2(cast(char*)db_name,
 --------------------------------------

 I always thought just casting to the pointer type isn't really safe. I
 guess you should use std.strings toStringz.

 (Off topic: std.strings toStringz uses the GC to store the c string. Now
 if I passed the returned pointer to a C function, but didn't store a
 reference in the D program, couldn't that c string get collected while
 the c library would expect it to be still there?)


 Hello there.

 I'm not an expert for anything related to programming, but I'm pretty
 sure the string/char[]/char* isn't needed by SQLite after the function
 returns, as you will always be using the sqlite4 object pointer.

 As for the issue about null-terminated C strings, I was initially
 expecting this to not work, but tried anyway. I guess this has to do
 with the fact that SQLite interprets the char* as UTF-8, which is what
 D uses. With only a few lines of code, UTF-16 would also work with the
 wrapper by the way, but I won't ever use it, so I didn't bother.

 I added quite a few unit tests to the file in order to verify stuff
 like this and all of them seem to pass without the slightest issue and
 there are no memory leaks either, so I'm assuming that at least the
 basic functionality of my connector class is absolutely safe to use.

 The only thing I changed except for the unit tests is the
 ptrtostr/ptrtoblob to access the pointer like an array instead of a
 while loop and the else -> else static if(T==sqlite3_value*) to ensure
 a valid parameter is always passed.

 Greetings, Max.


 Did you test with a string that was not in the code itself, e.g. from a
 config file?
 String literals are null terminated so you wouldn't have had an issue if
 all your strings were literals.
 Utf8 doesn't contain the string length, so you will run in to problems
 eventually.

 You have to use toStringz or your own null terminator. Unless of course
 you know that the function will always be
 taking string literals. But even then leaving something like that up to
 the programmer to remember is not exactly
 fool proof.

 Enjoy.
 ~Rory


Hey again and thanks for the hint. I tried finding something on the DM 
page about string literals being null terminated and while the section 
about string literals didn't even mention it, it was said some place else.

That explains why using string literals works even though I expected it 
to fail. It's indeed good to know and adding std.string.toStringz is 
probably a good idea ;). Thanks.

Greetings, Max.
Jul 19 2010
parent reply "Rory McGuire" <rmcguire neonova.co.za> writes:
On Tue, 20 Jul 2010 01:37:10 +0200, awishformore <awishformore gmail.com>  
wrote:


 Am 19.07.2010 21:34, schrieb Rory McGuire:

 On Mon, 19 Jul 2010 20:39:35 +0200, awishformore
 <awishformore gmail.com> wrote:


 Am 19.07.2010 17:50, schrieb Johannes Pfau:

 On 19.07.2010 00:44, awishformore wrote:

 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C  
 functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then  
 link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.


 Nice work!

 I'm by no means an expert in d, but I wonder whether the following in
 the connector could cause troubles:
 --------------------------------------
 int open(string db_name)
 {
 return sqlite3_open_v2(cast(char*)db_name,
 --------------------------------------

 I always thought just casting to the pointer type isn't really safe. I
 guess you should use std.strings toStringz.

 (Off topic: std.strings toStringz uses the GC to store the c string.  
 Now
 if I passed the returned pointer to a C function, but didn't store a
 reference in the D program, couldn't that c string get collected while
 the c library would expect it to be still there?)


 Hello there.

 I'm not an expert for anything related to programming, but I'm pretty
 sure the string/char[]/char* isn't needed by SQLite after the function
 returns, as you will always be using the sqlite4 object pointer.

 As for the issue about null-terminated C strings, I was initially
 expecting this to not work, but tried anyway. I guess this has to do
 with the fact that SQLite interprets the char* as UTF-8, which is what
 D uses. With only a few lines of code, UTF-16 would also work with the
 wrapper by the way, but I won't ever use it, so I didn't bother.

 I added quite a few unit tests to the file in order to verify stuff
 like this and all of them seem to pass without the slightest issue and
 there are no memory leaks either, so I'm assuming that at least the
 basic functionality of my connector class is absolutely safe to use.

 The only thing I changed except for the unit tests is the
 ptrtostr/ptrtoblob to access the pointer like an array instead of a
 while loop and the else -> else static if(T==sqlite3_value*) to ensure
 a valid parameter is always passed.

 Greetings, Max.


 Did you test with a string that was not in the code itself, e.g. from a
 config file?
 String literals are null terminated so you wouldn't have had an issue if
 all your strings were literals.
 Utf8 doesn't contain the string length, so you will run in to problems
 eventually.

 You have to use toStringz or your own null terminator. Unless of course
 you know that the function will always be
 taking string literals. But even then leaving something like that up to
 the programmer to remember is not exactly
 fool proof.

 Enjoy.
 ~Rory


 Hey again and thanks for the hint. I tried finding something on the DM  
 page about string literals being null terminated and while the section  
 about string literals didn't even mention it, it was said some place  
 else.

 That explains why using string literals works even though I expected it  
 to fail. It's indeed good to know and adding std.string.toStringz is  
 probably a good idea ;). Thanks.

 Greetings, Max.


sure, I must admit it is annoying when the same code can do different  
things just because of where the data came
from. It would be easier to notice the bug if d never added a null on  
literals, but then there would also be a lot more
usages of toStringz.

I think if you want to test it you can do:
auto s = "blah";
open(s[0..$].dup.ptr); // duplicating it should put it somewhere else
			// just slicing will not test
Jul 19 2010
parent reply awishformore <awishformore gmail.com> writes:
Am 20.07.2010 08:56, schrieb Rory McGuire:

 On Tue, 20 Jul 2010 01:37:10 +0200, awishformore
 <awishformore gmail.com> wrote:


 Am 19.07.2010 21:34, schrieb Rory McGuire:

 On Mon, 19 Jul 2010 20:39:35 +0200, awishformore
 <awishformore gmail.com> wrote:


 Am 19.07.2010 17:50, schrieb Johannes Pfau:

 On 19.07.2010 00:44, awishformore wrote:

 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C
 functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then
 link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.


 Nice work!

 I'm by no means an expert in d, but I wonder whether the following in
 the connector could cause troubles:
 --------------------------------------
 int open(string db_name)
 {
 return sqlite3_open_v2(cast(char*)db_name,
 --------------------------------------

 I always thought just casting to the pointer type isn't really safe. I
 guess you should use std.strings toStringz.

 (Off topic: std.strings toStringz uses the GC to store the c
 string. Now
 if I passed the returned pointer to a C function, but didn't store a
 reference in the D program, couldn't that c string get collected while
 the c library would expect it to be still there?)


 Hello there.

 I'm not an expert for anything related to programming, but I'm pretty
 sure the string/char[]/char* isn't needed by SQLite after the function
 returns, as you will always be using the sqlite4 object pointer.

 As for the issue about null-terminated C strings, I was initially
 expecting this to not work, but tried anyway. I guess this has to do
 with the fact that SQLite interprets the char* as UTF-8, which is what
 D uses. With only a few lines of code, UTF-16 would also work with the
 wrapper by the way, but I won't ever use it, so I didn't bother.

 I added quite a few unit tests to the file in order to verify stuff
 like this and all of them seem to pass without the slightest issue and
 there are no memory leaks either, so I'm assuming that at least the
 basic functionality of my connector class is absolutely safe to use.

 The only thing I changed except for the unit tests is the
 ptrtostr/ptrtoblob to access the pointer like an array instead of a
 while loop and the else -> else static if(T==sqlite3_value*) to ensure
 a valid parameter is always passed.

 Greetings, Max.


 Did you test with a string that was not in the code itself, e.g. from a
 config file?
 String literals are null terminated so you wouldn't have had an issue if
 all your strings were literals.
 Utf8 doesn't contain the string length, so you will run in to problems
 eventually.

 You have to use toStringz or your own null terminator. Unless of course
 you know that the function will always be
 taking string literals. But even then leaving something like that up to
 the programmer to remember is not exactly
 fool proof.

 Enjoy.
 ~Rory


 Hey again and thanks for the hint. I tried finding something on the DM
 page about string literals being null terminated and while the section
 about string literals didn't even mention it, it was said some place
 else.

 That explains why using string literals works even though I expected
 it to fail. It's indeed good to know and adding std.string.toStringz
 is probably a good idea ;). Thanks.

 Greetings, Max.


 sure, I must admit it is annoying when the same code can do different
 things just because of where the data came
 from. It would be easier to notice the bug if d never added a null on
 literals, but then there would also be a lot more
 usages of toStringz.

 I think if you want to test it you can do:
 auto s = "blah";
 open(s[0..$].dup.ptr); // duplicating it should put it somewhere else
 // just slicing will not test


When thinking about it, it makes sense to have string literals null 
terminated in order to have C functions work with them. However, I 
wonder about some stuff, for instance:

string s = "string";
// is s == "string\0" now?
char[] c = cast(char[])s;
// is c[6] == '\0' now?
char* p = s.ptr;
// is *(p+6) == '\0' now?

I think use of the zero terminator should be consistent. Either make 
every string (and char[] for that matter) zero terminated in the 
underlying memory for backwards compatibility with C or leave it to the 
user in all cases.

/Max
Jul 20 2010
next sibling parent "Rory McGuire" <rmcguire neonova.co.za> writes:
On Tue, 20 Jul 2010 14:23:07 +0200, awishformore <awishformore gmail.com>  
wrote:


 Am 20.07.2010 08:56, schrieb Rory McGuire:

 On Tue, 20 Jul 2010 01:37:10 +0200, awishformore
 <awishformore gmail.com> wrote:


 Am 19.07.2010 21:34, schrieb Rory McGuire:

 On Mon, 19 Jul 2010 20:39:35 +0200, awishformore
 <awishformore gmail.com> wrote:


 Am 19.07.2010 17:50, schrieb Johannes Pfau:

 On 19.07.2010 00:44, awishformore wrote:

 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it  
 will
 save someone some work.

 I've also written a nice connector class that wraps all the C
 functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of  
 statements
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to  
 work
 was compiling the latest SQLite dll from source with dmc and then
 link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file  
 to
 link into your app in order for it to compile and then use the  
 dll. I
 however couldn't figure it out and it works for me. Don't hesitate  
 to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.


 Nice work!

 I'm by no means an expert in d, but I wonder whether the following  
 in
 the connector could cause troubles:
 --------------------------------------
 int open(string db_name)
 {
 return sqlite3_open_v2(cast(char*)db_name,
 --------------------------------------

 I always thought just casting to the pointer type isn't really  
 safe. I
 guess you should use std.strings toStringz.

 (Off topic: std.strings toStringz uses the GC to store the c
 string. Now
 if I passed the returned pointer to a C function, but didn't store a
 reference in the D program, couldn't that c string get collected  
 while
 the c library would expect it to be still there?)


 Hello there.

 I'm not an expert for anything related to programming, but I'm pretty
 sure the string/char[]/char* isn't needed by SQLite after the  
 function
 returns, as you will always be using the sqlite4 object pointer.

 As for the issue about null-terminated C strings, I was initially
 expecting this to not work, but tried anyway. I guess this has to do
 with the fact that SQLite interprets the char* as UTF-8, which is  
 what
 D uses. With only a few lines of code, UTF-16 would also work with  
 the
 wrapper by the way, but I won't ever use it, so I didn't bother.

 I added quite a few unit tests to the file in order to verify stuff
 like this and all of them seem to pass without the slightest issue  
 and
 there are no memory leaks either, so I'm assuming that at least the
 basic functionality of my connector class is absolutely safe to use.

 The only thing I changed except for the unit tests is the
 ptrtostr/ptrtoblob to access the pointer like an array instead of a
 while loop and the else -> else static if(T==sqlite3_value*) to  
 ensure
 a valid parameter is always passed.

 Greetings, Max.


 Did you test with a string that was not in the code itself, e.g. from  
 a
 config file?
 String literals are null terminated so you wouldn't have had an issue  
 if
 all your strings were literals.
 Utf8 doesn't contain the string length, so you will run in to problems
 eventually.

 You have to use toStringz or your own null terminator. Unless of  
 course
 you know that the function will always be
 taking string literals. But even then leaving something like that up  
 to
 the programmer to remember is not exactly
 fool proof.

 Enjoy.
 ~Rory


 Hey again and thanks for the hint. I tried finding something on the DM
 page about string literals being null terminated and while the section
 about string literals didn't even mention it, it was said some place
 else.

 That explains why using string literals works even though I expected
 it to fail. It's indeed good to know and adding std.string.toStringz
 is probably a good idea ;). Thanks.

 Greetings, Max.


 sure, I must admit it is annoying when the same code can do different
 things just because of where the data came
 from. It would be easier to notice the bug if d never added a null on
 literals, but then there would also be a lot more
 usages of toStringz.

 I think if you want to test it you can do:
 auto s = "blah";
 open(s[0..$].dup.ptr); // duplicating it should put it somewhere else
 // just slicing will not test


 When thinking about it, it makes sense to have string literals null  
 terminated in order to have C functions work with them. However, I  
 wonder about some stuff, for instance:

 string s = "string";
 // is s == "string\0" now?
 char[] c = cast(char[])s;
 // is c[6] == '\0' now?
 char* p = s.ptr;
 // is *(p+6) == '\0' now?

 I think use of the zero terminator should be consistent. Either make  
 every string (and char[] for that matter) zero terminated in the  
 underlying memory for backwards compatibility with C or leave it to the  
 user in all cases.

 /Max


perhaps the NULL is there because its there in the executable file?
NULL is also often after a dynamic array simply because of d always  
initializing memory, and
when you get an allocation often a larger amount is allocated which  
remains NULL.
Jul 20 2010
prev sibling parent Pelle <pelle.mansson gmail.com> writes:
On 07/20/2010 02:23 PM, awishformore wrote:

 When thinking about it, it makes sense to have string literals null
 terminated in order to have C functions work with them. However, I
 wonder about some stuff, for instance:

 string s = "string";
 // is s == "string\0" now?


No, but the byte after the string (in the executable) is 0.


 char[] c = cast(char[])s;
 // is c[6] == '\0' now?


No, c.length == 6.


 char* p = s.ptr;
 // is *(p+6) == '\0' now?


Yes. Since it came from a literal.


 I think use of the zero terminator should be consistent. Either make
 every string (and char[] for that matter) zero terminated in the
 underlying memory for backwards compatibility with C or leave it to the
 user in all cases.

 /Max


The current situation allows for this:
printf("Hello!\n");

but not this:

string s = "Hello!\n";
printf(s);

Since the compiler knows about literals being null terminated, but not 
just any string. I think it's a good compromise.
Jul 22 2010
prev sibling next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
awishformore Wrote:


 Ah yeah, I guess I should have mentioned. I'm only working with D2 at 
 the moment, so it is D2 right now (because of the use of string). 
 However, it should be straight forward to replace all string occurrences 
 with char[] to get it to work on D1. I'm pretty sure that's the only 
 issue stopping it from working on D1.


D1 has 'string' defined as char[] so there is no need to replace it, and it
should not be replaced.
Jul 21 2010
prev sibling next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
awishformore Wrote:


 You could then even go as far as not iduping, but instead casting it to 
 a string to avoid unnecessary allocations.


In D2 you would use std.contracts.assumeUnique() for this.

http://digitalmars.com/d/2.0/phobos/std_contracts.html#assumeUnique
Jul 21 2010
prev sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Johannes Pfau Wrote:


 I always thought just casting to the pointer type isn't really safe. I
 guess you should use std.strings toStringz.
 
 (Off topic: std.strings toStringz uses the GC to store the c string. Now
 if I passed the returned pointer to a C function, but didn't store a
 reference in the D program, couldn't that c string get collected while
 the c library would expect it to be still there?)
 -- 
 Johannes Pfau


I had my own question about toStringz, yours was one of them, the function
returns a const(char)*, but C doesn't have such a notion so how is this
enforced?
Jul 21 2010