• Stefan Koch (9/9) Feb 27 2016 Hello,
• Suliman (2/2) Feb 27 2016 Andrei, would it be possible to replace old etc.c.sqlite3 with
• ketmar (1/1) Feb 27 2016 no.
• Suliman (2/3) Feb 27 2016 Why?
• Adam D. Ruppe (4/5) Feb 27 2016 etc.c is for the C interface. This is not the C interface.
• Stefan Koch (15/20) Feb 27 2016 The duration of that time depends heavily on the amount of
• Stefan Koch (5/14) Feb 28 2016 Update I just found another case that cannot be handled properly.
• Stefan Koch (2/4) Mar 08 2016 That bug is fixed now!
• Stefan Koch (7/7) Feb 29 2016 I made a huge performance improvement sqlite-d is now 6-8 times
• Stefan Koch (8/17) May 24 2016 Well not much has changed since I made this announcement.
• Suliman (4/29) May 24 2016 Even faster then official version?
• Stefan Koch (7/10) May 25 2016 Not really.
• Suliman (3/15) May 26 2016 Could you explain more details? What do you mean by indirection
• Stefan Koch (10/12) May 26 2016 Sure, I can explain.
• Suliman (7/20) May 26 2016 Oh! Look like for all time I misunderstood the purpose of your
• Stefan Koch (8/14) May 26 2016 Yes and no, currently it does only provide primitives to iterate
• Stefan Koch (6/9) May 25 2016 A 20% performance improvement has landed in master!
• jmh530 (2/9) May 25 2016 You might consider writing up some kind of comparison.
• xenon325 (7/18) May 26 2016 How does that play with transactions and ACID in general ?
• Stefan Koch (14/18) May 26 2016 sqlite-d provides no safety whatsoever.
• Stefan Koch (3/6) May 27 2016 Opps I was wrong :)
• Stefan Koch (3/9) May 27 2016 Maybe the thread in general got lost :)
• Stefan Koch (4/4) May 29 2016 Support for reading Index-Trees and there(WITHOUT ROWID) tables,

Stefan Koch <uplink.coder googlemail.com> writes:

Hello,

I am happy to announce the official alpha version of sqlite-d!

sqlite-d is a reader for the SQLite File Format 3.

In the future I will implement a SQL-like API on top of it.

Top-notch performance is one of the explicit goals of this 
project!

please note that currently only the ctfe branch is populated.

I welcome contributions!

Repo-URL : https://github.com/UplinkCoder/sqlite-d

Suliman <evermind live.ru> writes:

Andrei, would it be possible to replace old etc.c.sqlite3 with 
new native driver when Stefan will finish work on it?

ketmar <ketmar ketmar.no-ip.org> writes:

no.

Suliman <evermind live.ru> writes:

On Saturday, 27 February 2016 at 16:35:05 UTC, ketmar wrote:
 no.

Why?

Adam D. Ruppe <destructionator gmail.com> writes:

On Saturday, 27 February 2016 at 16:54:49 UTC, Suliman wrote:
 Why?

etc.c is for the C interface. This is not the C interface.

Besides, the original code will surely be ahead of features and 
compatibility for a long time anyway.

Stefan Koch <uplink.coder googlemail.com> writes:

On Saturday, 27 February 2016 at 17:03:17 UTC, Adam D. Ruppe 
wrote:
 On Saturday, 27 February 2016 at 16:54:49 UTC, Suliman wrote:
 Why?

 etc.c is for the C interface. This is not the C interface.

 Besides, the original code will surely be ahead of features and 
 compatibility for a long time anyway.

The duration of that time depends heavily on the amount of 
contributions :)

But the short term goal is not to reinvent SQLite rather to 
present a file-format compatible alternative.

The big advantage of sqlite-d is that you can write your logic in 
D without having to worry about generating SQL or converting your 
data into strings.
However in the long-term I plan to provide SQL support if there 
is sufficient demand for it.

Right now I am still experimenting with different approaches to 
make sure the API will not compromise performance.

Also I pushed ctfe to the master-branch now. the repo no longer 
looks empty.

Stefan Koch <uplink.coder googlemail.com> writes:

On Saturday, 27 February 2016 at 16:08:21 UTC, Stefan Koch wrote:
 Hello,

 I am happy to announce the official alpha version of sqlite-d!

 sqlite-d is a reader for the SQLite File Format 3.

 In the future I will implement a SQL-like API on top of it.

 Top-notch performance is one of the explicit goals of this 
 project!

 please note that currently only the ctfe branch is populated.

 I welcome contributions!

 Repo-URL : https://github.com/UplinkCoder/sqlite-d

Update I just found another case that cannot be handled properly. 
It just happens with insanely huge databases.

Also I commited my test-data now... I wonder why nobody said 
anything about the app failing on startup ...

Stefan Koch <uplink.coder googlemail.com> writes:

On Sunday, 28 February 2016 at 12:14:14 UTC, Stefan Koch wrote:

 Update I just found another case that cannot be handled 
 properly. It just happens with insanely huge databases.

That bug is fixed now!

Stefan Koch <uplink.coder googlemail.com> writes:

I made a huge performance improvement sqlite-d is now 6-8 times 
faster then on the day were it wad able to read the first 
payloads.

2. I am heavily working on write-support.

3.Sqlite-d will then implement the allocator interface! (although 
I am flexible on that should it turn out to be a bad idea.

I wish you all a happy day!

Stefan Koch <uplink.coder googlemail.com> writes:

On Saturday, 27 February 2016 at 16:08:21 UTC, Stefan Koch wrote:
 Hello,

 I am happy to announce the official alpha version of sqlite-d!

 sqlite-d is a reader for the SQLite File Format 3.

 In the future I will implement a SQL-like API on top of it.

 Top-notch performance is one of the explicit goals of this 
 project!

 please note that currently only the ctfe branch is populated.

 I welcome contributions!

 Repo-URL : https://github.com/UplinkCoder/sqlite-d

Well not much has changed since I made this announcement.
By fixing a really jarring bug I had a slight performance 
regression.
But this is still the fastest SQLite reader I know of.

This project is currently on the back burner.
However in the next days there will be another significant 
performance improvement :)

Suliman <evermind live.ru> writes:

On Wednesday, 25 May 2016 at 03:48:48 UTC, Stefan Koch wrote:
 On Saturday, 27 February 2016 at 16:08:21 UTC, Stefan Koch 
 wrote:
 Hello,

 I am happy to announce the official alpha version of sqlite-d!

 sqlite-d is a reader for the SQLite File Format 3.

 In the future I will implement a SQL-like API on top of it.

 Top-notch performance is one of the explicit goals of this 
 project!

 please note that currently only the ctfe branch is populated.

 I welcome contributions!

 Repo-URL : https://github.com/UplinkCoder/sqlite-d

 Well not much has changed since I made this announcement.
 By fixing a really jarring bug I had a slight performance 
 regression.
 But this is still the fastest SQLite reader I know of.

 This project is currently on the back burner.
 However in the next days there will be another significant 
 performance improvement :)

Even faster then official version?

What about futures, would it possible to make it's 100% 
compatibility with C version?

Stefan Koch <uplink.coder googlemail.com> writes:

On Wednesday, 25 May 2016 at 06:52:22 UTC, Suliman wrote:
 Even faster then official version?

 What about futures, would it possible to make it's 100% 
 compatibility with C version?

Not really.
The reason why it is faster is because there is no indirection in 
working with the data.
If I had full SQL support like there is in the C version, and you 
take that route rather then the direct access.
It will be as slow as the C version, and probably slower.

Suliman <evermind live.ru> writes:

On Wednesday, 25 May 2016 at 14:10:41 UTC, Stefan Koch wrote:
 On Wednesday, 25 May 2016 at 06:52:22 UTC, Suliman wrote:
 Even faster then official version?

 What about futures, would it possible to make it's 100% 
 compatibility with C version?

 Not really.
 The reason why it is faster is because there is no indirection 
 in working with the data.
 If I had full SQL support like there is in the C version, and 
 you take that route rather then the direct access.
 It will be as slow as the C version, and probably slower.

Could you explain more details? What do you mean by indirection 
work with data?

Stefan Koch <uplink.coder googlemail.com> writes:

On Thursday, 26 May 2016 at 14:45:58 UTC, Suliman wrote:
 Could you explain more details? What do you mean by indirection 
 work with data?

Sure, I can explain.
So, all that sqlite-d does is reading the sqlite-db files.
However the proper sqlite does much more:
It implements a whole ByteCode-Compiler and Interpreter to be 
able to precompile and execute SQL-statements.

Thus even for simple querys you have a significant overhead.
If you want to use sqlite-d you basically write all your queries 
as D code which can be directly executed, rather then having to 
be compiled from SQL to ByteCode.

Suliman <evermind live.ru> writes:

On Thursday, 26 May 2016 at 19:11:50 UTC, Stefan Koch wrote:
 On Thursday, 26 May 2016 at 14:45:58 UTC, Suliman wrote:
 Could you explain more details? What do you mean by 
 indirection work with data?

 Sure, I can explain.
 So, all that sqlite-d does is reading the sqlite-db files.
 However the proper sqlite does much more:
 It implements a whole ByteCode-Compiler and Interpreter to be 
 able to precompile and execute SQL-statements.

 Thus even for simple querys you have a significant overhead.
 If you want to use sqlite-d you basically write all your 
 queries as D code which can be directly executed, rather then 
 having to be compiled from SQL to ByteCode.

Oh! Look like for all time I misunderstood the purpose of your 
project. Do you mean that your tool is created not for being SQL 
compatible driver, but make possible to use D code for iteration 
with DB?

And does it's mean that is not possible to write SQL-query with 
this driver?

Stefan Koch <uplink.coder googlemail.com> writes:

On Thursday, 26 May 2016 at 19:35:22 UTC, Suliman wrote:

 Oh! Look like for all time I misunderstood the purpose of your 
 project. Do you mean that your tool is created not for being 
 SQL compatible driver, but make possible to use D code for 
 iteration with DB?

Yes and no, currently it does only provide primitives to iterate 
over sqlite-tables.
And originally it was only intended for this purpose.
However per-compiling SQL at compile-time is a great way for D's 
features to shine.

 And does it's mean that is not possible to write SQL-query with 
 this driver?

I want to add the capability to precompile and execute SQL, 
however that is still a long way away.

Stefan Koch <uplink.coder googlemail.com> writes:

On Wednesday, 25 May 2016 at 03:48:48 UTC, Stefan Koch wrote:
 This project is currently on the back burner.
 However in the next days there will be another significant 
 performance improvement :)

A 20% performance improvement has landed in master!
It is possible that there are more places in the code-base where 
unrolling can buy you performance.

Never forget.
PERFMATTERS!

jmh530 <john.michael.hall gmail.com> writes:

On Wednesday, 25 May 2016 at 03:48:48 UTC, Stefan Koch wrote:
 Well not much has changed since I made this announcement.
 By fixing a really jarring bug I had a slight performance 
 regression.
 But this is still the fastest SQLite reader I know of.

 This project is currently on the back burner.
 However in the next days there will be another significant 
 performance improvement :)

You might consider writing up some kind of comparison.

xenon325 <anm programmer.net> writes:

On Saturday, 27 February 2016 at 16:08:21 UTC, Stefan Koch wrote:
 Hello,

 I am happy to announce the official alpha version of sqlite-d!

 sqlite-d is a reader for the SQLite File Format 3.

On Thursday, 26 May 2016 at 19:11:50 UTC, Stefan Koch wrote:
 So, all that sqlite-d does is reading the sqlite-db files.
 However the proper sqlite does much more:
 It implements a whole ByteCode-Compiler and Interpreter to be 
 able to precompile and execute SQL-statements.

 Thus even for simple querys you have a significant overhead.
 If you want to use sqlite-d you basically write all your 
 queries as D code which can be directly executed, rather then 
 having to be compiled from SQL to ByteCode.

How does that play with transactions and ACID in general ?

For example, I have one thread with traditional (slow) SQLite 
client, which seldom updates data. And another thread which reads 
data with sqlite-d. Will not program crash or read 
trash/inconsistent data ?

Stefan Koch <uplink.coder googlemail.com> writes:

On Friday, 27 May 2016 at 05:03:43 UTC, xenon325 wrote:
 For example, I have one thread with traditional (slow) SQLite 
 client, which seldom updates data. And another thread which 
 reads data with sqlite-d. Will not program crash or read 
 trash/inconsistent data ?

sqlite-d provides no safety whatsoever.
You will the database read in whatever state the operation system 
see the underlying file.

It is unlikely that it will crash.
But of course it may read stale or incomplete data.

There is a locking mechanism in SQLite-proper.
However SQLite-D currently makes no attempt in looking for the 
lock-page.

As a side note:
I would like to make clear that SQLite is very fast for what it 
does.
It is among the fastest of SQL-databases, and has a high quality 
of implementation.

Stefan Koch <uplink.coder googlemail.com> writes:

On Friday, 27 May 2016 at 05:12:33 UTC, Stefan Koch wrote:
 There is a locking mechanism in SQLite-proper.
 However SQLite-D currently makes no attempt in looking for the 
 lock-page.

Opps I was wrong :)
The Lock-page has nothing todo with threading :)

Stefan Koch <uplink.coder googlemail.com> writes:

On Friday, 27 May 2016 at 07:08:30 UTC, Stefan Koch wrote:
 On Friday, 27 May 2016 at 05:12:33 UTC, Stefan Koch wrote:
 There is a locking mechanism in SQLite-proper.
 However SQLite-D currently makes no attempt in looking for the 
 lock-page.

 Opps I was wrong :)
 The Lock-page has nothing todo with threading :)

Maybe the thread in general got lost :)
SQLite-D works at compile-time.

Stefan Koch <uplink.coder googlemail.com> writes:

Support for reading Index-Trees and there(WITHOUT ROWID) tables, 
has landed in master!

Now the real fun can start.
CTFE Query translation and optimization.