digitalmars.D.learn - multithreading & sqlite
- Lars Johansson (51/51) Jan 27 I have created a program using multithreading and Sqlite3.
- monkyyy (12/64) Jan 27 `!` is a template call, and is always a template call; c syntax
- Andy Valencia (14/18) Jan 28 I wrestled a smaller/simpler SQLite3 into a module I could wrap
- H. S. Teoh (8/24) Jan 28 Have you seen this?
- monkyyy (22/23) Jan 27 strings are the foot gun, use ints and write your own,
- Lars Johansson (18/23) Jan 28 I'm not sure I follow:
- monkyyy (9/14) Jan 28 3, 1.23, "foo" are value-y and go in the original argument list
- Jordan Wilson (40/47) Jan 28 Consider `java/sql`, querying a `ResultSet`...you'd use
- monkyyy (3/5) Jan 28 I feel alone, trying to climb an overhang with no one ahead of me
- Jordan Wilson (3/11) Jan 29 Don't feel lonely. Perhaps despite our difference of opinion on
- monkyyy (7/19) Jan 30 feel free to submit a chapter
- Jordan Wilson (7/10) Jan 27 The `as!` could be because sqlite is actually dynamic, in that
- Lars Johansson (13/17) Feb 01 Thank you all,
- monkyyy (7/9) Feb 01 You need 3 patterns total to write every range function in the
- Lars Johansson (5/15) Feb 01 I do not follow. Are you implying there are a category of
- monkyyy (43/60) Feb 01 the 3 patterns that are enough to write unannotated range
- Lars Johansson (2/7) Feb 02 Thank you. The speed you are replying here is fantastic.
I have created a program using multithreading and Sqlite3.
This is a prestudy for map&reduce.
I found some parts of the program very hard to digest especially
the 'as!' crap.
Sqlite was not simple (for me) to set up and use. It took some
time to understand I must make a physical copy of the result (the
.idup).
It was hard to make this work.
I would appreciate, if anyone can comment on things I can do
better:
(On your own, it not easy to know what can be done better)
import std.parallelism;
import std.stdio;
import core.thread;
import std.array;
import d2sqlite3;
void main() {
writeln("starting");
auto myresults = getData("mydata.db");
auto pool = new TaskPool(2); // Create pool with only 2
worker threads
foreach (i,row; pool.parallel(myresults)) {
writeln(row);
auto ID = row[0];
auto Name = row[1];
auto Age= row[2];
writeln("Starting task ",i, " ",ID, " ",Name, " ", Age);
Thread.sleep(1.seconds); // Simulate work
writeln("finish task ",i);
}
pool.finish();
}
string[][] getData(string DB){
string[][] results;
writeln("getData starts");
auto db = Database(DB);
foreach (row; db.execute("SELECT * FROM users")) {
writeln("ID: ", row[0].as!int); //WTF is this
as! crap
writeln("Name: ", row[1].as!string);
writeln("Age: ", row[2].as!int);
results ~= [
row[0].as!string.idup, // .idup makes
immutable copy
row[1].as!string.idup, // why do i need this?
row[2].as!string.idup // and why do I have
to convert to string?
];}
writeln("getData done");
return results;
}
Jan 27
On Tuesday, 27 January 2026 at 16:38:08 UTC, Lars Johansson wrote:
I have created a program using multithreading and Sqlite3.
This is a prestudy for map&reduce.
I found some parts of the program very hard to digest
especially the 'as!' crap.
Sqlite was not simple (for me) to set up and use. It took some
time to understand I must make a physical copy of the result
(the .idup).
It was hard to make this work.
I would appreciate, if anyone can comment on things I can do
better:
(On your own, it not easy to know what can be done better)
import std.parallelism;
import std.stdio;
import core.thread;
import std.array;
import d2sqlite3;
void main() {
writeln("starting");
auto myresults = getData("mydata.db");
auto pool = new TaskPool(2); // Create pool with only 2
worker threads
foreach (i,row; pool.parallel(myresults)) {
writeln(row);
auto ID = row[0];
auto Name = row[1];
auto Age= row[2];
writeln("Starting task ",i, " ",ID, " ",Name, " ", Age);
Thread.sleep(1.seconds); // Simulate work
writeln("finish task ",i);
}
pool.finish();
}
string[][] getData(string DB){
string[][] results;
writeln("getData starts");
auto db = Database(DB);
foreach (row; db.execute("SELECT * FROM users")) {
writeln("ID: ", row[0].as!int); //WTF is this
as! crap
writeln("Name: ", row[1].as!string);
writeln("Age: ", row[2].as!int);
results ~= [
row[0].as!string.idup, // .idup makes
immutable copy
row[1].as!string.idup, // why do i need
this?
row[2].as!string.idup // and why do I
have to convert to string?
];}
writeln("getData done");
return results;
}
`!` is a template call, and is always a template call; c syntax
doesnt allow you to pass a type to a function 1 char and the
double argument list was considered the best way forward.
I dont know where as comes from, but dup is used when you get a
once off copy to a slice, `string` is *just* a one line of code
`alias string=immutable char[]` in object. This make it a
reference type.
---
I dont like the api's people make for deserialization, this looks
like a bad one. Look for something that parses an entire "row" as
a struct that way your passing bundles of types already.
Jan 27
On Tuesday, 27 January 2026 at 20:15:53 UTC, monkyyy wrote:I dont like the api's people make for deserialization, this looks like a bad one. Look for something that parses an entire "row" as a struct that way your passing bundles of types already.I wrestled a smaller/simpler SQLite3 into a module I could wrap my head around. For deserialization, yes, pulling to a struct is quite tidy--I even got it so it can match up column and struct field names. If you select a subset of the columns, I went with extracting to an explicitly typed tuple. If any field is incompatible with the tuple slot you provided, you get an exception. Writing is much less satisfactory. The libsqlite3 doesn't have introspection in the REPLACE INTO api, nor any related ones--so it all has to be positional. I even asked on their forum, but didn't get any responses. Andy https://sources.vsta.org:7100/tiny/file?name=sqlite.d&ci=tip
Jan 28
"H. S. Teoh" <hsteoh qfbox.info> On Wed, Jan 28, 2026 at 05:25:18PM +0000, Andy Valencia via Digitalmars-d-learn wrote:On Tuesday, 27 January 2026 at 20:15:53 UTC, monkyyy wrote:Have you seen this? https://dpldocs.info/this-week-in-d/Blog.Posted_2025_11_03.html In principle, it should be possible for D to use introspection to automate REPLACE INTO operations. T -- I ate a clock, and it was very time-consuming. I'm going back four seconds.I dont like the api's people make for deserialization, this looks like a bad one. Look for something that parses an entire "row" as a struct that way your passing bundles of types already.I wrestled a smaller/simpler SQLite3 into a module I could wrap my head around. For deserialization, yes, pulling to a struct is quite tidy--I even got it so it can match up column and struct field names. If you select a subset of the columns, I went with extracting to an explicitly typed tuple. If any field is incompatible with the tuple slot you provided, you get an exception. Writing is much less satisfactory. The libsqlite3 doesn't have introspection in the REPLACE INTO api, nor any related ones--so it all has to be positional. I even asked on their forum, but didn't get any responses.
Jan 28
On Tuesday, 27 January 2026 at 16:38:08 UTC, Lars Johansson wrote:This is a prestudy for map&reduce.strings are the foot gun, use ints and write your own, *unannotated* and simple to understand. using the std algorithms means they will be aviable, but it comes with bad naming, the "auto decoding" problems and complexity. ```d import std; auto counter(int i){ struct count{ int front; int end; void popFront(){front++;} bool empty()=>front>=end; } return count(0,i); } unittest{ foreach(i;counter(10)){ i.writeln; }} //todo make `counter(5).take(3).map!(a=>a*2)` return [0,2,4] ```
Jan 27
On Tuesday, 27 January 2026 at 20:26:56 UTC, monkyyy wrote:On Tuesday, 27 January 2026 at 16:38:08 UTC, Lars Johansson wrote:I'm not sure I follow: I had to convert int to string to be able to (i)dup, and I had to dup to transfer the SQLIte data to main. row[0].as!int.idup or row[0].idup result in Error: none of the overloads of template `object.idup` are callable using argument types `!()(ColumnData)` I tried a lot but nothing but row[0].as!string.idup worked. In general programming with D has been a pleasent experience. BUT templates, I do not get it. Instead of having well defined types, it seems you can have anyting with as!. And the error msg does not tell me anything. And how do I know I should use e.g. 'as!quirky' in some special case? Or are there a small set of well defined as! along with the types? I am still very ignorant of D, so this might be just plain stupid. If so tell me. I will now try to learn how to use a debugger.This is a prestudy for map&reduce.strings are the foot gun, use ints and write your own, *unannotated* and simple to understand.
Jan 28
On Wednesday, 28 January 2026 at 09:53:45 UTC, Lars Johansson wrote:BUT templates, I do not get it. And how do I know I should use e.g. 'as!quirky' in some special case? Or are there a small set of well defined as! along with the types?3, 1.23, "foo" are value-y and go in the original argument list int, a=>a are type-y and go in the prepended argument list and will use a `!` This will cover 95% of cases. And is also the difference between alias and enum.I will now try to learn how to use a debugger.I still dont know how to use a debugger; by all accounts its a nightmere.
Jan 28
On Wednesday, 28 January 2026 at 09:53:45 UTC, Lars Johansson wrote:In general programming with D has been a pleasent experience. BUT templates, I do not get it. Instead of having well defined types, it seems you can have anyting with as!.Consider `java/sql`, querying a `ResultSet`...you'd use `.getInt`, `.getString`, `.getFloat`, `.getDouble`...for `d2sqlite3`, it's `as!int`, `as!string`, `as!float`, `as!double`.And how do I know I should use e.g. 'as!quirky' in some special case? Or are there a small set of well defined as! along with the types?From a brief look at the source code, it seems like there are constraints: ```dlang auto as(T)(T defaultValue = T.init) if (isBoolean!T || isNumeric!T || isSomeString!T) ``` This gives you an indication of what kind of values `T` can be. If you wanted to deal with the whole row, you can also provide your own struct: ```dlang T as(T)() if (is(T == struct)) ``` From the unittest: ```dlang unittest { struct Item { int _id; string name; } auto db = Database(":memory:"); db.run("CREATE TABLE items (name TEXT); INSERT INTO items VALUES ('Light bulb')"); auto results = db.execute("SELECT rowid AS id, name FROM items"); auto row = results.front; auto thing = row.as!Item(); assert(thing == Item(1, "Light bulb")); } ``` Generally, I find templates useful, although a little bit of a learning curve. Jordan
Jan 28
On Thursday, 29 January 2026 at 04:09:38 UTC, Jordan Wilson wrote:Generally, I find templates useful, although a little bit of a learning curve.I feel alone, trying to climb an overhang with no one ahead of me https://www.youtube.com/watch?v=fUGRmkeRb-o
Jan 28
On Thursday, 29 January 2026 at 04:20:57 UTC, monkyyy wrote:On Thursday, 29 January 2026 at 04:09:38 UTC, Jordan Wilson wrote:Don't feel lonely. Perhaps despite our difference of opinion on templates, we can still be friends?Generally, I find templates useful, although a little bit of a learning curve.I feel alone, trying to climb an overhang with no one ahead of me https://www.youtube.com/watch?v=fUGRmkeRb-o
Jan 29
On Thursday, 29 January 2026 at 20:31:34 UTC, Jordan Wilson wrote:On Thursday, 29 January 2026 at 04:20:57 UTC, monkyyy wrote:feel free to submit a chapter https://crazymonkyyy.github.io/blackmagic-in-d/ The other template wizards were cowards whenever they found a compiler bug that leaked state and didnt start building abstractions and patterns on top (remind them of this whenever possible :) )On Thursday, 29 January 2026 at 04:09:38 UTC, Jordan Wilson wrote:Don't feel lonely. Perhaps despite our difference of opinion on templates, we can still be friends?Generally, I find templates useful, although a little bit of a learning curve.I feel alone, trying to climb an overhang with no one ahead of me https://www.youtube.com/watch?v=fUGRmkeRb-o
Jan 30
On Tuesday, 27 January 2026 at 16:38:08 UTC, Lars Johansson wrote:I have created a program using multithreading and Sqlite3. This is a prestudy for map&reduce. [...]The `as!` could be because sqlite is actually dynamic, in that you can have mixed data types in a single column, whereas D is statically typed. In regards to you needing to use `idup`, I'm not sure why that would be...I would expect `row[0].as!string` to work. Jordan
Jan 27
On Tuesday, 27 January 2026 at 16:38:08 UTC, Lars Johansson wrote:
I have created a program using multithreading and Sqlite3.
This is a prestudy for map&reduce.
...
}
Thank you all,
I appreciate all, eventhough some is not really about my initial
post.
After reading and thinking, I sort of changed my plans. I will
look for (or create) a 'row parser' for database communication. I
wait with debugger. I put templates on the shelf for now, my
skill level is to low to understand templates.
I have done some basic work on parallel and async (fibers), very
easy as far as I went.
Now I will explore networking. For me performance is key in
communication, I do not want to wait for msgs/signals/packages
etc. If any of you have up front recommendations please tell.
Feb 01
On Sunday, 1 February 2026 at 12:54:15 UTC, Lars Johansson wrote:I put templates on the shelf for now, my skill level is to low to understand templates.You need 3 patterns total to write every range function in the style I suggest; I stand by my original suggestion; the complexity of template hell comes in when you try to annotate, introspect, or have a some kind of big ask. Phoboes annotates functions and explodes its line count often by 10x to 100x
Feb 01
On Sunday, 1 February 2026 at 20:49:34 UTC, monkyyy wrote:On Sunday, 1 February 2026 at 12:54:15 UTC, Lars Johansson wrote:I do not follow. Are you implying there are a category of templates that is relatively easy to understand, and I can avoid what you call template hell. Can you please be a bit more explicit and give an example.I put templates on the shelf for now, my skill level is to low to understand templates.You need 3 patterns total to write every range function in the style I suggest; I stand by my original suggestion; the complexity of template hell comes in when you try to annotate, introspect, or have a some kind of big ask. Phoboes annotates functions and explodes its line count often by 10x to 100x
Feb 01
On Monday, 2 February 2026 at 06:54:53 UTC, Lars Johansson wrote:On Sunday, 1 February 2026 at 20:49:34 UTC, monkyyy wrote:the 3 patterns that are enough to write unannotated range functions: voldemort types: if you have some kind of templated function, structs defined inside them just work ```d auto map(alias F,R)(R r){ struct map_{ R r; auto front()=>F(r.front); void popFront(){r.popFront;} bool empty()=>r.empty; } return map_(r); } ``` lazy templates: given an empty template argument list, a name can exist without being initialized ```d auto bar(R)(R r){ struct maybehaslength{ R r; auto front()=>r.front; auto length()()=>r.length; } // ^ ^ return maybehaslength(r); } unittest{ auto foo=iota(int.max).filter!(a=>a>100).bar; assert(foo.front==101); // assert(foo.length); //errors out here because filter cant define length } ``` understanding default argument resolution; if you want counter(10), counter(5,10), and counter(3,7.5,double(.34)); to all compile from a single header youll need to understand the headers thats, not that trivial. --- Templates are bad lambda calculus machines that create turning machines; if you need to *calculate* something, well, thats means learning what I mean by that; if you dont, you dont need a lick of theory.On Sunday, 1 February 2026 at 12:54:15 UTC, Lars Johansson wrote:I do not follow. Are you implying there are a category of templates that is relatively easy to understand, and I can avoid what you call template hell. Can you please be a bit more explicit and give an example.I put templates on the shelf for now, my skill level is to low to understand templates.You need 3 patterns total to write every range function in the style I suggest; I stand by my original suggestion; the complexity of template hell comes in when you try to annotate, introspect, or have a some kind of big ask. Phoboes annotates functions and explodes its line count often by 10x to 100x
Feb 01
On Monday, 2 February 2026 at 07:58:25 UTC, monkyyy wrote:On Monday, 2 February 2026 at 06:54:53 UTC, Lars Johansson wrote:the 3 patterns that are enough to write unannotated range functions: [...]Thank you. The speed you are replying here is fantastic.
Feb 02