www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Postgres and other database interfaces

reply Joe <jma freedomcircle.com> writes:
Back on 13 January, I posted in the Learn forum some questions 
regarding using Postgres and got a reply that stated the 
following:

On Monday, 15 January 2018 at 02:28:29 UTC, Matthias Klumpp wrote:

 In any case, please don't start another Postgres library and 
 consider contributing to one of the existing ones, so that we 
 maybe have one really awesome, 100% complete library at some 
 point.


I'm revisiting this because I now have had the chance to look at 
the various libraries and IMHO the "really awesome, 100% complete 
library" depends on the users' goals and their definition of 
awesome and complete.

Aside from ddbc which is like ODBC/JDBC and thus supports 
multiple db's (not my interest), I've found five 
Postgres-specific libraries worthy of further exploration:

1. derelict-pq. This is the most downloaded one and is simply a 
binding over libpq, so AFAICT it's nearly 100% complete. It's 
also well-documented, in part because the Derelict libraries 
provide overall guidance, but mainly because a D user can refer 
directly to the comprehensive libpq documentation and (C) 
examples. The only strange fact is that it's been stuck in a 
beta.3 release for the last five months.

2. dpq2. Second most downloaded and built on top of derelict-pq. 
The "documentation" consists of a README listing the features and 
a single example, which appears to focus on support for JSON/BSON.

3. vibe-d-postgresql. Third most downloaded and built on top of 
dpq2. Docs consist of a single example program.

4. ddb. About the same number of downloads as the above. 
Implemented on top of front/backend protocol. No documentation 
although repository has a folder with two sample programs.

5. dpq. Implements its own wrapper over libpq and has some ORM 
functionality. Docs are a README with example program.

The main issue is that, other than derelict-pq, using any of 
these libraries involves reading the library code and 
understanding the sui generis interfaces implemented by each.

I'm new to the D community, but coming from Python, the contrast 
is significant. First there is the well-documented PEP 249 
(https://www.python.org/dev/peps/pep-0249/) promulgated by the 
Python DB-SIG (and note that 249 is v.2), which led to the 
implementation of psycopg (also well-documented including various 
extensions to the API) and many other adapters to Postgres and 
other databases.

Since I'm new, I don't know if there's any interest in moving in 
such a direction.
Feb 23 2018
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
There is plenty of desire to build a generalized SQL interface for Phobos.

But somebody needs to do it and it won't be all that much fun to do.
Feb 23 2018
next sibling parent reply Erik Smith <erik cruiserhouse.com> writes:
On Saturday, 24 February 2018 at 05:45:45 UTC, rikki cattermole 
wrote:

 There is plenty of desire to build a generalized SQL interface 
 for Phobos.

 But somebody needs to do it and it won't be all that much fun 
 to do.


Hi Joe and Rikki,

This is the goal of my dstddb project and I've picked it up again 
after a long hiatus.    I do have limited support for a native 
postgres driver and there is plenty of work to do in general, but 
I'm working on it.   I have some type support improvements for 
postgres that will arrive shortly.

erik
Feb 23 2018
parent Suliman <evermind live.ru> writes:
On Saturday, 24 February 2018 at 06:05:38 UTC, Erik Smith wrote:

 On Saturday, 24 February 2018 at 05:45:45 UTC, rikki cattermole 
 wrote:

 There is plenty of desire to build a generalized SQL interface 
 for Phobos.

 But somebody needs to do it and it won't be all that much fun 
 to do.


 Hi Joe and Rikki,

 This is the goal of my dstddb project and I've picked it up 
 again after a long hiatus.    I do have limited support for a 
 native postgres driver and there is plenty of work to do in 
 general, but I'm working on it.   I have some type support 
 improvements for postgres that will arrive shortly.

 erik


I wish it have designed in 
http://code.dlang.org/packages/mysql-native style. This driver 
IMHO is very well designed
Feb 24 2018
prev sibling parent reply Denis F <denis.feklushkin gmail.com> writes:
On Saturday, 24 February 2018 at 05:45:45 UTC, rikki cattermole 
wrote:

 There is plenty of desire to build a generalized SQL interface 
 for Phobos.

 But somebody needs to do it and it won't be all that much fun 
 to do.


I want to dwell on this.

It seems to me that this is a common misconception that it is 
possible to create a good universal tool for accessing the 
database in place of the many existing ones.

SQL is something like BASIC from the 70-80s: there is no single 
standard, there are many dialects. And the differences are both 
syntactic and deep, for example, Postgres and Oracle differently 
interpret the ANSI standard concerning transactions. The same 
applies to the connection layer: some of the databases allow you 
to organize a gradual download of the results, some - feedback in 
the form of something like "events", etc. Also, Postgres 
quarantiees that result of query will be immutable and this is 
very useful, but some one another DB maybe not.

At best, such a tool will be suitable for simple trivial things.

On the other hand, needing  for this is also not obvious - 
databases in real projects isn't switched usually every 
week/month because they are not interchangeable, except for some 
very simple cases. Usually RDBMS engine should be elected based 
on your requiriments before you start to use it.
Feb 24 2018
next sibling parent Denis F <denis.feklushkin gmail.com> writes:
On Saturday, 24 February 2018 at 18:56:23 UTC, Denis F wrote:

 Postgres quarantiees that result of query will be immutable


libpq, of course, not Postgres itself.
Feb 24 2018
prev sibling next sibling parent Erik Smith <erik cruiserhouse.com> writes:
On Saturday, 24 February 2018 at 18:56:23 UTC, Denis F wrote:

 On Saturday, 24 February 2018 at 05:45:45 UTC, rikki cattermole 
 wrote:

 There is plenty of desire to build a generalized SQL interface 
 for Phobos.

 But somebody needs to do it and it won't be all that much fun 
 to do.


 I want to dwell on this.

 It seems to me that this is a common misconception that it is 
 possible to create a good universal tool for accessing the 
 database in place of the many existing ones.

 SQL is something like BASIC from the 70-80s: there is no single 
 standard, there are many dialects. And the differences are both 
 syntactic and deep, for example, Postgres and Oracle 
 differently interpret the ANSI standard concerning 
 transactions. The same applies to the connection layer: some of 
 the databases allow you to organize a gradual download of the 
 results, some - feedback in the form of something like 
 "events", etc. Also, Postgres quarantiees that result of query 
 will be immutable and this is very useful, but some one another 
 DB maybe not.

 At best, such a tool will be suitable for simple trivial things.

 On the other hand, needing  for this is also not obvious - 
 databases in real projects isn't switched usually every 
 week/month because they are not interchangeable, except for 
 some very simple cases. Usually RDBMS engine should be elected 
 based on your requiriments before you start to use it.



I agree that the differences in both the client interface and the 
underlying SQL support are complicated issues.  However, it seems 
pretty clear that languages have some level of standardized db 
interface have benefited enormously in terms of adoption for 
building real word applications (JDBC for instance).  I think it 
is true people don't switch databases that much, but having a 
standard interface (especially one that is easy to use) that 
covers the most common uses cases does make the use of the 
platform itself that much more approachable.

Some of the differences can be hidden internally for the basic 
use case (such as gradual downloads) and optionally available as 
a introspective capability when the underlying driver supports it 
for more advanced usage.   The goal is to expose as much 
capability without overly complicating the interface.  This is 
easier to do now with modern abstractions then it was with ODBC 
in the 90's.

It would be good, however, to keep track of the specific problem 
areas that you mentioned to ensure that they are addressed well.

If one thing is clear from this thread, documentation is 
critical. I'm also aware that vibed support, for the databases 
that can most readily support it, is also a high priority.

erik
Feb 24 2018
prev sibling parent reply Joe <jma freedomcircle.com> writes:
On Saturday, 24 February 2018 at 18:56:23 UTC, Denis F wrote:

 It seems to me that this is a common misconception that it is 
 possible to create a good universal tool for accessing the 
 database in place of the many existing ones.


I don't think what is needed a "good universal tool" but a good 
enough common interface. Python PEP 249 is very similar to what 
I've seen in the D libraries I've looked at: there is a 
Connection class of some kind, and a Cursor (or command or query) 
class, each with a set of well-defined (and generally not 
unexpected) methods: commit/rollback on the connection, 
execute/fetch/etc. on the cursor). PEP 249 also standardizes the 
exceptions and some datatypes.

It's not perfect, or universal enough, but I believe it allowed 
db development under Python to proceed at a faster pace than 
otherwise. There's a similar story on the web side. When Django 
first came out, IIRC it had its own interface, but then the WSGI 
standard came out (PEP 333) and every Python web framework 
including Django moved to it (it also resulted in a proliferation 
of frameworks, but that's another story). And WSGI even 
influenced Ruby's Rack and other such interfaces.

Joe
Feb 24 2018
parent reply Denis F <denis.feklushkin gmail.com> writes:
On Saturday, 24 February 2018 at 20:45:03 UTC, Joe wrote:

 On Saturday, 24 February 2018 at 18:56:23 UTC, Denis F wrote:

 It seems to me that this is a common misconception that it is 
 possible to create a good universal tool for accessing the 
 database in place of the many existing ones.


 I don't think what is needed a "good universal tool" but a good 
 enough common interface. Python PEP 249 is very similar to what 
 I've seen in the D libraries I've looked at: there is a 
 Connection class of some kind, and a Cursor (or command or 
 query) class, each with a set of well-defined (and generally 
 not unexpected) methods: commit/rollback on the connection, 
 execute/fetch/etc. on the cursor). PEP 249 also standardizes 
 the exceptions and some datatypes.


Currently in dpq2 all of these things are implemented except 
cursors (I think cursors became "unpopular" lately)


 It's not perfect, or universal enough, but I believe it allowed 
 db development under Python to proceed at a faster pace than 
 otherwise.


If anyone really want to impliment your idea, at my first glance 
at the PEP 249 I had a feeling that this is work for time less 
than a 1-2 weeks. It can be a simple wrapper over dpq2, 
mysql-native, sqlite3, etc.
Feb 24 2018
parent reply Joe <jma freedomcircle.com> writes:
On Saturday, 24 February 2018 at 23:04:39 UTC, Denis F wrote:

 If anyone really want to impliment your idea, at my first 
 glance at the PEP 249 I had a feeling that this is work for 
 time less than a 1-2 weeks. It can be a simple wrapper over 
 dpq2, mysql-native, sqlite3, etc.


I'm not saying that anyone should implement something like PEP 
249, particularly *not* as a wrapper over those libraries. My 
point is that if the implementors of dpq2, mysql-native, etc., 
could "get together" (at DConf, in forums, by email) and discuss 
their interfaces, perhaps they could agree on a set of D database 
interface classes and then each would work towards that common 
set. That's more or less what the Python DB-SIG did. Then every 
library would benefit because, for one, the common set can be 
documented just once. Also, if it's a sufficiently reasonable 
spec, the libraries that don't move towards it should see less 
adoption because, unless the divergent library has a 
significantly enhanced interface, developers will shy away from 
having to learn a different one.

It's the same concept as the SQL (and other) Standards. There may 
be variances between implementations but no one has implemented a 
RETRIEVE or PROJECT statement in lieu of a SELECT statement (and 
note there *were* relational systems that used the former).

Joe
Feb 24 2018
parent reply Denis F <denis.feklushkin gmail.com> writes:
On Sunday, 25 February 2018 at 03:03:05 UTC, Joe wrote:

 On Saturday, 24 February 2018 at 23:04:39 UTC, Denis F wrote:

 If anyone really want to impliment your idea, at my first 
 glance at the PEP 249 I had a feeling that this is work for 
 time less than a 1-2 weeks. It can be a simple wrapper over 
 dpq2, mysql-native, sqlite3, etc.


 I'm not saying that anyone should implement something like PEP 
 249, particularly *not* as a wrapper over those libraries. My 
 point is that if the implementors of dpq2, mysql-native, etc., 
 could "get together" (at DConf, in forums, by email) and 
 discuss their interfaces, perhaps they could agree on a set of 
 D database interface classes and then each would work towards 
 that common set.


Libraries already are similar automatically because RDBMSes uses 
similar principles. But it is impossible to make libraries 
absolutely identical due to the presence of important significant 
nuances in each engine.


 That's more or less what the Python DB-SIG did. Then every 
 library would benefit because, for one, the common set can be 
 documented just once. Also, if it's a sufficiently reasonable 
 spec, the libraries that don't move towards it should see less 
 adoption because, unless the divergent library has a 
 significantly enhanced interface, developers will shy away from 
 having to learn a different one.

 It's the same concept as the SQL (and other) Standards.


Ok, I propose to be consistent and ask for compliance with the 
Standard from the RDBMSes. For example, arguments substitution:

MySQL uses the '?'
PostgreSQL uses $1
SQLite accepts both
Oracle uses a :name

(Really, it is very important to come to an agreement here 
because, for sure, the next obvious step is writing an ORM 
generator on top of the idea what you are proposing.)


 There may be variances between implementations but no one has 
 implemented a RETRIEVE or PROJECT statement in lieu of a SELECT 
 statement (and note there *were* relational systems that used 
 the former).


For example, I would not change unobvious at first sight system 
of classes "Result" and inherited from it class "Answer" for the 
sake of similarity to other libraries. Simply because this is the 
way what Postgres client library works, and if this classes will 
be removed some significant functionality will be lost.
I'm sure that in other libraries there is something similar.
Feb 25 2018
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 25/02/2018 10:07 PM, Denis F wrote:

 On Sunday, 25 February 2018 at 03:03:05 UTC, Joe wrote:

 On Saturday, 24 February 2018 at 23:04:39 UTC, Denis F wrote:

 If anyone really want to impliment your idea, at my first glance at 
 the PEP 249 I had a feeling that this is work for time less than a 
 1-2 weeks. It can be a simple wrapper over dpq2, mysql-native, 
 sqlite3, etc.


 I'm not saying that anyone should implement something like PEP 249, 
 particularly *not* as a wrapper over those libraries. My point is that 
 if the implementors of dpq2, mysql-native, etc., could "get together" 
 (at DConf, in forums, by email) and discuss their interfaces, perhaps 
 they could agree on a set of D database interface classes and then 
 each would work towards that common set.


 
 Libraries already are similar automatically because RDBMSes uses similar 
 principles. But it is impossible to make libraries absolutely identical 
 due to the presence of important significant nuances in each engine.
 

 That's more or less what the Python DB-SIG did. Then every library 
 would benefit because, for one, the common set can be documented just 
 once. Also, if it's a sufficiently reasonable spec, the libraries that 
 don't move towards it should see less adoption because, unless the 
 divergent library has a significantly enhanced interface, developers 
 will shy away from having to learn a different one.

 It's the same concept as the SQL (and other) Standards.


 
 Ok, I propose to be consistent and ask for compliance with the Standard 
 from the RDBMSes. For example, arguments substitution:
 
 MySQL uses the '?'
 PostgreSQL uses $1
 SQLite accepts both
 Oracle uses a :name
 
 (Really, it is very important to come to an agreement here because, for 
 sure, the next obvious step is writing an ORM generator on top of the 
 idea what you are proposing.)


On IRC earlier today we discussed database libs a bit, we agreed that 
both "?" and " name" needed to be supported. No other suggestions came 
up. We don't really need a third or fourth form I think.
Feb 25 2018
next sibling parent reply Denis F <denis.feklushkin gmail.com> writes:
On Sunday, 25 February 2018 at 09:23:18 UTC, rikki cattermole 
wrote:


 It's the same concept as the SQL (and other) Standards.


 
 Ok, I propose to be consistent and ask for compliance with the 
 Standard from the RDBMSes. For example, arguments substitution:
 
 MySQL uses the '?'
 PostgreSQL uses $1
 SQLite accepts both
 Oracle uses a :name
 
 (Really, it is very important to come to an agreement here 
 because, for sure, the next obvious step is writing an ORM 
 generator on top of the idea what you are proposing.)


 On IRC earlier today we discussed database libs a bit, we 
 agreed that both "?" and " name" needed to be supported. No 
 other suggestions came up. We don't really need a third or 
 fourth form I think.


...And you are planned to give up "$1" support in favor of '?' 
and ":name"?

But it is impossible to convert text :names or '?' into 
Postgres's "$1": Postgres isn't knows fields names at start of a 
query processing and you can't replace '?' to "$<num>" by simple 
'replace' call - it will need full syntax parsing of Posgres SQL 
query because queries can contain EXECUTE statement (whose 
purpose is executing dynamic commands).
Feb 25 2018
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 26/02/2018 12:46 AM, Denis F wrote:

 On Sunday, 25 February 2018 at 09:23:18 UTC, rikki cattermole wrote:
 

 It's the same concept as the SQL (and other) Standards.


 Ok, I propose to be consistent and ask for compliance with the 
 Standard from the RDBMSes. For example, arguments substitution:

 MySQL uses the '?'
 PostgreSQL uses $1
 SQLite accepts both
 Oracle uses a :name

 (Really, it is very important to come to an agreement here because, 
 for sure, the next obvious step is writing an ORM generator on top of 
 the idea what you are proposing.)


 On IRC earlier today we discussed database libs a bit, we agreed that 
 both "?" and " name" needed to be supported. No other suggestions came 
 up. We don't really need a third or fourth form I think.


 
 ...And you are planned to give up "$1" support in favor of '?' and ":name"?
 
 But it is impossible to convert text :names or '?' into Postgres's "$1": 
 Postgres isn't knows fields names at start of a query processing and you 
 can't replace '?' to "$<num>" by simple 'replace' call - it will need 
 full syntax parsing of Posgres SQL query because queries can contain 
 EXECUTE statement (whose purpose is executing dynamic commands).


Let's not go inventing a solution here, what does JDBC and ADO.net do?
Feb 25 2018
parent Denis F <denis.feklushkin gmail.com> writes:
On Sunday, 25 February 2018 at 12:12:58 UTC, rikki cattermole 
wrote:


 Let's not go inventing a solution here, what does JDBC and 
 ADO.net do?


JDBC isn't touches this - it is just passes statement "as is". 
I.e. it isn't drops support of any type of args.
Feb 25 2018
prev sibling parent reply kdevel <kdevel vogtner.de> writes:
On Sunday, 25 February 2018 at 11:46:26 UTC, Denis F wrote:

 But it is impossible to convert text :names or '?' into 
 Postgres's "$1": Postgres isn't knows fields names at start of 
 a query processing and you can't replace '?' to "$<num>" by 
 simple 'replace' call - it will need full syntax parsing of 
 Posgres SQL query because queries can contain EXECUTE statement 
 (whose purpose is executing dynamic commands).


You don't have to parse the full syntax. Quote from another 
language (which also supports underscore in integer literals):

'Placeholders

There are three types of placeholders that can be used in 
DBD::Pg. The first is the "question mark" type, in which each 
placeholder is represented by a single question mark character. 
This is the method recommended by the DBI specs and is the most 
portable. Each question mark is internally replaced by a "dollar 
sign number" in the order in which they appear in the query 
(important when using "bind_param").

The second type of placeholder is "dollar sign numbers". This is 
the method that Postgres uses internally and is overall probably 
the best method to use if you do not need compatibility with 
other database systems. DBD::Pg, like PostgreSQL, allows the same 
number to be used more than once in the query. Numbers must start 
with "1" and increment by one value (but can appear in any order 
within the query). If the same number appears more than once in a 
query, it is treated as a single parameter and all instances are 
replaced at once. [...]

The final placeholder type is "named parameters" in the format 
":foo". While this syntax is supported by DBD::Pg, its use is 
discouraged in favor of dollar-sign numbers.

The different types of placeholders cannot be mixed within a 
statement, but you may use different ones for each statement 
handle you have. This is confusing at best, so stick to one style 
within your program.

If your queries use operators that contain question marks (e.g. 
some of the native Postgres geometric operators and JSON 
operators) or array slices (e.g. data[100:300]), there are 
methods to instruct DBD::Pg to not treat some symbols as 
placeholders. First, you may simply add a backslash before the 
start of a placeholder, and DBD::Pg will strip the backslash and 
not treat the character as a placeholder.

You can also tell DBD::Pg to ignore any non-dollar sign 
placeholders by setting the pg_placeholder_dollaronly attribute 
at either the database handle or the statement handle level. 
[...] ' [1]

[1] 
http://search.cpan.org/~turnstep/DBD-Pg-3.7.4/Pg.pm#Placeholders
Feb 25 2018
parent Denis F <denis.feklushkin gmail.com> writes:
On Sunday, 25 February 2018 at 19:29:59 UTC, kdevel wrote:

 On Sunday, 25 February 2018 at 11:46:26 UTC, Denis F wrote:

 But it is impossible to convert text :names or '?' into 
 Postgres's "$1": Postgres isn't knows fields names at start of 
 a query processing and you can't replace '?' to "$<num>" by 
 simple 'replace' call - it will need full syntax parsing of 
 Posgres SQL query because queries can contain EXECUTE 
 statement (whose purpose is executing dynamic commands).


 You don't have to parse the full syntax.


You must if you want to make a replacement correctly for all 
types of statements.


 Quote from another language (which also supports underscore in 
 integer literals):


"This language" just drops off supporting of dynamic commands by 
using prepared statements for requests with arguments:

"WARNING: DBD::Pg now (as of version 1.40) uses true prepared 
statements by sending them to the backend to be prepared by the 
Postgres server. Statements that were legal before may no longer 
work."

It is trivial to replace any placeholders if you use only 
prepared statements, yes. But if you after that try to call 
procedure which calls EXECUTE IMMEDIATE command inside it will 
fail! How many of your users will understand why this happens?

Also I see too high level of abstraction in DBD::Pg. I am isn't 
add prepared statement wrappers into dpq2 because its 
implementation depends from async queries handling implementation.

dpq2 is "middle-level" - C calls and classes wrappers over some 
of it.
Feb 25 2018
prev sibling parent reply Joe <jma freedomcircle.com> writes:
On Sunday, 25 February 2018 at 09:23:18 UTC, rikki cattermole 
wrote:

 On IRC earlier today we discussed database libs a bit, we 
 agreed that both "?" and " name" needed to be supported. No 
 other suggestions came up. We don't really need a third or 
 fourth form I think.


" name"? Which DBMS uses that? Although Denis said Postgres uses 
$1, PG also supports :name, in either the command line tool 
(psql) and the procedural language PL/pgSQL.  Personally, I 
prefer what Python's PEP 249 calls 'format' (%s) or 'pyformat' 
(%(name)s).  Note that although the %s appears in the SQL string, 
the actual arguments are passed separately at invocation time, as 
either tuples or dicts (associative arrays).

Joe
Feb 25 2018
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 26/02/2018 5:06 AM, Joe wrote:

 On Sunday, 25 February 2018 at 09:23:18 UTC, rikki cattermole wrote:

 On IRC earlier today we discussed database libs a bit, we agreed that 
 both "?" and " name" needed to be supported. No other suggestions came 
 up. We don't really need a third or fourth form I think.


 
 " name"? Which DBMS uses that? Although Denis said Postgres uses $1, PG 
 also supports :name, in either the command line tool (psql) and the 
 procedural language PL/pgSQL.  Personally, I prefer what Python's PEP 
 249 calls 'format' (%s) or 'pyformat' (%(name)s).  Note that although 
 the %s appears in the SQL string, the actual arguments are passed 
 separately at invocation time, as either tuples or dicts (associative 
 arrays).
 
 Joe


ADO.net uses it, and since we're trying to copy some of its features, 
that is what came up.
Feb 25 2018
prev sibling parent reply Joe <jma freedomcircle.com> writes:
On Sunday, 25 February 2018 at 09:07:34 UTC, Denis F wrote:

 Libraries already are similar automatically because RDBMSes 
 uses similar principles. But it is impossible to make libraries 
 absolutely identical due to the presence of important 
 significant nuances in each engine.


I don't see how making libraries compatible with each other, 
i.e., complying with some standard, will result in losing 
functionality.  What I'm talking about is standardizing the names 
of certain classes and their methods, exceptions, and perhaps 
some datatypes and the expected order of certain arguments.  For 
example, instead of having classes named Database (sqlite3), 
PGConnection (ddb) and Connection (dpq2, mysql-native) there's 
just one name, and that class is expected to have certain 
methods. The library implementor can add others. For example, I 
see mysql-native has a connection pool and that doesn't need to 
be part of the standard. Also, some methods can be specified as 
being optional by the standard, e.g., something like callproc, 
because not all DBMSs support procedures.


 Ok, I propose to be consistent and ask for compliance with the 
 Standard from the RDBMSes. For example, arguments substitution:

 MySQL uses the '?'
 PostgreSQL uses $1
 SQLite accepts both
 Oracle uses a :name

 (Really, it is very important to come to an agreement here 
 because, for sure, the next obvious step is writing an ORM 
 generator on top of the idea what you are proposing.)


In PEP 249, this was left up to the implementors (see the 
paramstyle global), but a footnote suggests that some styles are 
supposed to be preferred over others.


 For example, I would not change unobvious at first sight system 
 of classes "Result" and inherited from it class "Answer" for 
 the sake of similarity to other libraries. Simply because this 
 is the way what Postgres client library works, and if this 
 classes will be removed some significant functionality will be 
 lost.
 I'm sure that in other libraries there is something similar.


I'm not sure I understand your example, but perhaps this is 
getting too specific and close to the discussions that would need 
to take place betweeen existing implementors.  There has to be a 
negotiation process for a standard to be developed.  In my 
example above, suppose the sqlite3 library developer says "I 
don't want to have a class named Connection, because my users 
don't connect to a server".  It will be up to the other 
developers to convince him or to arrive at some compromise (in 
the case of D an alias may be all that is needed to resolve such 
an impasse).

Joe
Feb 25 2018
parent reply Denis F <denis.feklushkin gmail.com> writes:
On Sunday, 25 February 2018 at 15:34:00 UTC, Joe wrote:

 On Sunday, 25 February 2018 at 09:07:34 UTC, Denis F wrote:

 Libraries already are similar automatically because RDBMSes 
 uses similar principles. But it is impossible to make 
 libraries absolutely identical due to the presence of 
 important significant nuances in each engine.


 I don't see how making libraries compatible with each other, 
 i.e., complying with some standard, will result in losing 
 functionality.  What I'm talking about is standardizing the 
 names of certain classes and their methods, exceptions, and 
 perhaps some datatypes and the expected order of certain 
 arguments.  For example, instead of having classes named 
 Database (sqlite3), PGConnection (ddb) and Connection (dpq2, 
 mysql-native) there's just one name, and that class is expected 
 to have certain methods.


I guess this is problem too: sqlite isn't connects at all, and 
its struct represents RDBMS itself. But other should represent 
only connections because they manipulate the connection and not 
controls its server, and connection is a potential source of 
connection-related exceptions. This is a subtle but important 
difference too.


 The library implementor can add others. For example, I see 
 mysql-native has a connection pool and that doesn't need to be 
 part of the standard. Also, some methods can be specified as 
 being optional by the standard, e.g., something like callproc, 
 because not all DBMSs support procedures.


This is another confirmation is that there are more differences 
than similarities  between databases. :-)


 Ok, I propose to be consistent and ask for compliance with the 
 Standard from the RDBMSes. For example, arguments substitution:

 MySQL uses the '?'
 PostgreSQL uses $1
 SQLite accepts both
 Oracle uses a :name

 (Really, it is very important to come to an agreement here 
 because, for sure, the next obvious step is writing an ORM 
 generator on top of the idea what you are proposing.)


 In PEP 249, this was left up to the implementors (see the 
 paramstyle global), but a footnote suggests that some styles 
 are supposed to be preferred over others.


 For example, I would not change unobvious at first sight 
 system of classes "Result" and inherited from it class 
 "Answer" for the sake of similarity to other libraries. Simply 
 because this is the way what Postgres client library works, 
 and if this classes will be removed some significant 
 functionality will be lost.
 I'm sure that in other libraries there is something similar.


 I'm not sure I understand your example, but perhaps this is 
 getting too specific and close to the discussions that would 
 need to take place betweeen existing implementors.  There has 
 to be a negotiation process for a standard to be developed.  In 
 my example above, suppose the sqlite3 library developer says "I 
 don't want to have a class named Connection, because my users 
 don't connect to a server".  It will be up to the other 
 developers to convince him or to arrive at some compromise


What are these sacrifices for? I do not like idea that I should 
drop some functionality of Postgres for conpatibility with 
another RDBMSes. Personally, I like specific databases because it 
have nuances of functionality that they provide to me.

And those who wish to "crossbreeding a hedgehog with a snake", at 
first, can try to write a wrapper around existing SQL libraries. 
This is faster and more humane in relation to the developers.
Feb 25 2018
parent reply Joe <jma freedomcircle.com> writes:
On Sunday, 25 February 2018 at 16:31:45 UTC, Denis F wrote:

 What are these sacrifices for? I do not like idea that I should 
 drop some functionality of Postgres for conpatibility with 
 another RDBMSes. Personally, I like specific databases because 
 it have nuances of functionality that they provide to me.


Nobody said or implied anything about dropping Postgres 
functionality. If you can, please take a look at the contents of 
the Psycopg docs (http://initd.org/psycopg/docs/), the most 
popular Python Postgres adapter: after the entries for the 
connection and cursor classes, you'll see pyscopg2.extensions, 
pyscopg2.extras, psycopg2.pool and more. It even includes support 
for the recently added logical replication feature.

The "sacrifice" benefits the library implementors because it 
expands their potential market.  Even though you decry the 
differences between RDBMS, when it comes to the basics, I would 
argue that 80% of the code written by a SQL DB developer is very 
similar: connect to the DBMS (or to a file for sqlite), issue a 
query, retrieve results, update some rows.  If they're writing in 
C, D, Python, Ruby or whatever, most of their time is spent 
writing or tweaking SQL statements.  We're not talking about 
MongoDB, Cassandra, OODBMS or other creatures that *do* have 
significant differences (it's telling that several have 
implemented SQL layers on top).


 And those who wish to "crossbreeding a hedgehog with a snake", 
 at first, can try to write a wrapper around existing SQL 
 libraries. This is faster and more humane in relation to the 
 developers.


Please take a look at the list of RDBMSs supported by SQLAlchemy 
(http://docs.sqlalchemy.org/en/latest/core/engines.html#supported-databases). 
I'm *not* in favor of ORMs, but do you think that Mike Bayer (the implementor)
would have had any incentive to write an ORM if a DB-API hadn't existed about
10 years before (v1 I believe is circa 1996, v2 from 1999, SQLA came out in
2006 and it already supported four DBMSs).  Even if he had had the incentive it
definitely would've been much slower and error-prone, because he would have had
to not only design and implement an ORM but also have to deal with
idiosyncracies of each library, whereas now he can support multiple libraries
for each DBMS, e.g., psycopg2 and pg8000 for Postgres, because they both
conform to the same API.

You're obviously reluctant of having to rewrite dpq2 just to 
"play along with the other kids".  I can understand that, but who 
knows, maybe your design is 90% of what the others want/have/like 
and you may only have to make some trivial changes here and there.

In any case, as I mentioned earlier, I will delve into dpq2 (and 
vibe-d-postgresql) for my own project.  So I'll keep in touch.

Joe
Feb 25 2018
parent Denis F <denis.feklushkin gmail.com> writes:
On Sunday, 25 February 2018 at 20:56:16 UTC, Joe wrote:

 On Sunday, 25 February 2018 at 16:31:45 UTC, Denis F wrote:

 What are these sacrifices for? I do not like idea that I 
 should drop some functionality of Postgres for conpatibility 
 with another RDBMSes. Personally, I like specific databases 
 because it have nuances of functionality that they provide to 
 me.


 Nobody said or implied anything about dropping Postgres 
 functionality. If you can, please take a look at the contents 
 of the Psycopg docs (http://initd.org/psycopg/docs/), the most 
 popular Python Postgres adapter: after the entries for the 
 connection and cursor classes, you'll see pyscopg2.extensions, 
 pyscopg2.extras, psycopg2.pool and more. It even includes 
 support for the recently added logical replication feature.


pyscopg2 passes request arguments as strings inside of it! This 
degrades type safety because in D we have types.

Looks like it is uses C format:

cur.execute("SELECT '%s', %s", (123,456,)) -- first arg expected 
as "%s" string
cur.fetchone()
('123', 456)


 You're obviously reluctant of having to rewrite dpq2 just to 
 "play along with the other kids".  I can understand that, but 
 who knows, maybe your design is 90% of what the others 
 want/have/like


No, sure. dpq2 is too lowlevel for this. Users need transactions, 
prepared statements wrappers, etc. But I can't implement it in 
dpq2 because these things is depedend from other more high level 
implementations.

At least, for dpk2, I clearly see that the wrapper is the best 
choice.
Feb 25 2018
prev sibling next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:

 Back on 13 January, I posted in the Learn forum some questions 
 regarding using Postgres and got a reply that stated the 
 following:

 On Monday, 15 January 2018 at 02:28:29 UTC, Matthias Klumpp 
 wrote:

 In any case, please don't start another Postgres library and 
 consider contributing to one of the existing ones, so that we 
 maybe have one really awesome, 100% complete library at some 
 point.


 I'm revisiting this because I now have had the chance to look 
 at the various libraries and IMHO the "really awesome, 100% 
 complete library" depends on the users' goals and their 
 definition of awesome and complete.

 Aside from ddbc which is like ODBC/JDBC and thus supports 
 multiple db's (not my interest), I've found five 
 Postgres-specific libraries worthy of further exploration:

 1. derelict-pq. This is the most downloaded one and is simply a 
 binding over libpq, so AFAICT it's nearly 100% complete. It's 
 also well-documented, in part because the Derelict libraries 
 provide overall guidance, but mainly because a D user can refer 
 directly to the comprehensive libpq documentation and (C) 
 examples. The only strange fact is that it's been stuck in a 
 beta.3 release for the last five months.

 2. dpq2. Second most downloaded and built on top of 
 derelict-pq. The "documentation" consists of a README listing 
 the features and a single example, which appears to focus on 
 support for JSON/BSON.

 3. vibe-d-postgresql. Third most downloaded and built on top of 
 dpq2. Docs consist of a single example program.

 4. ddb. About the same number of downloads as the above. 
 Implemented on top of front/backend protocol. No documentation 
 although repository has a folder with two sample programs.

 5. dpq. Implements its own wrapper over libpq and has some ORM 
 functionality. Docs are a README with example program.

 The main issue is that, other than derelict-pq, using any of 
 these libraries involves reading the library code and 
 understanding the sui generis interfaces implemented by each.

 I'm new to the D community, but coming from Python, the 
 contrast is significant. First there is the well-documented PEP 
 249 (https://www.python.org/dev/peps/pep-0249/) promulgated by 
 the Python DB-SIG (and note that 249 is v.2), which led to the 
 implementation of psycopg (also well-documented including 
 various extensions to the API) and many other adapters to 
 Postgres and other databases.


Some developers spend hours writing code but don't feel the need 
to speed minutes documenting or at least showing how to use their 
code. Part of the reason others roll their own (one they will 
understand).  Goes on and on.

If its for their personal use,  then they shouldn't put it on dub 
to saturate the ecosystem.

I sometimes go around sending pull requests to some with sample I 
got from their unittests or "example"  folder. Worst case,  they 
send you a link to the C library they wrote bindings for... "oh 
its easy to know by looking at the C docs". Some don't even 
border... this is common with C libraries. C has libs but hard to 
figure out how to use without going through code.

Laugh at,  as one may,  Javascript npm libraries are well 
documented so there's high adoption. OP even agrees derelict-pq 
is well documented so it has a higher usage. All Dub libraries 
with good docs are those that are downloaded. Most importantly, a 
description of package and a sample/demo is the README.md file. A 
link to generated docs may follow to more info. Only generated 
docs don't cut it.


If you document your code well,  others will use and contribute 
to it without rolling their own. Docs on library purpose and 
sample usage at least.


 "The most used Dub libraries are those well documented."
Feb 23 2018
next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Saturday, 24 February 2018 at 07:57:47 UTC, aberba wrote:

 On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:



 4. ddb. About the same number of downloads as the above. 
 Implemented on top of front/backend protocol. No documentation 
 although repository has a folder with two sample programs.





 Some developers spend hours writing code but don't feel the 
 need to speed minutes documenting or at least showing how to 
 use their code. Part of the reason others roll their own (one 
 they will understand).  Goes on and on.


Have you tried to generate the documentation?

Look for example inside the source files...

https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/postgres.d#L2

https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/db.d#L19

/Paolo
Feb 24 2018
parent reply aberba <karabutaworld gmail.com> writes:
On Saturday, 24 February 2018 at 08:32:38 UTC, Paolo Invernizzi 
wrote:

 On Saturday, 24 February 2018 at 07:57:47 UTC, aberba wrote:

 On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:



 4. ddb. About the same number of downloads as the above. 
 Implemented on top of front/backend protocol. No 
 documentation although repository has a folder with two 
 sample programs.





 Some developers spend hours writing code but don't feel the 
 need to speed minutes documenting or at least showing how to 
 use their code. Part of the reason others roll their own (one 
 they will understand).  Goes on and on.


 Have you tried to generate the documentation?

 Look for example inside the source files...

 https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/postgres.d#L2

 https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/db.d#L19

 /Paolo


Worst case answer I just talked about
Feb 24 2018
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Saturday, 24 February 2018 at 09:10:03 UTC, aberba wrote:

 On Saturday, 24 February 2018 at 08:32:38 UTC, Paolo Invernizzi 
 wrote:

 On Saturday, 24 February 2018 at 07:57:47 UTC, aberba wrote:

 On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:



 4. ddb. About the same number of downloads as the above. 
 Implemented on top of front/backend protocol. No 
 documentation although repository has a folder with two 
 sample programs.





 Some developers spend hours writing code but don't feel the 
 need to speed minutes documenting or at least showing how to 
 use their code. Part of the reason others roll their own (one 
 they will understand).  Goes on and on.


 Have you tried to generate the documentation?

 Look for example inside the source files...

 https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/postgres.d#L2

 https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/db.d#L19

 /Paolo


 Worst case answer I just talked about


I think no...

The documentation on how to use the package is present, it's just 
inside the modules as Ddoc documentation.

Joe, you can easily 'extract' it using the proper -D compiler 
switch

/Paolo
Feb 24 2018
parent reply Joe <jma freedomcircle.com> writes:
On Saturday, 24 February 2018 at 10:13:35 UTC, Paolo Invernizzi 
wrote:

 Have you tried to generate the documentation?

 Look for example inside the source files...

 https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/postgres.d#L2

 https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/db.d#L19




 I think no...

 The documentation on how to use the package is present, it's 
 just inside the modules as Ddoc documentation.

 Joe, you can easily 'extract' it using the proper -D compiler 
 switch


As I said, I'm new and still not familiar with those facilities 
although I saw those things and they looked like embedded HTML 
and was wondering what they were for.

Again, coming from Python, I'm familiar with RTD 
(https://readthedocs.org, my own open source Python/Postgres 
project is hosted there) and Sphinx 
(http://www.sphinx-doc.org/en/master/). Although it started with 
Python, RTD now hosts many other kinds of projects (Javascript, 
PHP, Ruby, etc.). There's even a project named 'dlang' (it's not 
what you expect!), but I did find "D Tips" and "Quick Start with 
D" which appear to be standalone Markdown. In any case, it would 
be nice to have such automatically generated docs available in a 
public site/repo of some kind, like RTD, so a potential user 
doesn't have to clone the code and run the compiler to see it.
Feb 24 2018
next sibling parent reply Martin Tschierschke <mt smartdolphin.de> writes:
On Saturday, 24 February 2018 at 14:13:18 UTC, Joe wrote:

 On Saturday, 24 February 2018 at 10:13:35 UTC, Paolo Invernizzi


[...]

 As I said, I'm new and still not familiar with those facilities 
 although I saw those things and they looked like embedded HTML 
 and was wondering what they were for.

 Again, coming from Python, I'm familiar with RTD 
 (https://readthedocs.org, my own open source Python/Postgres 
 project is hosted there) and Sphinx 
 (http://www.sphinx-doc.org/en/master/). Although it started 
 with Python, RTD now hosts many other kinds of projects 
 (Javascript, PHP, Ruby, etc.). There's even a project named 
 'dlang' (it's not what you expect!), but I did find "D Tips" 
 and "Quick Start with D" which appear to be standalone 
 Markdown. In any case, it would be nice to have such 
 automatically generated docs available in a public site/repo of 
 some kind, like RTD, so a potential user doesn't have to clone 
 the code and run the compiler to see it.


+1
I would even go so far 'force' people publishing to dub, to 
provide documentation.
If no docs, present, than the libs should be marked as *docs 
missing*. (Beside the number of Github stars)
Feb 24 2018
parent bachmeier <no spam.net> writes:
On Saturday, 24 February 2018 at 14:54:21 UTC, Martin 
Tschierschke wrote:


 I would even go so far 'force' people publishing to dub, to 
 provide documentation.
 If no docs, present, than the libs should be marked as *docs 
 missing*. (Beside the number of Github stars)


If there's no documentation, the project has no business 
cluttering up the website, and should not be published on Dub.
Feb 24 2018
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 24 February 2018 at 14:13:18 UTC, Joe wrote:

 Again, coming from Python, I'm familiar with RTD


So I actually just made this extension to my dpldocs website:

http://ddb.dpldocs.info/ddb.postgres.html


You can try going to

http://any-dub-package.dpldocs.info

and it will try to build the docs on demand.

for example:
http://dpq.dpldocs.info/dpq.html



If you're the first user to hit a page, it might take some time 
to load and send you to a random page once generation is 
complete, but then you can navigate with the sidebar to see the 
rest.


I literally just slapped this together now, so I expect it won't 
be perfect, but this has been on my todo list for a year (I even 
bought a VPS to host it on... 6 months ago and have been paying 
for it without even using it!), so it was time to do.
Feb 24 2018
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Saturday, 24 February 2018 at 15:32:32 UTC, Adam D. Ruppe 
wrote:

 On Saturday, 24 February 2018 at 14:13:18 UTC, Joe wrote:

 Again, coming from Python, I'm familiar with RTD


 So I actually just made this extension to my dpldocs website:

 http://ddb.dpldocs.info/ddb.postgres.html


 You can try going to

 http://any-dub-package.dpldocs.info

 and it will try to build the docs on demand.

 for example:
 http://dpq.dpldocs.info/dpq.html



 If you're the first user to hit a page, it might take some time 
 to load and send you to a random page once generation is 
 complete, but then you can navigate with the sidebar to see the 
 rest.


 I literally just slapped this together now, so I expect it 
 won't be perfect, but this has been on my todo list for a year 
 (I even bought a VPS to host it on... 6 months ago and have 
 been paying for it without even using it!), so it was time to 
 do.


:-O

Adam, you are the man!
I'll never stop of being surprised by your way of coding!

Great Job!

Joe: I think this is also a terrific 'welcome aboard' about how 
fast and quickly things can be done in D!

:-P

/Paolo
Feb 24 2018
next sibling parent reply Martin Tschierschke <mt smartdolphin.de> writes:
On Saturday, 24 February 2018 at 15:44:49 UTC, Paolo Invernizzi 
wrote:

 On Saturday, 24 February 2018 at 15:32:32 UTC, Adam D. Ruppe 
 wrote:

 On Saturday, 24 February 2018 at 14:13:18 UTC, Joe wrote:




[...]

 You can try going to

 http://any-dub-package.dpldocs.info




[...]

 :-O

 Adam, you are the man!
 I'll never stop of being surprised by your way of coding!

 Great Job!


+1
Fascinating!

Please make a post to announce and place the direct link to it 
inside code.dlang.org :-)

Would it be possible to use it for calculating/evaluating a 
measure for inlined documentation?
Feb 24 2018
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 24 February 2018 at 16:41:09 UTC, Martin 
Tschierschke wrote:

 Please make a post to announce and place the direct link to it 
 inside code.dlang.org :-)


Well, I forgot to log errors on the server so I see a few 
generation failures and I'm not sure if it is because the code is 
missing docs or if one of my components failed! I'll have to look 
at the packages in question.

But yeah, I'd like to have the link on code.dlang.org so people 
can discover it. Let's set a goal of having it ready for that 
next week.


 Would it be possible to use it for calculating/evaluating a 
 measure for inlined documentation?


Yes, certainly it can tell if it is there or not at least really 
easily. But if it is not there because it is an internal module 
and supposed to be not publicly documented or poorly done is 
harder to tell without some kind of user intervention.

Ideally, of course, I would ascend to godhood and force all 
packages to explicitly declare internal stuff so it can be 
automatically analyzed...
Feb 24 2018
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 24 February 2018 at 15:44:49 UTC, Paolo Invernizzi 
wrote:

 :-O

 Adam, you are the man!


worship me!


So I'm gonna tell you guys a dirty little secret: I intend to 
take the documentation throne by means of subterfuge. I've been 
generating docs for popular packages already but it has been a 
manual process (I, last night, did some hacks to better support 
gtkd than anyone else I have see: 
http://dpldocs.info/experimental-docs/gtk.ApplicationWindow.Ap
licationWindow.html i'm about 80% happy with it.)

The new thing here is just a little program to automate that 
process when the URL is fetched. But it doesn't do versioning or 
cross-package referencing or a few of the other things I have 
planned.... notably, it also ignores the examples/ folders, which 
are a really useful source (and btw my doc gen web source viewer 
has go-to-definition omg!)


Still, it is fairly usable as-is and I hope it encourages people 
to do more dox. And I hope it gets them to use my syntax and 
features: 
http://dpldocs.info/experimental-docs/adrdox.syntax.html that i 
may finally take the crown and rule the D-language area!!!!!!!!11 
muahhahah
Feb 24 2018
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 24 February 2018 at 16:42:26 UTC, Adam D. Ruppe 
wrote:

 source viewer


This now works:
http://dpq.dpldocs.info/source/dpq.query.d.html#L29

the "see implementation" links lead to pages like that. You'll 
notice in the source view that many names are links. You can 
click on them to go to their definition too. The sidebar has docs 
links, if applicable.

Note that the main dpldocs.info site has this for Phobos too 
http://dpldocs.info/experimental-docs/source/std.socket.d.html#L2593 thought I
don't update the source often there so it can get out of sync (Phobos is a bit
big to rebuild frequently)
Feb 24 2018
prev sibling parent Joe <jma freedomcircle.com> writes:
On Saturday, 24 February 2018 at 15:44:49 UTC, Paolo Invernizzi 
wrote:

 Joe: I think this is also a terrific 'welcome aboard' about how 
 fast and quickly things can be done in D!


I'm a bit overwhelmed by all the replies and "goodies" but I'd 
like to say thanks to Adam while I digest it all.

Joe
Feb 24 2018
prev sibling parent Martin Tschierschke <mt smartdolphin.de> writes:
On Saturday, 24 February 2018 at 07:57:47 UTC, aberba wrote:

 On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:

 Back on 13 January, I posted in the Learn forum some questions 
 regarding using Postgres and got a reply that stated the 
 following:

 On Monday, 15 January 2018 at 02:28:29 UTC, Matthias Klumpp 
 wrote:

 In any case, please don't start another Postgres library and 
 consider contributing to one of the existing ones, so that we 
 maybe have one really awesome, 100% complete library at some 
 point.


 I'm revisiting this because I now have had the chance to look 
 at the various libraries and IMHO the "really awesome, 100% 
 complete library" depends on the users' goals and their 
 definition of awesome and complete.

 Aside from ddbc which is like ODBC/JDBC and thus supports 
 multiple db's (not my interest), I've found five 
 Postgres-specific libraries worthy of further exploration:

 1. derelict-pq. This is the most downloaded one and is simply 
 a binding over libpq, so AFAICT it's nearly 100% complete. 
 It's also well-documented, in part because the Derelict 
 libraries provide overall guidance, but mainly because a D 
 user can refer directly to the comprehensive libpq 
 documentation and (C) examples. The only strange fact is that 
 it's been stuck in a beta.3 release for the last five months.

 2. dpq2. Second most downloaded and built on top of 
 derelict-pq. The "documentation" consists of a README listing 
 the features and a single example, which appears to focus on 
 support for JSON/BSON.

 3. vibe-d-postgresql. Third most downloaded and built on top 
 of dpq2. Docs consist of a single example program.

 4. ddb. About the same number of downloads as the above. 
 Implemented on top of front/backend protocol. No documentation 
 although repository has a folder with two sample programs.

 5. dpq. Implements its own wrapper over libpq and has some ORM 
 functionality. Docs are a README with example program.

 The main issue is that, other than derelict-pq, using any of 
 these libraries involves reading the library code and 
 understanding the sui generis interfaces implemented by each.

 I'm new to the D community, but coming from Python, the 
 contrast is significant. First there is the well-documented 
 PEP 249 (https://www.python.org/dev/peps/pep-0249/) 
 promulgated by the Python DB-SIG (and note that 249 is v.2), 
 which led to the implementation of psycopg (also 
 well-documented including various extensions to the API) and 
 many other adapters to Postgres and other databases.


 Some developers spend hours writing code but don't feel the 
 need to speed minutes documenting or at least showing how to 
 use their code. Part of the reason others roll their own (one 
 they will understand).  Goes on and on.

 If its for their personal use,  then they shouldn't put it on 
 dub to saturate the ecosystem.

 I sometimes go around sending pull requests to some with sample 
 I got from their unittests or "example"  folder. Worst case,  
 they send you a link to the C library they wrote bindings 
 for... "oh its easy to know by looking at the C docs". Some 
 don't even border... this is common with C libraries. C has 
 libs but hard to figure out how to use without going through 
 code.

 Laugh at,  as one may,  Javascript npm libraries are well 
 documented so there's high adoption. OP even agrees derelict-pq 
 is well documented so it has a higher usage. All Dub libraries 
 with good docs are those that are downloaded. Most importantly, 
 a description of package and a sample/demo is the README.md 
 file. A link to generated docs may follow to more info. Only 
 generated docs don't cut it.


 If you document your code well,  others will use and contribute 
 to it without rolling their own. Docs on library purpose and 
 sample usage at least.


 "The most used Dub libraries are those well documented."




+1
I want to subscribe to this.
To improve the D ecosystem it would be a plus point to mark all 
libs at code.dlang.org having documentation with a special sign.

Most people want a solution for their problems, they don't want 
to make an analysis,
which of n available libs is best to use. Most more widely 
adopted languages offer
the needed DB connectors in their std.

The medium step between expanding the std lib and having nothing 
official might be,
to try to adopt more for the dlang git repro. 
(https://github.com/dlang-community)
(As mentioned before)

Even if the first choice might be not the best of all available 
libs for one problem,
if the community has started to develop _one_ solution for a 
problem, the value of contributing to this increases 
dramatically. I think the community is not big enough to support 
five different well documented, well designed and understood libs 
for solving the same problem.
Regards mt.
Feb 24 2018
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:

 3. vibe-d-postgresql. Third most downloaded and built on top of 
 dpq2. Docs consist of a single example program.


If you write docs and examples, I'd expect vibe to accept them, 
but ask them anyway.
Feb 24 2018
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:

 The main issue is that, other than derelict-pq, using any of 
 these libraries involves reading the library code and 
 understanding the sui generis interfaces implemented by each.


So often in threads like this I chime in to point out my lib 
too... but ironically, it is poorly documented :P

http://dpldocs.info/experimental-docs/arsd.postgres.PostgreSql.html

Of course, my lib is so thin that there isn't much to document 
anyway (the source code is ~200 lines)
Feb 24 2018
prev sibling parent reply Denis F <denis.feklushkin gmail.com> writes:
On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:


 2. dpq2. Second most downloaded and built on top of 
 derelict-pq. The "documentation" consists of a README listing 
 the features and a single example, which appears to focus on 
 support for JSON/BSON.


Hi! Here is dpq2 and vibe-d-postgresql developer.

The problem is that I do not know English very well. So,I think 
it's better not to write any documentation than to write the 
wrong one.

PRs are welcomed!


 3. vibe-d-postgresql. Third most downloaded and built on top of 
 dpq2. Docs consist of a single example program.
Feb 24 2018
next sibling parent Joe <jma freedomcircle.com> writes:
On Saturday, 24 February 2018 at 18:41:21 UTC, Denis F wrote:

 On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote:


 2. dpq2. Second most downloaded and built on top of 
 derelict-pq. The "documentation" consists of a README listing 
 the features and a single example, which appears to focus on 
 support for JSON/BSON.


 Hi! Here is dpq2 and vibe-d-postgresql developer.

 The problem is that I do not know English very well. So,I think 
 it's better not to write any documentation than to write the 
 wrong one.

 PRs are welcomed!


 3. vibe-d-postgresql. Third most downloaded and built on top 
 of dpq2. Docs consist of a single example program.




Well Denis, you may get your PR wish (but I don't have a 
timeframe). My interest is to develop something using Vibe.d, so 
I'll be delving more into these two first.

Joe
Feb 24 2018
prev sibling parent reply bachmeier <no spam.net> writes:
On Saturday, 24 February 2018 at 18:41:21 UTC, Denis F wrote:


 Hi! Here is dpq2 and vibe-d-postgresql developer.

 The problem is that I do not know English very well. So,I think 
 it's better not to write any documentation than to write the 
 wrong one.


There are still some steps you can take:

1. Put a statement on the project page encouraging people to 
create an issue if there is something not documented. Even 
better, have a Gitter community where they can post questions. 
You can create an example to show how it's done. Those examples 
can then be added to the project.

2. Decide on the common use cases for your library. Provide 
examples useful to someone getting started for each of the common 
use cases.

3. Looking at the source code, you have already written a bunch 
of comments. Those are documentation once you turn them into html.
Feb 24 2018
parent reply Denis F <denis.feklushkin gmail.com> writes:
On Saturday, 24 February 2018 at 22:24:58 UTC, bachmeier wrote:

 On Saturday, 24 February 2018 at 18:41:21 UTC, Denis F wrote:


 Hi! Here is dpq2 and vibe-d-postgresql developer.

 The problem is that I do not know English very well. So,I 
 think it's better not to write any documentation than to write 
 the wrong one.


 There are still some steps you can take:

 1. Put a statement on the project page encouraging people to 
 create an issue if there is something not documented. Even 
 better, have a Gitter community where they can post questions. 
 You can create an example to show how it's done. Those examples 
 can then be added to the project.


Done!


 2. Decide on the common use cases for your library. Provide 
 examples useful to someone getting started for each of the 
 common use cases.


What is a bad with example?


 3. Looking at the source code, you have already written a bunch 
 of comments. Those are documentation once you turn them into 
 html.


I'm for the comments to be documentation. But I am paranoid and 
do not like that the  documentation generator should be granted 
write access to the repository.
Feb 24 2018
parent reply Seb <seb wilzba.ch> writes:
On Saturday, 24 February 2018 at 23:17:46 UTC, Denis F wrote:

 On Saturday, 24 February 2018 at 22:24:58 UTC, bachmeier wrote:

 3. Looking at the source code, you have already written a 
 bunch of comments. Those are documentation once you turn them 
 into html.


 I'm for the comments to be documentation. But I am paranoid and 
 do not like that the  documentation generator should be granted 
 write access to the repository.


If you talk about using Travis to build the documentation and 
push to gh-pages like e.g explained here [1]. There are solutions:

a) Use branch protection (GH offers this. No one can force push 
something to protected branches)
b) Use a dedicated bot account (if you are worried about leaking 
the credentials)
c) Use a dedicated *-docs repo to which Travis pushes
d) Push to a different location (e.g. S3, netlify
...

[1] 
https://forum.dlang.org/post/tkcndtapfypabsncxxla forum.dlang.org
Feb 24 2018
parent Denis F <denis.feklushkin gmail.com> writes:
On Sunday, 25 February 2018 at 03:20:08 UTC, Seb wrote:


 If you talk about using Travis to build the documentation and 
 push to gh-pages like e.g explained here [1]. There are 
 solutions:

 a) Use branch protection (GH offers this. No one can force push 
 something to protected branches)
 b) Use a dedicated bot account (if you are worried about 
 leaking the credentials)
 c) Use a dedicated *-docs repo to which Travis pushes
 d) Push to a different location (e.g. S3, netlify
 ...

 [1] 
 https://forum.dlang.org/post/tkcndtapfypabsncxxla forum.dlang.org


Thank! I will try to solve this
Feb 25 2018