www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A suggestion for dub

reply mahdi <xemailpro yahoo.co.uk> writes:
I just want to get some feedback to see if this is a good idea or 
no.

I propose adding a new command to dub (`dub dist` or `dub 
package`) which will produce a zipped version of the current 
package. This zip file can be uploaded to a central repository 
(code.dlang.com maybe), and others can use this package in their 
`dub.json` file by adding relevant dependency (Behind the scene, 
dub will fetch and decompress the package).
Feb 28 2016
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 29/02/16 5:46 PM, mahdi wrote:

 I just want to get some feedback to see if this is a good idea or no.

 I propose adding a new command to dub (`dub dist` or `dub package`)
 which will produce a zipped version of the current package. This zip
 file can be uploaded to a central repository (code.dlang.com maybe), and
 others can use this package in their `dub.json` file by adding relevant
 dependency (Behind the scene, dub will fetch and decompress the package).


Most of the work is already done.
code.dlang.org already grabs a zip of the contents of the repo from e.g. 
Github.
Dub then just downloads + extracts it. However I don't think its cached 
on code.dlang.org side.

Basically the hard part is writing a new provider and adding the ability 
to upload via api.


Of course the implementation may have changed in the last year or so.
Feb 28 2016
prev sibling next sibling parent reply Brad Anderson <eco gnuk.net> writes:
On Monday, 29 February 2016 at 04:46:13 UTC, mahdi wrote:

 I just want to get some feedback to see if this is a good idea 
 or no.

 I propose adding a new command to dub (`dub dist` or `dub 
 package`) which will produce a zipped version of the current 
 package. This zip file can be uploaded to a central repository 
 (code.dlang.com maybe), and others can use this package in 
 their `dub.json` file by adding relevant dependency (Behind the 
 scene, dub will fetch and decompress the package).


A neat feature would be to deploy it using the github releases 
feature. There is an API for it: 
https://developer.github.com/v3/repos/releases/

Being able to upload program binaries for end users 
(non-programmers) to use would be killer.
Feb 28 2016
parent reply mahdi <xemailpro yahoo.co.uk> writes:
On Monday, 29 February 2016 at 05:16:50 UTC, Brad Anderson wrote:

 On Monday, 29 February 2016 at 04:46:13 UTC, mahdi wrote:

 I just want to get some feedback to see if this is a good idea 
 or no.

 I propose adding a new command to dub (`dub dist` or `dub 
 package`) which will produce a zipped version of the current 
 package. This zip file can be uploaded to a central repository 
 (code.dlang.com maybe), and others can use this package in 
 their `dub.json` file by adding relevant dependency (Behind 
 the scene, dub will fetch and decompress the package).


 A neat feature would be to deploy it using the github releases 
 feature. There is an API for it: 
 https://developer.github.com/v3/repos/releases/

 Being able to upload program binaries for end users 
 (non-programmers) to use would be killer.


But definitely those binaries won't be cross-platform! That's why 
I think uploading/downloading source code is more robust.
Feb 28 2016
parent Brad Anderson <eco gnuk.net> writes:
On Monday, 29 February 2016 at 05:59:56 UTC, mahdi wrote:

 On Monday, 29 February 2016 at 05:16:50 UTC, Brad Anderson 
 wrote:

 On Monday, 29 February 2016 at 04:46:13 UTC, mahdi wrote:

 [...]


 A neat feature would be to deploy it using the github releases 
 feature. There is an API for it: 
 https://developer.github.com/v3/repos/releases/

 Being able to upload program binaries for end users 
 (non-programmers) to use would be killer.


 But definitely those binaries won't be cross-platform! That's 
 why I think uploading/downloading source code is more robust.


You can upload multiple platform binaries. I'm pretty sure LDC or 
GDC can do some cross compiling too. It's not reasonable to 
expect all end users to compile a program.
Feb 28 2016
prev sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Mon, 29 Feb 2016 04:46:13 +0000, mahdi wrote:


 I just want to get some feedback to see if this is a good idea or no.
 
 I propose adding a new command to dub (`dub dist` or `dub package`)
 which will produce a zipped version of the current package. This zip
 file can be uploaded to a central repository (code.dlang.com maybe), and
 others can use this package in their `dub.json` file by adding relevant
 dependency (Behind the scene, dub will fetch and decompress the
 package).


In other words, you want to distribute your source code, but you don't 
want to host a publicly accessibly git repository and don't want to use 
any of the free git hosting services around.

I'm not sure how this would be an advantage. You don't have to learn git, 
I guess? It's less VCS lockin? Though it probably wouldn't be so hard to 
add mercurial and svn plugins for dub.

Something that *would* be useful is a command that produces binaries and, 
in the case of libraries, D interface files so people can depend on the 
library even if the full source code isn't available. The obvious 
problem: D is typically compiled to native code. That means you have to 
compile once for each platform. Also, you need a different binary for 
every set of version() flags the thing supports. That makes binary 
distribution tricky in the general case.
Feb 28 2016
parent reply mahdi <xemailpro yahoo.co.uk> writes:
On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright wrote:

 On Mon, 29 Feb 2016 04:46:13 +0000, mahdi wrote:


 [...]


 In other words, you want to distribute your source code, but 
 you don't want to host a publicly accessibly git repository and 
 don't want to use any of the free git hosting services around.

 I'm not sure how this would be an advantage. You don't have to 
 learn git, I guess? It's less VCS lockin? Though it probably 
 wouldn't be so hard to add mercurial and svn plugins for dub.

 Something that *would* be useful is a command that produces 
 binaries and, in the case of libraries, D interface files so 
 people can depend on the library even if the full source code 
 isn't available. The obvious problem: D is typically compiled 
 to native code. That means you have to compile once for each 
 platform. Also, you need a different binary for every set of 
 version() flags the thing supports. That makes binary 
 distribution tricky in the general case.


The aim is to make package distribution easier and more 
straightforward. If someone has done some development on his 
local machine and wants to distribute it, it's better not to 
force him to create a repo on Github or ... .
Feb 29 2016
parent reply sigod <sigod.mail gmail.com> writes:
On Monday, 29 February 2016 at 08:02:51 UTC, mahdi wrote:

 On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright wrote:

 [...]


 The aim is to make package distribution easier and more 
 straightforward. If someone has done some development on his 
 local machine and wants to distribute it, it's better not to 
 force him to create a repo on Github or ... .


And you end up creating your own repository hosting. Or you would 
allow users to not use (D)VCS. Which will end up as a nightmare.
Feb 29 2016
next sibling parent mahdi <xemailpro yahoo.co.uk> writes:
On Monday, 29 February 2016 at 09:27:50 UTC, sigod wrote:

 On Monday, 29 February 2016 at 08:02:51 UTC, mahdi wrote:

 On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright 
 wrote:

 [...]


 The aim is to make package distribution easier and more 
 straightforward. If someone has done some development on his 
 local machine and wants to distribute it, it's better not to 
 force him to create a repo on Github or ... .


 And you end up creating your own repository hosting. Or you 
 would allow users to not use (D)VCS. Which will end up as a 
 nightmare.


Well, not necessarily. The benefit of this method is that it 
separates source code hosting (local, github, ...) from 
distribution.

This is the strategy that is followed in Python (PyPI) and Perl 
(CPAN) community.
Feb 29 2016
prev sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 29 February 2016 at 09:27:50 UTC, sigod wrote:

 On Monday, 29 February 2016 at 08:02:51 UTC, mahdi wrote:

 On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright 
 wrote:

 [...]


 The aim is to make package distribution easier and more 
 straightforward. If someone has done some development on his 
 local machine and wants to distribute it, it's better not to 
 force him to create a repo on Github or ... .


 And you end up creating your own repository hosting. Or you 
 would allow users to not use (D)VCS. Which will end up as a 
 nightmare.


Yeah that would be a nightmare. Plus the storage requirements. 
Better have github/bitbucket do it for you.

The thing I do run into occasionally is when I am modifying two 
dub package, where one depends on the other, and I am pushing a 
new version/tag to github. It generally takes a while before 
code.dlang.org picks up on new versions being available, and 
until that time I cannot compile the other package which depends 
on the new version.

Locally I can do `dub add-local`, but I don't want to do that on 
the CI, nor on other developers machine whenever they get my 
updated dub.json.

Granted, it doesn't take that long, but seeing how fast other 
services pick up on github/bitbucket changes (wercker starts 
building a mere 0.5 secs after a push), I wonder why 
code.dlang.org is being so slow about it.
Feb 29 2016
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 29.02.2016 um 10:46 schrieb Sebastiaan Koppe:

 On Monday, 29 February 2016 at 09:27:50 UTC, sigod wrote:

 On Monday, 29 February 2016 at 08:02:51 UTC, mahdi wrote:

 On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright wrote:

 [...]


 The aim is to make package distribution easier and more
 straightforward. If someone has done some development on his local
 machine and wants to distribute it, it's better not to force him to
 create a repo on Github or ... .


 And you end up creating your own repository hosting. Or you would
 allow users to not use (D)VCS. Which will end up as a nightmare.


 Yeah that would be a nightmare. Plus the storage requirements. Better
 have github/bitbucket do it for you.


Storage requirements would definitely be a problem. Hosting a database 
with metadata is one thing, but uploading potentially hundreds of 
megabytes or even gigabytes per package is another. The metadata alone 
takes up about 1.5 GiB right now. Should we ever grow as much as other 
languages have in terms of package count, hosting package contents is 
definitely out of the question without some sort of corporate investment.


 The thing I do run into occasionally is when I am modifying two dub
 package, where one depends on the other, and I am pushing a new
 version/tag to github. It generally takes a while before code.dlang.org
 picks up on new versions being available, and until that time I cannot
 compile the other package which depends on the new version.

 Locally I can do `dub add-local`, but I don't want to do that on the CI,
 nor on other developers machine whenever they get my updated dub.json.

 Granted, it doesn't take that long, but seeing how fast other services
 pick up on github/bitbucket changes (wercker starts building a mere 0.5
 secs after a push), I wonder why code.dlang.org is being so slow about it.


Currently all packages are polled for changes every 30 minutes.

We will at some point have to implement the use of push notifications 
for changes, but that requires some work (supporting GitHub-OAuth and 
implementing the push endpoint) and it requires the repositories to be 
re-registered with OAuth.

BTW, there is a "Trigger manual update" button on the package management 
page. If you are the owner, you can usually get instant updates that way.
Feb 29 2016
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 29 February 2016 at 10:07:56 UTC, Sönke Ludwig wrote:

 We will at some point have to implement the use of push 
 notifications for changes, but that requires some work 
 (supporting GitHub-OAuth and implementing the push endpoint) 
 and it requires the repositories to be re-registered with OAuth.


Does code.dlang.org have a public repo? I could help out with 
some impl. details.


 BTW, there is a "Trigger manual update" button on the package 
 management page. If you are the owner, you can usually get 
 instant updates that way.


My experience with that button has been mixed.
Feb 29 2016
next sibling parent Mike Parker <aldacron gmail.com> writes:
On Monday, 29 February 2016 at 15:09:00 UTC, Sebastiaan Koppe 
wrote:


 BTW, there is a "Trigger manual update" button on the package 
 management page. If you are the owner, you can usually get 
 instant updates that way.


 My experience with that button has been mixed.


How so? If the package is not already queued for an update, 
pressing the button queues it up. It's not a 'update right now' 
sort of thing. Sometimes there are several packages already 
queued up, sometimes there are none. Depends on what time you 
click it.

If the button is disabled, it means the package is already in the 
queue and there's no reason to trigger a manual update anyway.

Have you had a different experience?
Feb 29 2016
prev sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 29.02.2016 um 16:09 schrieb Sebastiaan Koppe:

 On Monday, 29 February 2016 at 10:07:56 UTC, Sönke Ludwig wrote:

 We will at some point have to implement the use of push notifications
 for changes, but that requires some work (supporting GitHub-OAuth and
 implementing the push endpoint) and it requires the repositories to be
 re-registered with OAuth.


 Does code.dlang.org have a public repo? I could help out with some impl.
 details.


That would be great! The public repo is at 
https://github.com/D-Programming-Language/dub-registry


 BTW, there is a "Trigger manual update" button on the package
 management page. If you are the owner, you can usually get instant
 updates that way.


 My experience with that button has been mixed.


So the behavior that Mike outlined is generally correct. Although it has 
to be said that there were times where a single sweep through all 
repositories took about half an hour (now about 5 minutes), so the 
button was rarely effective back then. During the last few weeks, that 
task starvation bug also frequently caused the queue to stall for about 
2 hours, so that could be another reason. But apart from that, I usually 
get instant updates in maybe about 95% of the cases.
Feb 29 2016
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 29 February 2016 at 17:07:39 UTC, Sönke Ludwig wrote:

 That would be great! The public repo is at 
 https://github.com/D-Programming-Language/dub-registry


Will have a look.


 So the behavior that Mike outlined is generally correct. 
 Although it has to be said that there were times where a single 
 sweep through all repositories took about half an hour (now 
 about 5 minutes), so the button was rarely effective back then. 
 During the last few weeks, that task starvation bug also 
 frequently caused the queue to stall for about 2 hours, so that 
 could be another reason. But apart from that, I usually get 
 instant updates in maybe about 95% of the cases.


My past experiences were ok. I hit the button, the page 
refreshes, and it tells me how many packages are queued before 
mine. Then its just waiting until the counter hits zero.

Recently I don't see this x-queued-before-you message anymore, 
and no matter how often I press the button, or refresh, nothing 
seems to happen. Until it does. But by that time I have already 
switched to some other task.

Maybe my updates just coincided with that starvation bug. Who 
knows.

Regardless, lets focus on those push notifications.
Feb 29 2016