www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A suggestion for dub

reply mahdi <xemailpro yahoo.co.uk> writes:
I just want to get some feedback to see if this is a good idea or 
no.

I propose adding a new command to dub (`dub dist` or `dub 
package`) which will produce a zipped version of the current 
package. This zip file can be uploaded to a central repository 
(code.dlang.com maybe), and others can use this package in their 
`dub.json` file by adding relevant dependency (Behind the scene, 
dub will fetch and decompress the package).
Feb 28 2016
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 29/02/16 5:46 PM, mahdi wrote:
 I just want to get some feedback to see if this is a good idea or no.

 I propose adding a new command to dub (`dub dist` or `dub package`)
 which will produce a zipped version of the current package. This zip
 file can be uploaded to a central repository (code.dlang.com maybe), and
 others can use this package in their `dub.json` file by adding relevant
 dependency (Behind the scene, dub will fetch and decompress the package).
Most of the work is already done. code.dlang.org already grabs a zip of the contents of the repo from e.g. Github. Dub then just downloads + extracts it. However I don't think its cached on code.dlang.org side. Basically the hard part is writing a new provider and adding the ability to upload via api. Of course the implementation may have changed in the last year or so.
Feb 28 2016
prev sibling next sibling parent reply Brad Anderson <eco gnuk.net> writes:
On Monday, 29 February 2016 at 04:46:13 UTC, mahdi wrote:
 I just want to get some feedback to see if this is a good idea 
 or no.

 I propose adding a new command to dub (`dub dist` or `dub 
 package`) which will produce a zipped version of the current 
 package. This zip file can be uploaded to a central repository 
 (code.dlang.com maybe), and others can use this package in 
 their `dub.json` file by adding relevant dependency (Behind the 
 scene, dub will fetch and decompress the package).
A neat feature would be to deploy it using the github releases feature. There is an API for it: https://developer.github.com/v3/repos/releases/ Being able to upload program binaries for end users (non-programmers) to use would be killer.
Feb 28 2016
parent reply mahdi <xemailpro yahoo.co.uk> writes:
On Monday, 29 February 2016 at 05:16:50 UTC, Brad Anderson wrote:
 On Monday, 29 February 2016 at 04:46:13 UTC, mahdi wrote:
 I just want to get some feedback to see if this is a good idea 
 or no.

 I propose adding a new command to dub (`dub dist` or `dub 
 package`) which will produce a zipped version of the current 
 package. This zip file can be uploaded to a central repository 
 (code.dlang.com maybe), and others can use this package in 
 their `dub.json` file by adding relevant dependency (Behind 
 the scene, dub will fetch and decompress the package).
A neat feature would be to deploy it using the github releases feature. There is an API for it: https://developer.github.com/v3/repos/releases/ Being able to upload program binaries for end users (non-programmers) to use would be killer.
But definitely those binaries won't be cross-platform! That's why I think uploading/downloading source code is more robust.
Feb 28 2016
parent Brad Anderson <eco gnuk.net> writes:
On Monday, 29 February 2016 at 05:59:56 UTC, mahdi wrote:
 On Monday, 29 February 2016 at 05:16:50 UTC, Brad Anderson 
 wrote:
 On Monday, 29 February 2016 at 04:46:13 UTC, mahdi wrote:
 [...]
A neat feature would be to deploy it using the github releases feature. There is an API for it: https://developer.github.com/v3/repos/releases/ Being able to upload program binaries for end users (non-programmers) to use would be killer.
But definitely those binaries won't be cross-platform! That's why I think uploading/downloading source code is more robust.
You can upload multiple platform binaries. I'm pretty sure LDC or GDC can do some cross compiling too. It's not reasonable to expect all end users to compile a program.
Feb 28 2016
prev sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Mon, 29 Feb 2016 04:46:13 +0000, mahdi wrote:

 I just want to get some feedback to see if this is a good idea or no.
 
 I propose adding a new command to dub (`dub dist` or `dub package`)
 which will produce a zipped version of the current package. This zip
 file can be uploaded to a central repository (code.dlang.com maybe), and
 others can use this package in their `dub.json` file by adding relevant
 dependency (Behind the scene, dub will fetch and decompress the
 package).
In other words, you want to distribute your source code, but you don't want to host a publicly accessibly git repository and don't want to use any of the free git hosting services around. I'm not sure how this would be an advantage. You don't have to learn git, I guess? It's less VCS lockin? Though it probably wouldn't be so hard to add mercurial and svn plugins for dub. Something that *would* be useful is a command that produces binaries and, in the case of libraries, D interface files so people can depend on the library even if the full source code isn't available. The obvious problem: D is typically compiled to native code. That means you have to compile once for each platform. Also, you need a different binary for every set of version() flags the thing supports. That makes binary distribution tricky in the general case.
Feb 28 2016
parent reply mahdi <xemailpro yahoo.co.uk> writes:
On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright wrote:
 On Mon, 29 Feb 2016 04:46:13 +0000, mahdi wrote:

 [...]
In other words, you want to distribute your source code, but you don't want to host a publicly accessibly git repository and don't want to use any of the free git hosting services around. I'm not sure how this would be an advantage. You don't have to learn git, I guess? It's less VCS lockin? Though it probably wouldn't be so hard to add mercurial and svn plugins for dub. Something that *would* be useful is a command that produces binaries and, in the case of libraries, D interface files so people can depend on the library even if the full source code isn't available. The obvious problem: D is typically compiled to native code. That means you have to compile once for each platform. Also, you need a different binary for every set of version() flags the thing supports. That makes binary distribution tricky in the general case.
The aim is to make package distribution easier and more straightforward. If someone has done some development on his local machine and wants to distribute it, it's better not to force him to create a repo on Github or ... .
Feb 29 2016
parent reply sigod <sigod.mail gmail.com> writes:
On Monday, 29 February 2016 at 08:02:51 UTC, mahdi wrote:
 On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright wrote:
 [...]
The aim is to make package distribution easier and more straightforward. If someone has done some development on his local machine and wants to distribute it, it's better not to force him to create a repo on Github or ... .
And you end up creating your own repository hosting. Or you would allow users to not use (D)VCS. Which will end up as a nightmare.
Feb 29 2016
next sibling parent mahdi <xemailpro yahoo.co.uk> writes:
On Monday, 29 February 2016 at 09:27:50 UTC, sigod wrote:
 On Monday, 29 February 2016 at 08:02:51 UTC, mahdi wrote:
 On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright 
 wrote:
 [...]
The aim is to make package distribution easier and more straightforward. If someone has done some development on his local machine and wants to distribute it, it's better not to force him to create a repo on Github or ... .
And you end up creating your own repository hosting. Or you would allow users to not use (D)VCS. Which will end up as a nightmare.
Well, not necessarily. The benefit of this method is that it separates source code hosting (local, github, ...) from distribution. This is the strategy that is followed in Python (PyPI) and Perl (CPAN) community.
Feb 29 2016
prev sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 29 February 2016 at 09:27:50 UTC, sigod wrote:
 On Monday, 29 February 2016 at 08:02:51 UTC, mahdi wrote:
 On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright 
 wrote:
 [...]
The aim is to make package distribution easier and more straightforward. If someone has done some development on his local machine and wants to distribute it, it's better not to force him to create a repo on Github or ... .
And you end up creating your own repository hosting. Or you would allow users to not use (D)VCS. Which will end up as a nightmare.
Yeah that would be a nightmare. Plus the storage requirements. Better have github/bitbucket do it for you. The thing I do run into occasionally is when I am modifying two dub package, where one depends on the other, and I am pushing a new version/tag to github. It generally takes a while before code.dlang.org picks up on new versions being available, and until that time I cannot compile the other package which depends on the new version. Locally I can do `dub add-local`, but I don't want to do that on the CI, nor on other developers machine whenever they get my updated dub.json. Granted, it doesn't take that long, but seeing how fast other services pick up on github/bitbucket changes (wercker starts building a mere 0.5 secs after a push), I wonder why code.dlang.org is being so slow about it.
Feb 29 2016
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 29.02.2016 um 10:46 schrieb Sebastiaan Koppe:
 On Monday, 29 February 2016 at 09:27:50 UTC, sigod wrote:
 On Monday, 29 February 2016 at 08:02:51 UTC, mahdi wrote:
 On Monday, 29 February 2016 at 06:10:17 UTC, Chris Wright wrote:
 [...]
The aim is to make package distribution easier and more straightforward. If someone has done some development on his local machine and wants to distribute it, it's better not to force him to create a repo on Github or ... .
And you end up creating your own repository hosting. Or you would allow users to not use (D)VCS. Which will end up as a nightmare.
Yeah that would be a nightmare. Plus the storage requirements. Better have github/bitbucket do it for you.
Storage requirements would definitely be a problem. Hosting a database with metadata is one thing, but uploading potentially hundreds of megabytes or even gigabytes per package is another. The metadata alone takes up about 1.5 GiB right now. Should we ever grow as much as other languages have in terms of package count, hosting package contents is definitely out of the question without some sort of corporate investment.
 The thing I do run into occasionally is when I am modifying two dub
 package, where one depends on the other, and I am pushing a new
 version/tag to github. It generally takes a while before code.dlang.org
 picks up on new versions being available, and until that time I cannot
 compile the other package which depends on the new version.

 Locally I can do `dub add-local`, but I don't want to do that on the CI,
 nor on other developers machine whenever they get my updated dub.json.

 Granted, it doesn't take that long, but seeing how fast other services
 pick up on github/bitbucket changes (wercker starts building a mere 0.5
 secs after a push), I wonder why code.dlang.org is being so slow about it.
Currently all packages are polled for changes every 30 minutes. We will at some point have to implement the use of push notifications for changes, but that requires some work (supporting GitHub-OAuth and implementing the push endpoint) and it requires the repositories to be re-registered with OAuth. BTW, there is a "Trigger manual update" button on the package management page. If you are the owner, you can usually get instant updates that way.
Feb 29 2016
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 29 February 2016 at 10:07:56 UTC, Sönke Ludwig wrote:
 We will at some point have to implement the use of push 
 notifications for changes, but that requires some work 
 (supporting GitHub-OAuth and implementing the push endpoint) 
 and it requires the repositories to be re-registered with OAuth.
Does code.dlang.org have a public repo? I could help out with some impl. details.
 BTW, there is a "Trigger manual update" button on the package 
 management page. If you are the owner, you can usually get 
 instant updates that way.
My experience with that button has been mixed.
Feb 29 2016
next sibling parent Mike Parker <aldacron gmail.com> writes:
On Monday, 29 February 2016 at 15:09:00 UTC, Sebastiaan Koppe 
wrote:

 BTW, there is a "Trigger manual update" button on the package 
 management page. If you are the owner, you can usually get 
 instant updates that way.
My experience with that button has been mixed.
How so? If the package is not already queued for an update, pressing the button queues it up. It's not a 'update right now' sort of thing. Sometimes there are several packages already queued up, sometimes there are none. Depends on what time you click it. If the button is disabled, it means the package is already in the queue and there's no reason to trigger a manual update anyway. Have you had a different experience?
Feb 29 2016
prev sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 29.02.2016 um 16:09 schrieb Sebastiaan Koppe:
 On Monday, 29 February 2016 at 10:07:56 UTC, Sönke Ludwig wrote:
 We will at some point have to implement the use of push notifications
 for changes, but that requires some work (supporting GitHub-OAuth and
 implementing the push endpoint) and it requires the repositories to be
 re-registered with OAuth.
Does code.dlang.org have a public repo? I could help out with some impl. details.
That would be great! The public repo is at https://github.com/D-Programming-Language/dub-registry
 BTW, there is a "Trigger manual update" button on the package
 management page. If you are the owner, you can usually get instant
 updates that way.
My experience with that button has been mixed.
So the behavior that Mike outlined is generally correct. Although it has to be said that there were times where a single sweep through all repositories took about half an hour (now about 5 minutes), so the button was rarely effective back then. During the last few weeks, that task starvation bug also frequently caused the queue to stall for about 2 hours, so that could be another reason. But apart from that, I usually get instant updates in maybe about 95% of the cases.
Feb 29 2016
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 29 February 2016 at 17:07:39 UTC, Sönke Ludwig wrote:
 That would be great! The public repo is at 
 https://github.com/D-Programming-Language/dub-registry
Will have a look.
 So the behavior that Mike outlined is generally correct. 
 Although it has to be said that there were times where a single 
 sweep through all repositories took about half an hour (now 
 about 5 minutes), so the button was rarely effective back then. 
 During the last few weeks, that task starvation bug also 
 frequently caused the queue to stall for about 2 hours, so that 
 could be another reason. But apart from that, I usually get 
 instant updates in maybe about 95% of the cases.
My past experiences were ok. I hit the button, the page refreshes, and it tells me how many packages are queued before mine. Then its just waiting until the counter hits zero. Recently I don't see this x-queued-before-you message anymore, and no matter how often I press the button, or refresh, nothing seems to happen. Until it does. But by that time I have already switched to some other task. Maybe my updates just coincided with that starvation bug. Who knows. Regardless, lets focus on those push notifications.
Feb 29 2016