www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - code.dlang.org reliability update

reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
As of yesterday, code.dlang.org now points to a more powerful dedicated 
server that can host the DUB registry without the danger of freezing due 
to excessive swapping - this is what happened on the 26th last month [1].

In addition to that, the server that previously hosted the registry is 
now used to run an official mirror, reachable at codemirror.dlang.org. 
This will be configured as a built-in fallback server starting with DMD 
2.091.0/DUB 1.12.0 and, at least in theory, will lead to an uptime of 
virtually 100%.

To make use of the mirror today, it is also possible to configure it in 
DUB's settings.json as a custom registry:

{
    "registryUrls": ["https://codemirror.dlang.org/"]
}

settings.json is found/needs to be created in %APPDATA%\dub\ on Windows 
and in ~/.dub/ on all other systems. The custom entry should be removed 
once DUB 1.12.0 is used, to avoid redundant requests in certain situations.


[1]: https://forum.dlang.org/thread/ontwwoxuhnoczcokacwm forum.dlang.org
Mar 02 2020
next sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:

 As of yesterday, code.dlang.org now points to a more powerful 
 dedicated server that can host the DUB registry without the 
 danger of freezing due to excessive swapping - this is what 
 happened on the 26th last month [1].

 In addition to that, the server that previously hosted the 
 registry is now used to run an official mirror, reachable at 
 codemirror.dlang.org. This will be configured as a built-in 
 fallback server starting with DMD 2.091.0/DUB 1.12.0 and, at 
 least in theory, will lead to an uptime of virtually 100%.


Excellent, thank you!

Bastiaan.
Mar 02 2020
prev sibling next sibling parent reply MrSmith <mrsmith33 yandex.ru> writes:
On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:


 settings.json is found/needs to be created in %APPDATA%\dub\ on 
 Windows


I guess that should be %APPDATA%\Local\dub\
Mar 02 2020
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 02.03.2020 um 23:50 schrieb MrSmith:

 On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:
 

 settings.json is found/needs to be created in %APPDATA%\dub\ on Windows


 
 I guess that should be %APPDATA%\Local\dub\


Nope, should be right. At least on my machines, %APPDATA% points to 
AppData\Roaming, which is the right base folder.
Mar 03 2020
parent MrSmith <mrsmith33 yandex.ru> writes:
On Tuesday, 3 March 2020 at 08:26:19 UTC, Sönke Ludwig wrote:

 Nope, should be right. At least on my machines, %APPDATA% 
 points to AppData\Roaming, which is the right base folder.


My bad, confused %APPDATA% with AppData. It is indeed 
AppData\Roaming\dub\
Mar 03 2020
prev sibling next sibling parent Dejan Lekic <dejan.lekic gmail.com> writes:
On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:

 As of yesterday, code.dlang.org now points to a more powerful 
 dedicated server that can host the DUB registry without the 
 danger of freezing due to excessive swapping - this is what 
 happened on the 26th last month [1].


Sönke, thanks a lot for all the effort you are putting into this 
over the years. Very much appreciated!
Mar 03 2020
prev sibling next sibling parent reply WebFreak001 <d.forum webfreak.org> writes:
On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:

 As of yesterday, code.dlang.org now points to a more powerful 
 dedicated server that can host the DUB registry without the 
 danger of freezing due to excessive swapping - this is what 
 happened on the 26th last month [1].

 In addition to that, the server that previously hosted the 
 registry is now used to run an official mirror, reachable at 
 codemirror.dlang.org. This will be configured as a built-in 
 fallback server starting with DMD 2.091.0/DUB 1.12.0 and, at 
 least in theory, will lead to an uptime of virtually 100%.

 To make use of the mirror today, it is also possible to 
 configure it in DUB's settings.json as a custom registry:

 {
    "registryUrls": ["https://codemirror.dlang.org/"]
 }

 settings.json is found/needs to be created in %APPDATA%\dub\ on 
 Windows and in ~/.dub/ on all other systems. The custom entry 
 should be removed once DUB 1.12.0 is used, to avoid redundant 
 requests in certain situations.


 [1]: 
 https://forum.dlang.org/thread/ontwwoxuhnoczcokacwm forum.dlang.org


thank you very much for this Sönke! Is throwing so much more RAM 
(= money) and power (= more money) at it going to be a good 
solution in the long run though?

Is there maybe a plan for remaking the registry architecture like 
adding better supported mirrors and load balancing to it?
Mar 04 2020
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 04.03.2020 um 20:06 schrieb WebFreak001:

 On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:

 As of yesterday, code.dlang.org now points to a more powerful 
 dedicated server that can host the DUB registry without the danger of 
 freezing due to excessive swapping - this is what happened on the 26th 
 last month [1].

 In addition to that, the server that previously hosted the registry is 
 now used to run an official mirror, reachable at codemirror.dlang.org. 
 This will be configured as a built-in fallback server starting with 
 DMD 2.091.0/DUB 1.12.0 and, at least in theory, will lead to an uptime 
 of virtually 100%.

 To make use of the mirror today, it is also possible to configure it 
 in DUB's settings.json as a custom registry:

 {
    "registryUrls": ["https://codemirror.dlang.org/"]
 }

 settings.json is found/needs to be created in %APPDATA%\dub\ on 
 Windows and in ~/.dub/ on all other systems. The custom entry should 
 be removed once DUB 1.12.0 is used, to avoid redundant requests in 
 certain situations.


 [1]: https://forum.dlang.org/thread/ontwwoxuhnoczcokacwm forum.dlang.org


 
 thank you very much for this Sönke! Is throwing so much more RAM (= 
 money) and power (= more money) at it going to be a good solution in the 
 long run though?
 
 Is there maybe a plan for remaking the registry architecture like adding 
 better supported mirrors and load balancing to it?


The server has a different main purpose, so this is just a nice benefit 
for the registry in that regard. The main issue is that there is some 
kind of GC related memory leak that is hard to debug with the basically 
non-existent tooling. Decoupling the package scanning from the web 
frontend into separate processes will play a major role in making that 
much less of an issue, though.

There is at least one fully supported mirror now and the cache based 
mirror by Sebastiaan could also be added as a reliable (albeit partial) 
fallback. So in that regard, at least the pressure for taking more 
involved steps has been mostly relieved now.

But making changes to the general architecture - be it in the form of 
adding load-balancing, or more deeply rooted, such as using GIT for 
storing and distributing the package index - is something that still is 
definitely desirable to push forward.
Mar 05 2020
next sibling parent mork <mork aloha.com> writes:
On Thursday, 5 March 2020 at 12:41:41 UTC, Sönke Ludwig wrote:

 The main issue is that there is some kind of GC related memory 
 leak that is hard to debug with the basically non-existent 
 tooling.


What a shame for D, how is this possible?
Mar 05 2020
prev sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 5 March 2020 at 12:41:41 UTC, Sönke Ludwig wrote:

 But making changes to the general architecture - be it in the 
 form of adding load-balancing, or more deeply rooted, such as 
 using GIT for storing and distributing the package index - is 
 something that still is definitely desirable to push forward.


A week ago I created a git dub package index at 
https://github.com/skoppe/dub-packages-index it gets updated 
every 15 minutes by scraping code.dlang.org

I use it to feed my registry-mirror, which is now fully standalone

A possible next step is creating a PackageSupplier class in dub 
that clones and uses this repo for resolution.

If I was aware of Sönke's work I might have skipped it, but it 
was a good exercise regardless.
Mar 05 2020
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/2/2020 11:17 AM, Sönke Ludwig wrote:

 As of yesterday, code.dlang.org now points to a more powerful dedicated server 
 that can host the DUB registry without the danger of freezing due to excessive 
 swapping - this is what happened on the 26th last month [1].


Wonderful news!
Mar 04 2020