www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Experience: Developing Cloud Foundry applications with D

reply Andre <andre s-e-a-p.de> writes:
I want to share my experiences with D and the platform as a 
service solution Cloud Foundry. Cloud Foundry supports any 
development language as long it is runnable on Linux 64 bit and 
the application has a http Server listening on a port which Cloud 
Foundry provides you using system environment variable PORT.

I am used to develop on Windows. Using VirtuaBox I created a 
Linux virtual machine. I defined a shared directory between my 
windows host system and the Linux virtual machine. In Addition I 
use Putty as SSH client. In the Putty terminal I execute dub and 
after that the Cloud Foundry console application to push the 
binary to the cloud.

I tried different http servers available for D. At the end I 
decided to build my own.

vibe-d: The linking time of a vibe-d application on a Linux 
system is quite high, I think this is due to the dependencies 
like pthreads and so on. Also the binary size is also quite high. 
As far as I remember around 20 mb for a test application. The 
size matters as you have to upload the application again and 
again. Cloud Foundry provides some shared objects libraries. In 
case you have special requirements like pthreads you must create 
a build pack. With a build pack you can download external 
dependencies and install them while pushing the binary to the 
cloud. There already exists build packs for vibe-d but they are 
outdated. They run a long time and then stops.
 From windows I know vibe-d is a great library but on Linux out of 
the box it doesn't make fun due to the usage of external 
dependencies.

I also had a look at the arsdnet http server. But I didn't 
achieved to get my scenario running. In addition to my http 
server component I have a thread which executes 10 some database 
calls and some http requests. If I am not completely wrong the 
scenario of starting a thread in addition to the main server 
thread is not supported unless I duplicate a big amount of coding 
from the template  "main" of the arsdnet http server component.

In the end I build my own very simple http server component. By 
using the pure phobos library compiling and linking is super fast 
and the size of the full application is around 8 mb. Compile, 
link and pushing to the cloud is possible in less than a minute.

In sum, it is just fun to develop cloud applications with D!
Oct 04 2015
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Sunday, 4 October 2015 at 19:13:58 UTC, Andre wrote:

 I tried different http servers available for D.


One more (my own): 
https://github.com/CyberShadow/ae/tree/master/net

ae.net is used on:
- http://forum.dlang.org/
- http://dtest.thecybershadow.net/
- Digger's local web UI
- (other projects not related to D)
Oct 04 2015
parent Andre <andre s-e-a-p.de> writes:
On Sunday, 4 October 2015 at 22:47:46 UTC, Vladimir Panteleev 
wrote:

 On Sunday, 4 October 2015 at 19:13:58 UTC, Andre wrote:

 I tried different http servers available for D.


 One more (my own): 
 https://github.com/CyberShadow/ae/tree/master/net

 ae.net is used on:
 - http://forum.dlang.org/
 - http://dtest.thecybershadow.net/
 - Digger's local web UI
 - (other projects not related to D)


Thanks a lot. Unfortunatelly in the office I have to avoid MPL 
licensed libraries if possible due to legal reasons ):
Oct 05 2015
prev sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 05/10/15 8:13 AM, Andre wrote:

 I want to share my experiences with D and the platform as a service
 solution Cloud Foundry. Cloud Foundry supports any development language
 as long it is runnable on Linux 64 bit and the application has a http
 Server listening on a port which Cloud Foundry provides you using system
 environment variable PORT.

 I am used to develop on Windows. Using VirtuaBox I created a Linux
 virtual machine. I defined a shared directory between my windows host
 system and the Linux virtual machine. In Addition I use Putty as SSH
 client. In the Putty terminal I execute dub and after that the Cloud
 Foundry console application to push the binary to the cloud.

 I tried different http servers available for D. At the end I decided to
 build my own.

 vibe-d: The linking time of a vibe-d application on a Linux system is
 quite high, I think this is due to the dependencies like pthreads and so
 on. Also the binary size is also quite high. As far as I remember around
 20 mb for a test application. The size matters as you have to upload the
 application again and again. Cloud Foundry provides some shared objects
 libraries. In case you have special requirements like pthreads you must
 create a build pack. With a build pack you can download external
 dependencies and install them while pushing the binary to the cloud.
 There already exists build packs for vibe-d but they are outdated. They
 run a long time and then stops.
  From windows I know vibe-d is a great library but on Linux out of the
 box it doesn't make fun due to the usage of external dependencies.

 I also had a look at the arsdnet http server. But I didn't achieved to
 get my scenario running. In addition to my http server component I have
 a thread which executes 10 some database calls and some http requests.
 If I am not completely wrong the scenario of starting a thread in
 addition to the main server thread is not supported unless I duplicate a
 big amount of coding from the template  "main" of the arsdnet http
 server component.

 In the end I build my own very simple http server component. By using
 the pure phobos library compiling and linking is super fast and the size
 of the full application is around 8 mb. Compile, link and pushing to the
 cloud is possible in less than a minute.

 In sum, it is just fun to develop cloud applications with D!


Vibe.d has a provider called libasync. Libasync is fully implemented in 
D. You probably should have tried that at least.
Although I still would recommend trying it ;) It's a lot better then 
what we have in Phobos.
Oct 04 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/5/15 1:34 AM, Rikki Cattermole wrote:

 Vibe.d has a provider called libasync. Libasync is fully implemented in
 D. You probably should have tried that at least.
 Although I still would recommend trying it ;) It's a lot better then
 what we have in Phobos.


Cue choir asking for porting of libasync to phobos. I've first asked 
this a couple of years ago. -- Andrei
Oct 04 2015
next sibling parent Jonas Drewsen <nospam4321 hotmail.com> writes:
On Monday, 5 October 2015 at 06:24:44 UTC, Andrei Alexandrescu 
wrote:

 On 10/5/15 1:34 AM, Rikki Cattermole wrote:

 Vibe.d has a provider called libasync. Libasync is fully 
 implemented in
 D. You probably should have tried that at least.
 Although I still would recommend trying it ;) It's a lot 
 better then
 what we have in Phobos.


 Cue choir asking for porting of libasync to phobos. I've first 
 asked this a couple of years ago. -- Andrei


++++
Oct 05 2015
prev sibling parent reply Etienne Cimon <etcimon gmail.com> writes:
On Monday, 5 October 2015 at 06:24:44 UTC, Andrei Alexandrescu 
wrote:

 On 10/5/15 1:34 AM, Rikki Cattermole wrote:

 Vibe.d has a provider called libasync. Libasync is fully 
 implemented in
 D. You probably should have tried that at least.
 Although I still would recommend trying it ;) It's a lot 
 better then
 what we have in Phobos.


 Cue choir asking for porting of libasync to phobos. I've first 
 asked this a couple of years ago. -- Andrei


To be fair, libasync was release only october last year :)

Will work on adding libasync to Phobos once I'm finished adding a 
few more features to my main project! (3-4 months). Will need to 
strip memutils, will you have std.allocator ready? Anyone is free 
to pick up this project and move libasync to phobos if ready 
before me. I've gotten this all-D web framework (vibe.d, botan-D 
TLS, D libhttp/2, etc) into production on my end and it works 
great!
Oct 06 2015
next sibling parent John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 6 October 2015 at 11:30:00 UTC, Etienne Cimon wrote:

 Will need to strip memutils, will you have std.allocator ready?


https://github.com/D-Programming-Language/phobos/tree/master/std/experimental/allocator

:)
Oct 06 2015
prev sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 07/10/15 12:29 AM, Etienne Cimon wrote:

 On Monday, 5 October 2015 at 06:24:44 UTC, Andrei Alexandrescu wrote:

 On 10/5/15 1:34 AM, Rikki Cattermole wrote:

 Vibe.d has a provider called libasync. Libasync is fully implemented in
 D. You probably should have tried that at least.
 Although I still would recommend trying it ;) It's a lot better then
 what we have in Phobos.


 Cue choir asking for porting of libasync to phobos. I've first asked
 this a couple of years ago. -- Andrei


 To be fair, libasync was release only october last year :)

 Will work on adding libasync to Phobos once I'm finished adding a few
 more features to my main project! (3-4 months). Will need to strip
 memutils, will you have std.allocator ready? Anyone is free to pick up
 this project and move libasync to phobos if ready before me. I've gotten
 this all-D web framework (vibe.d, botan-D TLS, D libhttp/2, etc) into
 production on my end and it works great!


I'm hoping as part of this we will be able to merge the platform 
abstraction out to something like: 
https://github.com/rikkimax/alphaPhobos/blob/master/source/std/experimental/platform.d#L10
That way windowing as well can take advantage of it :)
Oct 06 2015
prev sibling parent reply Andre <andre s-e-a-p.de> writes:
On Sunday, 4 October 2015 at 23:34:58 UTC, Rikki Cattermole wrote:

 On 05/10/15 8:13 AM, Andre wrote:



 Vibe.d has a provider called libasync. Libasync is fully 
 implemented in D. You probably should have tried that at least.
 Although I still would recommend trying it ;) It's a lot better 
 then what we have in Phobos.


Looks good, it seems all external dependencies issues are gone 
for vibe-d on Cloud Foundry. I haven't a working test application 
so far but I think the issue is now releated to the correct bind 
address in a Cloud Foundry environment.
Oct 05 2015
parent reply Andre <andre s-e-a-p.de> writes:
On Monday, 5 October 2015 at 15:51:09 UTC, Andre wrote:

 On Sunday, 4 October 2015 at 23:34:58 UTC, Rikki Cattermole 
 wrote:

 On 05/10/15 8:13 AM, Andre wrote:



 Vibe.d has a provider called libasync. Libasync is fully 
 implemented in D. You probably should have tried that at least.
 Although I still would recommend trying it ;) It's a lot 
 better then what we have in Phobos.


 Looks good, it seems all external dependencies issues are gone 
 for vibe-d on Cloud Foundry. I haven't a working test 
 application so far but I think the issue is now releated to the 
 correct bind address in a Cloud Foundry environment.


As far as I understand vibe-d is not runnable out of the box in 
Cloud Foundry.
I cannot point directly to the issue, I just get the information, 
that the
test application does not accept connections during a health 
check run in the log file.

My simple test application is the example from the homepage 
except,
that the port is read from the environment variable PORT.

import vibe.d;

shared static this()
{
	import std.process: environment;
	auto settings = new HTTPServerSettings;
	settings.port = to!ushort(environment.get("PORT", "8080"));

	listenHTTP(settings, &handleRequest);
}

void handleRequest(HTTPServerRequest req,
                    HTTPServerResponse res)
{
	if (req.path == "/")
		res.writeBody("Hello, World!", "text/plain");
}

I assume some issues with memutils, but have no chance to 
validate this.
What I can see, if I run the application in my virtubox (Ubuntu) 
instead
of the Cloud Foundry, I get following reponse:

vagrant vagrant-ubuntu-trusty-64:~/projects/tests/vibed_test$ dub
Target memutils 0.4.1 is up to date. Use --force to rebuild.
Target libasync 0.7.5 is up to date. Use --force to rebuild.
Target vibe-d 0.7.25 is up to date. Use --force to rebuild.
Building vibed_test ~master configuration "debug", build type 
debug.
Compiling using dmd...
Enhanced memory security is enabled.
Using Linux EPOLL for events
Linking...
Running ./bin/app
Listening for HTTP requests on :::8080
Listening for HTTP requests on 0.0.0.0:8080
E: Could not mlock 65536 bytes
Oct 05 2015
parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Tuesday, 6 October 2015 at 05:45:18 UTC, Andre wrote:

 vagrant vagrant-ubuntu-trusty-64:~/projects/tests/vibed_test$ 
 dub
 Target memutils 0.4.1 is up to date. Use --force to rebuild.
 Target libasync 0.7.5 is up to date. Use --force to rebuild.
 Target vibe-d 0.7.25 is up to date. Use --force to rebuild.
 Building vibed_test ~master configuration "debug", build type 
 debug.
 Compiling using dmd...
 Enhanced memory security is enabled.
 Using Linux EPOLL for events
 Linking...
 Running ./bin/app
 Listening for HTTP requests on :::8080
 Listening for HTTP requests on 0.0.0.0:8080
 E: Could not mlock 65536 bytes


Does it keep running? AFAIK, the last line is just a warning from 
the botan library that attempts to allocate non-swappable memory 
for holding secret keys etc.
Oct 06 2015
next sibling parent Etienne Cimon <etcimon gmail.com> writes:
On Tuesday, 6 October 2015 at 09:36:42 UTC, Marc Schütz wrote:

 On Tuesday, 6 October 2015 at 05:45:18 UTC, Andre wrote:

 vagrant vagrant-ubuntu-trusty-64:~/projects/tests/vibed_test$ 
 dub
 Target memutils 0.4.1 is up to date. Use --force to rebuild.
 Target libasync 0.7.5 is up to date. Use --force to rebuild.
 Target vibe-d 0.7.25 is up to date. Use --force to rebuild.
 Building vibed_test ~master configuration "debug", build type 
 debug.
 Compiling using dmd...
 Enhanced memory security is enabled.
 Using Linux EPOLL for events
 Linking...
 Running ./bin/app
 Listening for HTTP requests on :::8080
 Listening for HTTP requests on 0.0.0.0:8080
 E: Could not mlock 65536 bytes


 Does it keep running? AFAIK, the last line is just a warning 
 from the botan library that attempts to allocate non-swappable 
 memory for holding secret keys etc.


The error is with mlock, the ulimit for locked memory is too low 
on non-root user accounts, so it falls back to simple zeroize of 
swappable memory
Oct 06 2015
prev sibling parent Andre <andre s-e-a-p.de> writes:
On Tuesday, 6 October 2015 at 09:36:42 UTC, Marc Schütz wrote:

 On Tuesday, 6 October 2015 at 05:45:18 UTC, Andre wrote:

 vagrant vagrant-ubuntu-trusty-64:~/projects/tests/vibed_test$ 
 dub
 Target memutils 0.4.1 is up to date. Use --force to rebuild.
 Target libasync 0.7.5 is up to date. Use --force to rebuild.
 Target vibe-d 0.7.25 is up to date. Use --force to rebuild.
 Building vibed_test ~master configuration "debug", build type 
 debug.
 Compiling using dmd...
 Enhanced memory security is enabled.
 Using Linux EPOLL for events
 Linking...
 Running ./bin/app
 Listening for HTTP requests on :::8080
 Listening for HTTP requests on 0.0.0.0:8080
 E: Could not mlock 65536 bytes


 Does it keep running? AFAIK, the last line is just a warning 
 from the botan library that attempts to allocate non-swappable 
 memory for holding secret keys etc.


I got it working. *Argh* I forgot the command parameter in the 
cloud foundry manifest file. vibe-d is working fine, out of the 
box in cloud foundry with libasync :)
Oct 06 2015