www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - AWS SDK

reply Jadbox <jadit2 gmail.com> writes:
Is there an AWS library in the works for D? It's seriously the 
main blocker for me to push adoption of the language internally.

If not, I can try to invest time into making one, but I could use 
help.

(fyi, there's one in the works for Rust: 
https://github.com/rusoto/rusoto)
Jun 26 2016
next sibling parent bachmeier <no spam.net> writes:
On Sunday, 26 June 2016 at 23:06:02 UTC, Jadbox wrote:

 Is there an AWS library in the works for D? It's seriously the 
 main blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could 
 use help.

 (fyi, there's one in the works for Rust: 
 https://github.com/rusoto/rusoto)


http://forum.dlang.org/post/jqotefrswxhaiqteefdq forum.dlang.org
Jun 26 2016
prev sibling next sibling parent yawniek <dlang srtnwz.com> writes:
On Sunday, 26 June 2016 at 23:06:02 UTC, Jadbox wrote:

 Is there an AWS library in the works for D? It's seriously the 
 main blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could 
 use help.

 (fyi, there's one in the works for Rust: 
 https://github.com/rusoto/rusoto)


what services do you need?
the important bits, such as auth are around, but you likely need 
to implement some stuff for yourself.

we have an incomplete s3 libary with
https://github.com/tamediadigital/vibe-s3
there is also some code for kinesis and dynamo around but its 
likely outdated.
Jun 27 2016
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 26 June 2016 at 23:06:02 UTC, Jadbox wrote:

 Is there an AWS library in the works for D? It's seriously the 
 main blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could 
 use help.

 (fyi, there's one in the works for Rust: 
 https://github.com/rusoto/rusoto)


The C++ SDK appears to use a Java generator, maybe you can make 
it work for D since it is quite close to C++ (but probably a lot 
of work):

https://github.com/aws/aws-sdk-cpp/tree/master/code-generation

https://aws.amazon.com/blogs/aws/introducing-the-aws-sdk-for-c/
Jun 27 2016
prev sibling next sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:

 Is there an AWS library in the works for D? It's seriously the main
 blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could use help.

 (fyi, there's one in the works for Rust: https://github.com/rusoto/rusoto)


I have some old code here:

     https://github.com/braddr/downloads.dlang.org/tree/master/src

It has sig v2 signing and some s3 object related code.  Super minimal 
and sig v2 is out of date (though still very usable, in most aws regions).

Ideally, someone would partner with aws -- there's a developer tools 
forum -- to add D to the suite of languages.  I'm fairly sure they code 
generate at least a large part of the tool kits from the api definitions 
for each of the tons of services.  Trying to manage them all by hand is 
a loosing battle.

I've been tempted to do this a couple times, but the time investment 
would likely be more than I'm willing to spend.
Jun 27 2016
parent jadbox <jadit2 gmail.com> writes:
On Monday, 27 June 2016 at 17:53:32 UTC, Brad Roberts wrote:

 https://github.com/braddr/downloads.dlang.org/tree/master/src

 It has sig v2 signing and some s3 object related code.  Super 
 minimal and sig v2 is out of date (though still very usable, in 
 most aws regions).

 Ideally, someone would partner with aws -- there's a developer 
 tools forum -- to add D to the suite of languages.  I'm fairly 
 sure they code generate at least a large part of the tool kits 
 from the api definitions for each of the tons of services.  
 Trying to manage them all by hand is a loosing battle.

 I've been tempted to do this a couple times, but the time 
 investment would likely be more than I'm willing to spend.


Yep, the sdk would need to be autogenerated from their spec 
files. I use a great number of services from s3, Kinesis, SQS, 
SNS, Redshift, DynamoDB, EC2 cmds, Route 53 reconfiguring. This 
is a common story for most cloud developers. With the advent of 
microservices, it's a relatively easy sell to get developers to 
try a new language for a small service, but most of these 
services rely on cloud drivers.
Jun 27 2016
prev sibling parent reply Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:

 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:

 Is there an AWS library in the works for D? It's seriously the main
 blocker for me to push adoption of the language internally.

 If not, I can try to invest time into making one, but I could use help.

 (fyi, there's one in the works for Rust: https://github.com/rusoto/rusoto)


 I have some old code here:

     https://github.com/braddr/downloads.dlang.org/tree/master/src

 It has sig v2 signing and some s3 object related code.  Super minimal and sig
v2 is out of date
 (though still very usable, in most aws regions).

 Ideally, someone would partner with aws -- there's a developer tools forum --
to add D to the suite
 of languages.  I'm fairly sure they code generate at least a large part of the
tool kits from the
 api definitions for each of the tons of services.  Trying to manage them all
by hand is a loosing
 battle.

 I've been tempted to do this a couple times, but the time investment would
likely be more than I'm
 willing to spend.


I've started talking with a few of the aws sdk engineers (some of who I worked
with when I was 
there) today about their sdk and the tooling involved.  It might not be a
horrible job to add 
another language to the mix.  I'm going to meet with them and dig through some
of the tooling to get 
a better feel for it.

Later,
Brad
Jun 27 2016
parent reply Ky-Anh Huynh <saigon example.net> writes:
On Tuesday, 28 June 2016 at 00:36:31 UTC, Brad Roberts wrote:

 On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:

 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:

 [...]


 I have some old code here:

     
 https://github.com/braddr/downloads.dlang.org/tree/master/src

 It has sig v2 signing and some s3 object related code.  Super 
 minimal and sig v2 is out of date
 (though still very usable, in most aws regions).

 Ideally, someone would partner with aws -- there's a developer 
 tools forum -- to add D to the suite
 of languages.  I'm fairly sure they code generate at least a 
 large part of the tool kits from the
 api definitions for each of the tons of services.  Trying to 
 manage them all by hand is a loosing
 battle.

 I've been tempted to do this a couple times, but the time 
 investment would likely be more than I'm
 willing to spend.


 I've started talking with a few of the aws sdk engineers (some 
 of who I worked with when I was there) today about their sdk 
 and the tooling involved.  It might not be a horrible job to 
 add another language to the mix.  I'm going to meet with them 
 and dig through some of the tooling to get a better feel for it.

 Later,
 Brad


Hi Brad,

Do we have any good update on this?

I'm writting my devops tools in Dlang now (Golang is too hot to 
use, isn't it:D). It's a great win if we can have AWS SDK support.

I'm not a master however if there is any thing I can help, please 
let me know.

Thanks for your reading.
Oct 17 2017
next sibling parent Brad Roberts <braddr puremagic.com> writes:
On 10/17/2017 6:32 PM, Ky-Anh Huynh via Digitalmars-d wrote:

 On Tuesday, 28 June 2016 at 00:36:31 UTC, Brad Roberts wrote:

 On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:

 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:

 [...]


 I have some old code here:

 https://github.com/braddr/downloads.dlang.org/tree/master/src

 It has sig v2 signing and some s3 object related code.  Super 
 minimal and sig v2 is out of date
 (though still very usable, in most aws regions).

 Ideally, someone would partner with aws -- there's a developer tools 
 forum -- to add D to the suite
 of languages.  I'm fairly sure they code generate at least a large 
 part of the tool kits from the
 api definitions for each of the tons of services.  Trying to manage 
 them all by hand is a loosing
 battle.

 I've been tempted to do this a couple times, but the time investment 
 would likely be more than I'm
 willing to spend.


 I've started talking with a few of the aws sdk engineers (some of who 
 I worked with when I was there) today about their sdk and the tooling 
 involved.  It might not be a horrible job to add another language to 
 the mix.  I'm going to meet with them and dig through some of the 
 tooling to get a better feel for it.

 Later,
 Brad


 Hi Brad,

 Do we have any good update on this?

 I'm writting my devops tools in Dlang now (Golang is too hot to use, 
 isn't it:D). It's a great win if we can have AWS SDK support.

 I'm not a master however if there is any thing I can help, please let 
 me know.

 Thanks for your reading.


Unfortunately, I haven't made the time.  Too much contract work being 
thrown my way for the last year or so.
Oct 17 2017
prev sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 18 October 2017 at 01:32:51 UTC, Ky-Anh Huynh wrote:

 On Tuesday, 28 June 2016 at 00:36:31 UTC, Brad Roberts wrote:

 On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:

 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:

 [...]


 I have some old code here:

     
 https://github.com/braddr/downloads.dlang.org/tree/master/src

 It has sig v2 signing and some s3 object related code.  Super 
 minimal and sig v2 is out of date
 (though still very usable, in most aws regions).

 Ideally, someone would partner with aws -- there's a 
 developer tools forum -- to add D to the suite
 of languages.  I'm fairly sure they code generate at least a 
 large part of the tool kits from the
 api definitions for each of the tons of services.  Trying to 
 manage them all by hand is a loosing
 battle.

 I've been tempted to do this a couple times, but the time 
 investment would likely be more than I'm
 willing to spend.


 I've started talking with a few of the aws sdk engineers (some 
 of who I worked with when I was there) today about their sdk 
 and the tooling involved.  It might not be a horrible job to 
 add another language to the mix.  I'm going to meet with them 
 and dig through some of the tooling to get a better feel for 
 it.

 Later,
 Brad


 Hi Brad,

 Do we have any good update on this?

 I'm writting my devops tools in Dlang now (Golang is too hot to 
 use, isn't it:D). It's a great win if we can have AWS SDK 
 support.

 I'm not a master however if there is any thing I can help, 
 please let me know.

 Thanks for your reading.


Hi,

for an inhouse project I developed a generator which generates 
based on the API definitions from here 
https://github.com/aws/aws-sdk-js/tree/master/apis
classes for the services and structures for the input/output 
requests.


     auto client = new AwsClient();
     auto dynamoDBService = new DynamoDBService(client);

     CreateTableInput request = {
             tableName: "zapp-test1",
             provisionedThroughput: {
                 readCapacityUnits: 1,
                 writeCapacityUnits: 1
             },
             attributeDefinitions: [
                 {attributeName: "A", attributeType: 
ScalarAttributeType.S},
                 {attributeName: "B", attributeType: 
ScalarAttributeType.S}
             ],
             keySchema: [
                 {attributeName: "A", keyType: KeyType.HASH}
             ]
         };

     dynamoDBService.createTable(request);


Actually the generated classes do not call AWS directly but uses 
the AWS console client.
Unfortunately the generator is proprietary.
The good thing is, it was easily written within 4-5 hours.

PS. Almost every service was working out of the box except EMR. 
There is a difference between the EMR commands / parameters in 
the API definitions and in the AWS console client.

Kind regards
André
Oct 18 2017
parent reply aberba <karabutaworld gmail.com> writes:
On Wednesday, 18 October 2017 at 11:11:35 UTC, Andre Pany wrote:

 On Wednesday, 18 October 2017 at 01:32:51 UTC, Ky-Anh Huynh 
 wrote:

 On Tuesday, 28 June 2016 at 00:36:31 UTC, Brad Roberts wrote:

 On 6/27/16 10:53 AM, Brad Roberts via Digitalmars-d wrote:

 On 6/26/2016 4:06 PM, Jadbox via Digitalmars-d wrote:

 [...]


 I have some old code here:

     
 https://github.com/braddr/downloads.dlang.org/tree/master/src

 It has sig v2 signing and some s3 object related code.  
 Super minimal and sig v2 is out of date
 (though still very usable, in most aws regions).

 Ideally, someone would partner with aws -- there's a 
 developer tools forum -- to add D to the suite
 of languages.  I'm fairly sure they code generate at least a 
 large part of the tool kits from the
 api definitions for each of the tons of services.  Trying to 
 manage them all by hand is a loosing
 battle.

 I've been tempted to do this a couple times, but the time 
 investment would likely be more than I'm
 willing to spend.


 I've started talking with a few of the aws sdk engineers 
 (some of who I worked with when I was there) today about 
 their sdk and the tooling involved.  It might not be a 
 horrible job to add another language to the mix.  I'm going 
 to meet with them and dig through some of the tooling to get 
 a better feel for it.

 Later,
 Brad


 Hi Brad,

 Do we have any good update on this?

 I'm writting my devops tools in Dlang now (Golang is too hot 
 to use, isn't it:D). It's a great win if we can have AWS SDK 
 support.

 I'm not a master however if there is any thing I can help, 
 please let me know.

 Thanks for your reading.


 Hi,

 for an inhouse project I developed a generator which generates 
 based on the API definitions from here 
 https://github.com/aws/aws-sdk-js/tree/master/apis
 classes for the services and structures for the input/output 
 requests.


     auto client = new AwsClient();
     auto dynamoDBService = new DynamoDBService(client);

     CreateTableInput request = {
             tableName: "zapp-test1",
             provisionedThroughput: {
                 readCapacityUnits: 1,
                 writeCapacityUnits: 1
             },
             attributeDefinitions: [
                 {attributeName: "A", attributeType: 
 ScalarAttributeType.S},
                 {attributeName: "B", attributeType: 
 ScalarAttributeType.S}
             ],
             keySchema: [
                 {attributeName: "A", keyType: KeyType.HASH}
             ]
         };

     dynamoDBService.createTable(request);


 Actually the generated classes do not call AWS directly but 
 uses the AWS console client.
 Unfortunately the generator is proprietary.
 The good thing is, it was easily written within 4-5 hours.

 PS. Almost every service was working out of the box except EMR. 
 There is a difference between the EMR commands / parameters in 
 the API definitions and in the AWS console client.

 Kind regards
 André


What's up with proprietary stuff in this? At least give us the 
setup for us to generate the apis on our own if you have your 
private stuff in there. I have been waiting for a D AWS SDK for 
"years"... At least any cloud SDK. The generated api approach 
seem like the way to go cus its a lot of work to build and 
maintain a D API by hand.

I wanted to just do an S3 api plus some few services to use in 
some of our microservices. We opted to use Nodejs because it 
already has all the Cloud SDK for almost every provider. 
Technically D is much better...

You know, cloud and microservices is the way to build any 
"scalable" server-side services in 2017 yet it seem those using D 
for server-side dev are still writing monoliths.
Oct 18 2017
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 18 October 2017 at 19:28:30 UTC, aberba wrote:

 What's up with proprietary stuff in this? At least give us the 
 setup for us to generate the apis on our own if you have your 
 private stuff in there. I have been waiting for a D AWS SDK for 
 "years"... At least any cloud SDK. The generated api approach 
 seem like the way to go cus its a lot of work to build and 
 maintain a D API by hand.

 I wanted to just do an S3 api plus some few services to use in 
 some of our microservices. We opted to use Nodejs because it 
 already has all the Cloud SDK for almost every provider. 
 Technically D is much better...

 You know, cloud and microservices is the way to build any 
 "scalable" server-side services in 2017 yet it seem those using 
 D for server-side dev are still writing monoliths.


Although I wrote the generator, it is property of my employer. I 
am unfortunately not allowed to make it public.

The implementation is straight forward. For every api definition 
(e.g. 
https://github.com/aws/aws-sdk-js/blob/master/apis/iam-2010-05-08.normal.json)
create a module with a class. Read the json file with std.json and for every
operation in the json create a method with the same name in the class.
For every shape in the json create a struct with the same name.

Every operation defines either an input shape, output shape, or 
both. During the method call serialize the input struct to json, 
call the aws client with the json and deserialize the output json 
to the output struct.

Kind regards
André
Oct 18 2017
parent reply ikod <geller.garry gmail.com> writes:
Hello,

On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany wrote:

 On Wednesday, 18 October 2017 at 19:28:30 UTC, aberba wrote:
 The implementation is straight forward. For every api 
 definition (e.g. 
 https://github.com/aws/aws-sdk-js/blob/master/apis/iam-2010-05-08.normal.json)
create a module with a class. Read the json file with std.json and for every
operation in the json create a method with the same name in the class.
 For every shape in the json create a struct with the same name.

 Every operation defines either an input shape, output shape, or 
 both. During the method call serialize the input struct to 
 json, call the aws client with the json and deserialize the 
 output json to the output struct.

 Kind regards
 André


Is there any docs from Amazon on this, beside json's itself?  I 
will be very grateful for link.
Oct 18 2017
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 18 October 2017 at 20:19:20 UTC, ikod wrote:

 Hello,

 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany wrote:

 On Wednesday, 18 October 2017 at 19:28:30 UTC, aberba wrote:
 The implementation is straight forward. For every api 
 definition (e.g. 
 https://github.com/aws/aws-sdk-js/blob/master/apis/iam-2010-05-08.normal.json)
create a module with a class. Read the json file with std.json and for every
operation in the json create a method with the same name in the class.
 For every shape in the json create a struct with the same name.

 Every operation defines either an input shape, output shape, 
 or both. During the method call serialize the input struct to 
 json, call the aws client with the json and deserialize the 
 output json to the output struct.

 Kind regards
 André


 Is there any docs from Amazon on this, beside json's itself?  I 
 will be very grateful for link.


I think as a starting point the console client help is very 
interesting.
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-using.html

Especially these two topics
-Generate CLI Skeleton and CLI Input JSON Parameters
-Controlling Command Output from the AWS Command Line Interface

Kind regards
André
Oct 18 2017
prev sibling parent reply Stephan Dilly <stephan extrawurst.org> writes:
On 2017-10-18 20:19:20 +0000, ikod said:


 Hello,
 
 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany wrote:

 On Wednesday, 18 October 2017 at 19:28:30 UTC, aberba wrote:
 The implementation is straight forward. For every api definition (e.g. 
 https://github.com/aws/aws-sdk-js/blob/master/apis/iam-2010-05-08.normal.json) 
 create a module with a class. Read the json file with std.json and for 
 every operation in the json create a method with the same name in the 
 class.
 For every shape in the json create a struct with the same name.
 
 Every operation defines either an input shape, output shape, or both. 
 During the method call serialize the input struct to json, call the aws 
 client with the json and deserialize the output json to the output 
 struct.
 
 Kind regards
 André


 
 Is there any docs from Amazon on this, beside json's itself?  I will be 
 very grateful for link.


Are you going to take over the task?

--Stephan
Oct 18 2017
parent reply ikod <geller.garry gmail.com> writes:
On Wednesday, 18 October 2017 at 20:51:48 UTC, Stephan Dilly 
wrote:

 On 2017-10-18 20:19:20 +0000, ikod said:


 Hello,
 
 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany 
 wrote:

 [...]


 
 Is there any docs from Amazon on this, beside json's itself?  
 I will be very grateful for link.


 Are you going to take over the task?

 --Stephan


I'd like, but It depends on the required time investment. I need 
some time to look at the docs etc...

Regards,
Igor
Oct 18 2017
parent reply Stephan Dilly <stephan extrawurst.org> writes:
On 2017-10-18 22:25:23 +0000, ikod said:


 On Wednesday, 18 October 2017 at 20:51:48 UTC, Stephan Dilly wrote:

 On 2017-10-18 20:19:20 +0000, ikod said:
 

 Hello,
 
 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany wrote:

 [...]


 
 Is there any docs from Amazon on this, beside json's itself?  I will be 
 very grateful for link.


 
 Are you going to take over the task?
 
 --Stephan


 
 I'd like, but It depends on the required time investment. I need some 
 time to look at the docs etc...
 
 Regards,
 Igor


Hi Igor,

I started a little tool (POC) that can parse the API spec and creates 
types and the API interface only for now:
https://github.com/Extrawurst/aws-sdk-dlang-gen

Its a start, not tested and does not come with any dependency to vibe 
or what. AWS specific request signing also needs to be done on top of 
it. but using that tool the types can be automatically generated.

--Stephan
Oct 18 2017
next sibling parent ikod <geller.garry gmail.com> writes:
On Wednesday, 18 October 2017 at 23:02:27 UTC, Stephan Dilly 
wrote:

 On 2017-10-18 22:25:23 +0000, ikod said:


 On Wednesday, 18 October 2017 at 20:51:48 UTC, Stephan Dilly 
 wrote:

 On 2017-10-18 20:19:20 +0000, ikod said:
 

 Hello,
 
 On Wednesday, 18 October 2017 at 20:05:28 UTC, Andre Pany 
 wrote:

 [...]


 
 Is there any docs from Amazon on this, beside json's itself?
  I will be very grateful for link.


 
 Are you going to take over the task?
 
 --Stephan


 
 I'd like, but It depends on the required time investment. I 
 need some time to look at the docs etc...
 
 Regards,
 Igor


 Hi Igor,

 I started a little tool (POC) that can parse the API spec and 
 creates types and the API interface only for now: 
 https://github.com/Extrawurst/aws-sdk-dlang-gen

 Its a start, not tested and does not come with any dependency 
 to vibe or what. AWS specific request signing also needs to be 
 done on top of it. but using that tool the types can be 
 automatically generated.

 --Stephan


Thanks, Stephan!

I'm playing with it, but it will take some time. I'm also looking 
into some other bindings, both generated from descriptions and 
not. I have some experience with s3 and ec2 using boto library, 
but generate library code, tie it with http, auth, e.t.c  is 
something different.
Oct 20 2017
prev sibling parent aberba <karabutaworld gmail.com> writes:
On Wednesday, 18 October 2017 at 23:02:27 UTC, Stephan Dilly 
wrote:

 On 2017-10-18 22:25:23 +0000, ikod said:


 On Wednesday, 18 October 2017 at 20:51:48 UTC, Stephan Dilly 
 wrote:

 On 2017-10-18 20:19:20 +0000, ikod said:
 

  [...]


 
 Are you going to take over the task?
 
 --Stephan


 
 I'd like, but It depends on the required time investment. I 
 need some time to look at the docs etc...
 
 Regards,
 Igor


 Hi Igor,

 I started a little tool (POC) that can parse the API spec and 
 creates types and the API interface only for now: 
 https://github.com/Extrawurst/aws-sdk-dlang-gen

 Its a start, not tested and does not come with any dependency 
 to vibe or what. AWS specific request signing also needs to be 
 done on top of it. but using that tool the types can be 
 automatically generated.

 --Stephan


I'm really happy about this.
Oct 28 2017