AWS Cloud Control API, a Uniform API to Access AWS & Third-Party Services

At the moment, I’m glad to announce the provision of AWS Cloud Management API a set of frequent utility programming interfaces (APIs) which might be designed to make it simple for builders to handle their AWS and third-party providers.

AWS delivers the broadest and deepest portfolio of cloud providers. Builders leverage these to construct any sort of cloud infrastructure. It began with Amazon Easy Storage Service (Amazon S3) 15 years in the past and grew over 200+ providers. Every AWS service has a particular API with its personal vocabulary, enter parameters, and error reporting. For instance, you utilize the S3 CreateBucket API to create an Amazon Easy Storage Service (Amazon S3) bucket and the Amazon Elastic Compute Cloud (Amazon EC2) RunInstances API to create an EC2 situations.

A few of you utilize AWS APIs to construct infrastructure-as-code, some to examine and robotically enhance your safety posture, some others for configuration administration, or to provision and to configure excessive efficiency compute clusters. The use instances are numerous.

As functions and infrastructures turn into more and more refined and you’re employed throughout extra AWS providers, it turns into more and more troublesome to study and handle distinct APIs. This problem is exacerbated while you additionally use third-party providers in your infrastructure, since it’s a must to construct and preserve customized code to handle each the AWS and third-party providers collectively.

Cloud Management API is an ordinary set of APIs to Create, Learn, Replace, Delete, and Checklist (CRUDL) assets throughout a whole bunch of AWS Providers (extra being added) and dozens of third-party providers (and rising).

It exposes 5 frequent verbs (CreateResource, GetResource, UpdateResource, DeleteResource, ListResource) to handle the lifecycle of providers. For instance, to create an Amazon Elastic Container Service (Amazon ECS) cluster or an AWS Lambda perform, you name the identical CreateResource API, passing as parameters the kind and attributes of the useful resource you wish to create: an Amazon ECS cluster or an Lambda perform. The enter parameters are outlined by an unified useful resource mannequin utilizing JSON. Equally, the return varieties and error messages are uniform throughout all verbs and all assets.

Cloud Management API offers help for a whole bunch of AWS assets at the moment, and we are going to proceed so as to add help for present AWS assets throughout providers resembling Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Easy Storage Service (Amazon S3) within the coming months. It’s going to help new AWS assets usually on the day of launch.

Till at the moment, after I wish to get the main points about an Lambda perform or a Amazon Kinesis stream, I take advantage of the get-function API to name Lambda and the describe-stream API to name Kinesis. Discover within the instance beneath how totally different these two API calls are: they’ve totally different names, totally different naming conventions, totally different JSON outputs, and many others.

aws lambda get-function --function-name TictactoeDatabaseCdkStack
{
    "Configuration": {
        "FunctionName": "TictactoeDatabaseCdkStack",
        "FunctionArn": "arn:aws:lambda:us-west-2:0123456789:perform:TictactoeDatabaseCdkStack",
        "Runtime": "nodejs14.x",
        "Position": "arn:aws:iam::0123456789:position/TictactoeDatabaseCdkStack",
        "Handler": "framework.onEvent",
        "CodeSize": 21539,
        "Timeout": 900,
        "MemorySize": 128,
        "LastModified": "2021-06-07T11:26:39.767+0000",

...

aws kinesis describe-stream --stream-name AWSNewsBlog
	

In distinction, when utilizing the Cloud Management API, I take advantage of a single API identify get-resource, and I obtain a constant output.

aws cloudcontrol get-resource        
    --type-name AWS::Kinesis::Stream 
    --identifier NewsBlogDemo
	

Similary, to create the useful resource above I used the create-resource API.

aws cloudcontrol create-resource    
   --type-name AWS::Kinesis::Stream 
   --desired-state ""

For my part, there are three sorts of builders which might be going to undertake Cloud Management API:

Builders
The primary neighborhood is builders utilizing AWS Providers APIs to handle their infrastructure or their buyer’s infrastructure. Those requiring utilization of low-level AWS Providers APIs slightly than increased stage instruments. For instance, I do know firms that manages AWS infrastructures on behalf of their purchasers. Many developed options to listing and describe all assets deployed of their shopper’s AWS Accounts, for administration and billing functions. Usually, they constructed particular instruments to handle their necessities, however discover it laborious to maintain up with new AWS Providers and options. Cloud Management API simplifies any such instruments by offering a constant, resource-centric method. It makes simpler to maintain up with new AWS Providers and options.

One other instance: Stedi is a developer-focused platform for constructing automated Digital Information Interchange (EDI) options that combine with any enterprise system. “We’ve a powerful give attention to infrastructure as code (IaC) inside Stedi and have been searching for a programmatic technique to uncover and delete legacy cloud assets which might be now not managed by CloudFormation – serving to us scale back complexity and handle price,” mentioned Olaf Conjin, Serverless Engineer at Stedi, Inc. “With AWS Cloud Management API, our groups can simply listing every of those legacy assets, cross-reference them in opposition to CloudFormation managed assets, apply further logic and delete the legacy assets. By deleting these unused legacy assets utilizing Cloud Management API, we are able to handle our cloud spend in a less complicated and sooner method. Cloud Management API permits us to take away the necessity to creator and preserve customized code to find and delete every sort of useful resource, serving to us enhance our developer velocity”.

APN Companions
The second neighborhood that advantages from Cloud Management API is APN Companions, resembling HashiCorp (maker of Terraform) and Pulumi, and different APN Companions providing options that depends on AWS Providers APIs. When AWS releases a brand new service or characteristic, our associate’s engineering groups must study, combine, and check a brand new set of AWS Service APIs to reveal it of their choices. It is a time consuming course of and infrequently results in a lag between the AWS launch and the provision of the service or characteristic of their answer. With the brand new Cloud Management API, companions are actually capable of construct a novel REST API code base, utilizing unified API verbs, frequent enter parameters, and customary error varieties. They simply must merge the standardized pre-defined uniform useful resource mannequin to work together with new AWS Providers uncovered as REST assets.

Launch Companions
HashiCorp and Pulumi are our launch companions, each options are built-in with Cloud Management API at the moment.

HashiCorp offers cloud infrastructure automation software program that allows organizations to provision, safe, join, and run any infrastructure for any utility. “AWS Cloud Management API makes it simpler for our groups to construct options to combine with new and present AWS providers,” mentioned James Bayer – EVP Product, HashiCorp. “Integrating HashiCorp Terraform with AWS Cloud Management API means builders are in a position to make use of the newly launched AWS options and providers, usually on the day of launch.”

Pulumi’s new AWS Native Supplier, powered by the AWS Cloud Management API, “offers Pulumi’s customers sooner entry to the newest AWS improvements, usually the day they launch, with none want for us to manually implement help,” mentioned Joe Duffy, CEO at Pulumi. “The total floor space of AWS assets offered by AWS Cloud Management API can now be automated from acquainted languages like Python, TypeScript, .NET, and Go, with customary IDEs, bundle managers, and check frameworks, with excessive constancy and nice high quality. Utilizing this new supplier, builders and infrastructure groups can develop and ship trendy AWS functions and infrastructure sooner and with extra confidence than ever earlier than.”

To study extra about HashiCorp and Pulumi’s integration with Cloud Management API, consult with their weblog submit and bulletins.

AWS Prospects
The third sort of builders that can profit from Cloud Management API is AWS prospects utilizing answer resembling Terraform or Pulumi. You’ll be able to profit from Cloud Management API too. For instance, when utilizing the brand new Terraform AWS Cloud Management supplier or Pulumi’s AWS Native Supplier, you’ll be able to profit from availability of latest AWS Providers and options usually on the day of launch.

Now that you just perceive the advantages, let’s see Cloud Management API in motion.

How It Works?
To start out utilizing Cloud Management API, I first make certain I take advantage of the newest AWS Command Line Interface (CLI) model. Relying on how the CLI was put in, there are totally different strategies to replace the CLI. Cloud Management API is out there from our AWS SDKs as effectively.

To create an AWS Lambda perform, I first create an index.py handler, I zip it, and add the zip file to one among my non-public bucket. I concentrate that the S3 bucket is in the identical AWS Area the place I’ll create the Lambda perform:

cat << EOF > index.py  
heredoc> import json 
def lambda_handler(occasion, context):
    return 
        'statusCode': 200,
        'physique': json.dumps('Hiya from Lambda!')
    
EOF

zip index.zip index.py
aws s3 cp index.zip s3://private-bucket-seb/index.zip

Then, I name the create-resource API, passing the identical set of arguments as required by the corresponding CloudFormation useful resource. On this instance, the Code, Position, Runtime, and Handler arguments are obligatory, as per the CloudFormation AWS::Lambda::Operate documentation.

aws cloudcontrol create-resource          
       --type-name AWS::Lambda::Operate   
       --desired-state '"Code":,"Position":"arn:aws:iam::0123456789:position/lambda_basic_execution","Runtime":"python3.9","Handler":"index.lambda_handler"' 
       --client-token 123



    "ProgressEvent": 
        "TypeName": "AWS::Lambda::Operate",
        "RequestToken": "56a0782b-2b26-491c-b082-18f63d571bbd",
        "Operation": "CREATE",
        "OperationStatus": "IN_PROGRESS",
        "EventTime": "2021-09-26T12:05:42.210000+02:00"
    

I could name the identical command once more to get the standing or to find out about an eventual error:

aws cloudcontrol create-resource          
       --type-name AWS::Lambda::Operate   
       --desired-state '"Code":,"Position":"arn:aws:iam::0123456789:position/lambda_basic_execution","Runtime":"python3.9","Handler":"index.lambda_handler"' 
       --client-token 123


    "ProgressEvent": 

Right here, the OperationStatus is SUCCESS and the perform identify is ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 (I can move my very own identify if I need one thing extra descriptive 🙂 )

I then invoke the Lambda perform to make sure it really works as anticipated:

aws lambda invoke 
    --function-name ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 
    out.txt && cat out.txt && rm out.txt 


"statusCode": 200, "physique": ""Hiya from Lambda!""

When completed, I delete the Lambda perform utilizing Cloud Management API:

aws cloudcontrol delete-resource 
     --type-name AWS::Lambda::Operate 
     --identifier ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9 

    "ProgressEvent": 
        "TypeName": "AWS::Lambda::Operate",
        "Identifier": "ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9",
        "RequestToken": "8923991d-72b3-4981-8160-4d9a585965a3",
        "Operation": "DELETE",
        "OperationStatus": "IN_PROGRESS",
        "EventTime": "2021-09-26T20:06:22.013000+02:00"
    

Idempotency
You may need observed the client-token parameter I handed to the create-resource API name. Create, Replace, and Delete requests all settle for a ClientToken, which is used to make sure idempotency of the request.

• We advocate at all times passing a shopper token. It will disambiguate requests in case a retry is required. In any other case, it’s possible you’ll encounter surprising errors like ConcurrentOperationException or AlreadyExists.
• We advocate that shopper tokens at all times be distinctive for each single request, resembling by passing a UUID.

One Extra Factor
On the coronary heart of AWS Cloud Management API supply of information, there may be the CloudFormation Public Registry, which my colleague Steve introduced final June on this weblog submit. It permits anybody to reveal a set of AWS assets by CloudFormation and AWS CDK. That is the mechanism AWS Service groups are actually utilizing to launch their providers and options as CloudFormation and AWS CDK assets. A number of third-party distributors are additionally publishing their options within the CloudFormation Public Registry. All assets printed are modelled with an ordinary schema that defines the useful resource, its properties, and their attributes in a uniform approach.

AWS Cloud Management API is a CRUDL API layer on prime of assets printed within the CloudFormation Public Registry. Any useful resource printed within the registry exposes its attributes with customary JSON schemas. The useful resource can then be created, up to date, deleted, or listed utilizing Cloud Management API with no further work.

For instance, think about I determine to reveal a public CloudFormation stack to let any AWS buyer create VPN servers, primarily based on EC2 situations. I mannequin the VPNServer useful resource sort and publish it within the CloudFormation Public Registry. With no further work on my aspect, my customized useful resource “VPNServer” is now accessible to all AWS prospects by the Cloud Management API REST API. Not solely, it’s also robotically accessible by options like Hashicorp’s Terraform and Pulumi, and probably others who undertake Cloud Management API sooner or later.

It’s price mentioning Cloud Management API isn’t geared toward changing the normal AWS service-level APIs. They’re nonetheless there and can at all times be there, however we expect that Cloud Management API is less complicated and extra constant to make use of and you need to use it for brand new apps.

Availability and Pricing
Cloud Management API is out there in all AWS Areas, besides China.

You’ll solely pay for the utilization of underlying AWS assets, resembling a CloudWatch logs or Lambda capabilities invocations, or pay for the variety of handler operations and handler operation period related to utilizing third-party assets (resembling Datadog screens or MongoDB Atlas clusters). There are not any minimal charges and no required upfront commitments.

I can’t wait to find what you’ll construct on prime of this new Cloud Management API. Go construct!

— seb