Introducing a Public Registry for AWS CloudFormation

AWS CloudFormation and the AWS Cloud Improvement Package (CDK) present scalable and constant provisioning of AWS sources (for instance, compute infrastructure, monitoring instruments, databases, and extra). We’ve heard from many purchasers that they’d like to learn from the identical consistency and scalability when provisioning sources from AWS Accomplice Community (APN) members, third-party distributors, and open-source applied sciences, no matter whether or not they’re utilizing CloudFormation templates or have adopted the CDK to outline their cloud infrastructure.

I’m happy to announce a brand new public registry for CloudFormation, offering a searchable assortment of extensions – useful resource sorts or modules – printed by AWS, APN companions, third events, and the developer group. The registry makes it simple to find and provision these extensions in your CloudFormation templates and CDK purposes in the identical method you employ AWS-provided sources. Utilizing extensions, you not have to create and preserve customized provisioning logic for useful resource sorts from third-party distributors. And, you’ll be able to use a single infrastructure as code instrument, CloudFormation, to provision and handle AWS and third-party sources, additional simplifying the infrastructure provisioning course of (the CDK makes use of CloudFormation beneath the hood).

Launch Companions
We’re excited to be joined by over a dozen APN Companions for the launch of the registry, with greater than 35 extensions obtainable so that you can use in the present day. Weblog posts and bulletins from the APN Companions who collaborated on this launch, together with AWS Fast Begins, will be discovered beneath (some will probably be added within the subsequent few days).

Registries and Useful resource Sorts
In 2019, CloudFormation launched assist for personal registries. These enabled registration and use of useful resource suppliers (Lambda capabilities) in your account, together with suppliers from AWS and third-party distributors. After you registered a supplier you may use useful resource sorts, comprised of customized provisioning logic, from the supplier in your CloudFormation templates. Useful resource sorts have been uploaded by suppliers to an Amazon Easy Storage Service (Amazon S3) bucket, and also you used the kinds by referencing the related S3 URL. The general public registry offers consistency within the sourcing of useful resource sorts and modules, and also you not want to make use of a group of Amazon Easy Storage Service (Amazon S3) buckets.

Third-party useful resource sorts within the public registry additionally combine with drift detection. After making a useful resource from a third-party useful resource sort, CloudFormation will detect adjustments to the useful resource from its template configuration, often called configuration drift, simply as it might with AWS sources. You too can use AWS Config to handle compliance for third-party sources consumed from the registry. The useful resource sorts are robotically tracked as Configuration objects when you have got configured AWS Config to file them, and used CloudFormation to create, replace, and delete them. Whether or not the useful resource sorts you employ are third-party or AWS sources, you’ll be able to view configuration historical past for them, along with with the ability to write AWS Config guidelines to confirm configuration finest practices.

The general public registry additionally helps Kind Configuration, enabling you to configure third-party useful resource sorts with API keys and OAuth tokens per account and area. As soon as set, the configuration is saved securely and will be up to date. This additionally offers a centralized approach to configure third-party useful resource sorts.

Publishing Extensions to the Public Registry
Extension publishers should be verified as AWS Market sellers, or as GitHub or BitBucket customers, and extensions are validated towards finest practices. To publish extensions (useful resource sorts or modules) to the registry, you could first register in an AWS Area, utilizing one of many talked about account sorts.

After you’ve registered, you subsequent publish your extension to a personal registry in the identical Area. Then, it is advisable to check that the extension meets publishing necessities. For a useful resource sort extension, this implies it should go all of the contract exams outlined for the sort. Modules are topic to completely different necessities, and you could find extra particulars within the documentation. With testing full, you’ll be able to publish your extension to the general public registry to your Area. See the consumer information for detailed data on publishing extensions.

Utilizing Extensions within the Public Registry
I made a decision to attempt a few extensions associated to Kubernetes, contributed by AWS Fast Begins, to make configuration adjustments to a cluster. Personally, I don’t have quite a lot of expertise with Kubernetes and its API so this was a terrific likelihood to look at how extensions may save me important effort and time. In the course of the technique of penning this put up I realized from others that utilizing the Kubernetes API (the same old approach to obtain the adjustments I had in thoughts) would usually contain effort even for these with extra expertise.

For this instance I wanted a Kubernetes cluster, so I adopted this tutorial to set one up in Amazon Elastic Kubernetes Service (EKS), utilizing the Managed nodes – Linux node sort. With my cluster prepared, I need to make two configuration adjustments.

First, I need to add a brand new namespace to the cluster. A namespace is a partitioning assemble that lets me deploy the identical set of sources to completely different namespaces in the identical cluster with out battle due to the isolation namespaces present. Second, I need to arrange and use Helm, a bundle supervisor for Kubernetes. I’ll use Helm to put in the kube-state-metrics bundle from the Prometheus helm-charts repository for gathering cluster metrics. Whereas I can use CloudFormation to provision clusters and compute sources, beforehand, to carry out these two configuration duties, I’d have needed to change to the API or varied bespoke instrument chains. With the registry, and these two extensions, I can now do all the things utilizing CloudFormation (and naturally, as I discussed earlier, I may additionally use the extensions with the CDK, which I’ll present later).

Earlier than utilizing an extension, it must be activated in my account. Whereas activation is simple to do for single accounts utilizing the console, as we’ll see in a second, if I have been utilizing AWS Organizations and needed to activate varied third-party extensions throughout my whole group, or for a particular group unit (OU), I may obtain this utilizing Service-Managed StackSets in CloudFormation. Utilizing the useful resource sort AWS::CloudFormation::TypeActivation in a template submitted to a Service-Managed StackSet, I can goal a whole Group, or a specific OU, passing the Amazon Useful resource Identify (ARN) figuring out the third-party extension to be activated. Activation of extensions can be very simple to realize (whether or not utilizing AWS Organizations or not) utilizing the CDK with just some strains of code, once more making use of the aforementioned TypeActivation useful resource sort.

To activate the extensions, I head to the CloudFormation console and click on Public extensions from the navigation bar. This takes me to the Registry:Public extensions dwelling web page, the place I change to viewing third occasion useful resource sort extensions.

The extensions I need are AWSQS::Kubernetes::Useful resource and AWSQS::Kubernetes::Helm. The Useful resource extension is used to use a manifest describing configuration adjustments to a cluster. In my case, the manifest requests a namespace be created. Clicking the title of the AWSQS::Kubernetes::Useful resource extension takes me to a web page the place I can view schema, configuration particulars, and variations for the extension.

What occurs should you deactivate an extension you’re utilizing, or an extension is withdrawn by the writer? When you deactivate an extension a stack is determined by, any sources created from that extension received’t be affected, however you’ll be unable to carry out additional stack operations, corresponding to Learn, Replace, Delete, and Record (these will fail till the extension is re-activated). Publishers should request their extensions be withdrawn from the registry (there isn’t a “delete” API). If the request is granted, clients who activated the extension previous to withdrawal can nonetheless carry out Create/Learn/Replace/Delete/Record operations, utilizing what’s successfully a snapshot of the extension of their account.

Clicking Activate takes me to a web page the place I have to specify the ARN of an execution position that CloudFormation will assume when it runs the code behind the extension. I create a job following this consumer information matter, however the fundamental belief relationship is beneath for reference.

I additionally add permissions for the useful resource sorts I’m utilizing to my execution position. Particulars on the permissions wanted for the kinds I selected will be discovered on GitHub, for Helm, and for Kubernetes (word the GitHub examples embrace the belief relationship too).

When activating an extension, I can elect to make use of the default title, which is how I’ll confer with the sort in my templates or CDK purposes, or I can enter a brand new title. The title chosen must be distinctive inside my account, so if I’ve enabled a model of an extension with its default title, and need to allow a special model, I need to change the title. As soon as I’ve stuffed within the particulars, and chosen my versioning technique (extensions use semantic versioning, and I can elect to simply accept computerized updates for minor model adjustments, or to “lock” to a particular model) clicking Activate extension completes the method.

That completes the method for the primary extension, and I observe the identical steps for the AWSQS::Kubernetes::Helm extension. Navigating to Activated extensions I can view a listing of all my enabled extensions.

I’ve yet another set of permissions to replace. Useful resource sorts make calls to the Kubernetes API on my behalf so I have to replace the aws-auth ConfigMap for my cluster to reference the execution position I simply used, in any other case the calls made by the useful resource sorts I’m utilizing will fail. To do that, I run the command kubectl edit cm aws-auth -n kube-system at a command immediate. Within the textual content editor that opens, I replace the ConfigMap with a brand new group referencing my CfnRegistryExtensionExecRole, proven beneath (should you’re following alongside, be sure you change the account ID and position title to match yours).

apiVersion: v1
information:
  mapRoles: |
    - teams:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::111122223333:position/myAmazonEKSNodeRole
      username: system:node:
    - teams:
      - system:masters
      rolearn: arn:aws:iam::111122223333:position/CfnRegistryExtensionExecRole
      username: cfnresourcetypes
form: ConfigMap
metadata:
  creationTimestamp: "2021-06-04T20:44:24Z"
  title: aws-auth
  namespace: kube-system
  resourceVersion: "6355"
  selfLink: /api/v1/namespaces/kube-system/configmaps/aws-auth
  uid: dc91bfa8-1663-45d0-8954-1e841913b324

Now I’m prepared to make use of the extensions to configure my cluster with a brand new namespace, Helm, and the kube-state-metrics bundle. I create a CloudFormation template that makes use of the extensions, including parameters for the weather I need to specify when making a stack: the title of the cluster to replace, and the namespace title. The properties for the KubeStateMetrics useful resource reference the bundle I need Helm to put in.

AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  ClusterName:
    Kind: String
  Namespace:
    Kind: String
Assets:
  KubeStateMetrics:
    Kind: AWSQS::Kubernetes::Helm
    Properties:
      ClusterID: !Ref ClusterName
      Identify: kube-state-metrics
      Namespace: !GetAtt KubeNamespace.Identify
      Repository: https://prometheus-community.github.io/helm-charts
      Chart: prometheus-community/kube-state-metrics
  KubeNamespace:
    Kind: AWSQS::Kubernetes::Useful resource
    Properties:
      ClusterName: !Ref ClusterName
      Namespace: default
      Manifest: !Sub |
        apiVersion: v1
        form: Namespace
        metadata:
          title: $Namespace
          labels:
            title: $Namespace

On the Stacks web page of the CloudFormation console, I click on Create stack, add my template, after which give my stack a reputation and the values for my declared parameters.

I click on Subsequent to proceed by way of the remainder of the wizard, leaving different settings at their default values, after which Create stack to finish the method.

As soon as stack creation is full, I confirm my adjustments utilizing the kubectl command line instrument. I first verify that the brand new namespace, newsblog-sample-namespace, is current with the command kubectl get namespaces. I then run the kubectl get all --namespace newsblog-sample-namespace command to confirm the kube-state-metrics bundle is put in.

Extensions will also be used with the AWS Cloud Improvement Package. To wrap up this exploration of utilizing the brand new registry, I’ve included an instance beneath of a CDK software snippet in TypeScript that achieves the identical impact, utilizing the identical extensions, because the YAML template I confirmed earlier (I may even have written this utilizing any of the languages supported by the CDK – C#, Java, or Python).

import Stack, Assemble, CfnResource from '@aws-cdk/core';
export class UnoStack extends Stack {
  constructor(scope: Assemble, id: string) {
    tremendous(scope, id);
    const clusterName="newsblog-cluster";
    const namespace="newsblog-sample-namespace";

    const kubeNamespace = new CfnResource(this, 'KubeNamespace', {
      sort: 'AWSQS::Kubernetes::Useful resource',
      properties: {
        ClusterName: clusterName,
        Namespace: 'default',
        Manifest: this.toJsonString(
          apiVersion: 'v1',
          form: 'Namespace',
          metadata: ,
        ),
      },
    });
    
    new CfnResource(this, 'KubeStateMetrics', 
      sort: 'AWSQS::Kubernetes::Helm',
      properties: 
        ClusterID: clusterName,
        Identify: 'kube-state-metrics',
        Namespace: kubeNamespace.getAtt('Identify').toString(),
        Repository: 'https://prometheus-community.github.io/helm-charts',
        Chart: 'prometheus-community/kube-state-metrics',
      ,
    );
  }
};

As talked about earlier on this put up, I don’t have a lot expertise with the Kubernetes API, and Kubernetes basically. Nevertheless, by making use of the useful resource sorts within the public registry, together with CloudFormation, I used to be capable of simply configure my cluster utilizing a well-recognized setting, without having to resort to the API or bespoke instrument chains.

Get Began with the CloudFormation Public Registry
Pricing for the general public registry is similar as for the present registry and personal useful resource sorts. There isn’t any extra cost for utilizing native AWS useful resource sorts; for third-party useful resource sorts you’ll incur fees based mostly on the variety of handler operations (add, delete, record, and so on.) you run per 30 days. For particulars, see the AWS CloudFormation Pricing web page. The brand new public registry is accessible in the present day within the US East (N. Virginia, Ohio), US West (Oregon, N. California), Canada (Central), Europe (Eire, Frankfurt, London, Stockholm, Paris, Milan), Asia Pacific (Hong Kong, Mumbai, Osaka, Singapore, Sydney, Seoul, Tokyo), South America (Sao Paulo), Center East (Bahrain), and Africa (Cape City) AWS Areas.

For extra data, see the AWS CloudFormation Person Information and Person Information for Extension Improvement, and begin publishing or utilizing extensions in the present day!

— Steve