Introducing Amazon S3 Object Lambda – Use Your Code to Process Data as It Is Being Retrieved from S3

Whenever you retailer knowledge in Amazon Easy Storage Service (S3), you possibly can simply share it to be used by a number of purposes. Nevertheless, every utility has its personal necessities and may have a unique view of the info. For instance, a dataset created by an e-commerce utility could embrace personally identifiable data (PII) that’s not wanted when the identical knowledge is processed for analytics and ought to be redacted. On the opposite aspect, if the identical dataset is used for a advertising and marketing marketing campaign, chances are you’ll want to counterpoint the info with further particulars, corresponding to data from the client loyalty database.

To offer completely different views of knowledge to a number of purposes, there are at the moment two choices. You both create, retailer, and preserve further by-product copies of the info, so that every utility has its personal customized dataset, otherwise you construct and handle infrastructure as a proxy layer in entrance of S3 to intercept and course of knowledge as it’s requested. Each choices add complexity and prices, so the S3 staff determined to construct a greater resolution.

At present, I’m very pleased to announce the provision of S3 Object Lambda, a brand new functionality that permits you to add your personal code to course of knowledge retrieved from S3 earlier than returning it to an utility. S3 Object Lambda works together with your current purposes and makes use of AWS Lambda capabilities to mechanically course of and rework your knowledge as it’s being retrieved from S3. The Lambda perform is invoked inline with a regular S3 GET request, so that you don’t want to alter your utility code.

On this method, you possibly can simply current a number of views from the identical dataset, and you may replace the Lambda capabilities to switch these views at any time.

There are numerous use circumstances that may be simplified by this method, for instance:

• Redacting personally identifiable data for analytics or non-production environments.
• Changing throughout knowledge codecs, corresponding to changing XML to JSON.
• Augmenting knowledge with data from different companies or databases.
• Compressing or decompressing information as they’re being downloaded.
• Resizing and watermarking photographs on the fly utilizing caller-specific particulars, such because the person who requested the article.
• Implementing customized authorization guidelines to entry knowledge.

You can begin utilizing S3 Object Lambda with just a few easy steps:

1. Create a Lambda Perform to remodel knowledge in your use case.
2. Create an S3 Object Lambda Entry Level from the S3 Administration Console.
3. Choose the Lambda perform that you simply created above.
4. Present a supporting S3 Entry Level to provide S3 Object Lambda entry to the unique object.
5. Replace your utility configuration to make use of the brand new S3 Object Lambda Entry Level to retrieve knowledge from S3.

To get a greater understanding of how S3 Object Lambda works, let’s put it in observe.

Learn how to Create a Lambda Perform for S3 Object Lambda
To create the perform, I begin by wanting on the syntax of the enter occasion the Lambda perform receives from S3 Object Lambda:

    "xAmzRequestId": "1a5ed718-5f53-471d-b6fe-5cf62d88d02a",
    "getObjectContext": 
        "inputS3Url": "https://myap-123412341234.s3-accesspoint.us-east-1.amazonaws.com/s3.txt?X-Amz-Safety-Token=...",
        "outputRoute": "io-iad-cell001",
        "outputToken": "..."
    ,
    "configuration": 
        "accessPointArn": "arn:aws:s3-object-lambda:us-east-1:123412341234:accesspoint/myolap",
        "supportingAccessPointArn": "arn:aws:s3:us-east-1:123412341234:accesspoint/myap",
        "payload": "check"
    ,
    "userRequest": ,
    "userIdentity": ,
    "protocolVersion": "1.00"

The getObjectContext property incorporates among the most helpful data for the Lambda perform:

• The inputS3Url is a presigned URL that the perform can use to obtain the unique object from the supporting Entry Level. On this method, the Lambda perform doesn’t must have S3 learn permissions to retrieve the unique object and might solely entry the article processed by every invocation.
• The outputRoute and the outputToken are two parameters which are used to ship again the modified object utilizing the brand new WriteGetObjectResponse API.

The configuration property incorporates the Amazon Useful resource Title (ARN) of the Object Lambda Entry Level and of the supporting Entry Level.

The userRequest property offers extra data of the unique request, corresponding to the trail within the URL, and the HTTP headers.

Lastly, the userIdentity part returns the small print of who made the unique request and can be utilized to customise entry to the info.

Now that I do know the syntax of the occasion, I can create the Lambda perform. To maintain issues easy, right here’s a perform written in Python that modifications all textual content within the unique object to uppercase:

import boto3
import requests

def lambda_handler(occasion, context):
    print(occasion)

    object_get_context = occasion["getObjectContext"]
    request_route = object_get_context["outputRoute"]
    request_token = object_get_context["outputToken"]
    s3_url = object_get_context["inputS3Url"]

    # Get object from S3
    response = requests.get(s3_url)
    original_object = response.content material.decode('utf-Eight')

    # Rework object
    transformed_object = original_object.higher()

    # Write object again to S3 Object Lambda
    s3 = boto3.consumer('s3')
    s3.write_get_object_response(
        Physique=transformed_object,
        RequestRoute=request_route,
        RequestToken=request_token)

    return 'status_code': 200

Trying on the code of the perform, there are three predominant sections:

• First, I exploit the inputS3Url property of the enter occasion to obtain the unique object. Because the worth is a presigned URL, the perform doesn’t want permissions to learn from S3.
• Then, I rework the textual content to be all uppercase. To customise the conduct of the perform in your use case, that is the half you have to change. For instance, to detect and redact personally identifiable data (PII), I can use Amazon Comprehend to find PII entities with the DetectPiiEntities API and change them with asterisks or an outline of the redacted entity sort.
• Lastly, I exploit the brand new WriteGetObjectResponse API to ship the results of the transformation again to S3 Object Lambda. On this method, the reworked object could be a lot bigger than the utmost measurement of the response returned by a Lambda perform. For bigger objects, the WriteGetObjectResponse API helps chunked switch encoding to implement a streaming knowledge switch. The Lambda perform solely must return the standing code (200 OK on this case), eventual errors, and optionally customise the metadata of the returned object as described within the S3 GetObject API.

I bundle the perform, together with the dependencies, and add it to Lambda. Notice that the utmost period for a Lambda perform utilized by S3 Object Lambda is 60 seconds, and that the Lambda perform wants AWS Id and Entry Administration (IAM) permissions to name the WriteGetObjectResponse API.

Learn how to Create an S3 Object Lambda Entry Level from the Console
Within the S3 console, I create an S3 Entry Level on one among my S3 buckets:

Then, I create an S3 Object Lambda Entry Level utilizing the supporting Entry Level I simply created. The Lambda perform goes to make use of the supporting Entry Level to obtain the unique objects.

Through the configuration of the S3 Object Lambda Entry Level as proven beneath, I choose the most recent model of the Lambda perform I created above. Optionally, I can allow help for requests utilizing a byte vary, or utilizing half numbers. For now, I depart them disabled. To know the way to use byte vary and half numbers with S3 Object Lambda, please see the documentation.

When configuring the S3 Object Lambda Entry Level, I can arrange a string as a payload that’s handed to the Lambda perform in all invocations coming from that Entry Level, as you possibly can see within the configuration property of the pattern occasion I described earlier than. On this method, I can configure the identical Lambda perform for a number of S3 Object Lambda Entry Factors, and use the worth of thepayload to customise the conduct for every of them.

Lastly, I can arrange a coverage, much like what I can do with regular S3 Entry Factors, to supply entry to the objects accessible by means of this Object Lambda Entry Level. For now, I preserve the coverage empty. Then, I depart the default possibility to dam all public entry and create the Object Lambda Entry Level.

Now that the S3 Object Lambda Entry Level is prepared, let’s see how I can use it.

Learn how to Use the S3 Object Lambda Entry Level
Within the S3 console, I choose the newly created Object Lambda Entry Level. Within the properties, I copy the ARN to have it out there later.

With the AWS Command Line Interface (CLI), I add a textual content file containing just a few sentences to the S3 bucket behind the S3 Object Lambda Entry Level:

aws cp s3.txt s3://danilop-data/

Utilizing S3 Object Lambda with my current purposes could be very easy. I simply want to exchange the S3 bucket with the ARN of the S3 Object Lambda Entry Level and replace the AWS SDKs to simply accept the brand new syntax utilizing the S3 Object Lambda ARN.

For instance, it is a Python script that downloads the textual content file I simply uploaded: first, straight from the S3 bucket, after which from the S3 Object Lambda Entry Level. The one distinction between the 2 downloads is the worth of the Bucket parameter.

import boto3

s3 = boto3.consumer('s3')

print('Authentic object from the S3 bucket:')
unique = s3.get_object(
  Bucket="danilop-data",
  Key='s3.txt')
print(unique['Body'].learn().decode('utf-Eight'))

print('Object processed by S3 Object Lambda:')
reworked = s3.get_object(
  Bucket="arn:aws:s3-object-lambda:us-east-1:123412341234:accesspoint/myolap",
  Key='s3.txt')
print(reworked['Body'].learn().decode('utf-Eight'))

I begin the script on my laptop computer:

python3 read_original_and_transformed_object.py

And that is the consequence I get:

Authentic object on S3:
Amazon Easy Storage Service (Amazon S3) is an object storage service that gives industry-leading scalability, knowledge availability, safety, and efficiency. This implies prospects of all sizes and industries can use it to retailer and shield any quantity of knowledge for a variety of use circumstances, corresponding to knowledge lakes, web sites, cell purposes, backup and restore, archive, enterprise purposes, IoT gadgets, and massive knowledge analytics.


Object processed by S3 Object Lambda:
AMAZON SIMPLE STORAGE SERVICE (AMAZON S3) IS AN OBJECT STORAGE SERVICE THAT OFFERS INDUSTRY-LEADING SCALABILITY, DATA AVAILABILITY, SECURITY, AND PERFORMANCE. THIS MEANS CUSTOMERS OF ALL SIZES AND INDUSTRIES CAN USE IT TO STORE AND PROTECT ANY AMOUNT OF DATA FOR A RANGE OF USE CASES, SUCH AS DATA LAKES, WEBSITES, MOBILE APPLICATIONS, BACKUP AND RESTORE, ARCHIVE, ENTERPRISE APPLICATIONS, IOT DEVICES, AND BIG DATA ANALYTICS.


The primary output is downloaded straight from the supply bucket, and I see the unique content material as anticipated. The second time, the article is processed by the Lambda perform as it’s being retrieved and, because the consequence, all textual content is uppercase!

Extra Use Instances for S3 Object Lambda
When retrieving an object utilizing S3 Object Lambda, there isn’t any want for an object with the identical title to exist within the S3 bucket. The Lambda perform can use data within the title of the file or within the HTTP headers to generate a customized object.

For instance, if you happen to ask to make use of an S3 Object Lambda Entry Level for a picture with title sunset_600x400.jpg, the Lambda perform can search for a picture named sundown.jpg and resize it to suit the utmost width and peak as described within the file title. On this case, the Lambda perform would wish entry permission to learn the unique picture, as a result of the article key’s completely different from what was used within the presigned URL.

One other fascinating use case could be to retrieve JSON or CSV paperwork, corresponding to order.json or objects.csv, which are generated on the fly primarily based on the content material of a database. The metadata within the request HTTP headers can be utilized to go the orderId to make use of. As standard, I anticipate our prospects’ creativity to far exceed the use circumstances I described right here.

Right here’s a brief video describing how S3 Object Lambda works and the way you need to use it:

Availability and Pricing
S3 Object Lambda is obtainable at the moment in all AWS Areas apart from the Asia Pacific (Osaka), AWS GovCloud (US-East), AWS GovCloud (US-West), China (Beijing), and China (Ningxia) Areas. You need to use S3 Object Lambda with the AWS Administration Console, AWS Command Line Interface (CLI), and AWS SDKs. Presently, the AWS CLI high-level S3 instructions, corresponding to aws s3 cp, don’t help objects from S3 Object Lambda Entry Factors, however you need to use the low-level S3 API instructions, corresponding to aws s3api get-object.

With S3 Object Lambda, you pay for the AWS Lambda compute and request expenses required to course of the info, and for the info S3 Object Lambda returns to your utility. You additionally pay for the S3 requests which are invoked by your Lambda perform. For extra pricing data, please see the Amazon S3 pricing web page.

This new functionality makes it a lot simpler to share and convert knowledge throughout a number of purposes.

Begin utilizing S3 Object Lambda to simplify your storage structure at the moment.

— Danilo