Mountpoint for Amazon S3 – Generally Available and Ready for Production Workloads

Mountpoint for Amazon S3 is an open supply file consumer that makes it simple to your file-aware Linux purposes to attach on to Amazon Easy Storage Service (Amazon S3) buckets. Introduced earlier this yr as an alpha launch, it’s now usually accessible and prepared for manufacturing use in your large-scale read-heavy purposes: knowledge lakes, machine studying coaching, picture rendering, autonomous automobile simulation, ETL, and extra. It helps file-based workloads that carry out sequential and random reads, sequential (append solely) writes, and that don’t want full POSIX semantics.

Why Information?
Many AWS clients use the S3 APIs and the AWS SDKs to construct purposes that may record, entry, and course of the contents of an S3 bucket. Nonetheless, many shoppers have present purposes, instructions, instruments, and workflows that know the best way to entry recordsdata in UNIX model: studying directories, opening & studying present recordsdata, and creating & writing new ones. These clients have requested us for an official, enterprise-ready consumer that helps performant entry to S3 at scale. After talking with these clients and asking plenty of questions, we realized that efficiency and stability have been their main considerations, and that POSIX compliance was not a necessity.

After I first wrote about Amazon S3 again in 2006 I used to be very clear that it was supposed for use as an object retailer, not as a file system. Whereas you wouldn’t need use the Mountpoint / S3 combo to retailer your Git repositories or the like, utilizing it along with instruments that may learn and write recordsdata, whereas profiting from S3’s scale and sturdiness, is sensible in lots of conditions.

All About Mountpoint
Mountpoint is conceptually quite simple. You create a mount level and mount an Amazon S3 bucket (or a path inside a bucket) on the mount level, after which entry the bucket utilizing shell instructions (ls, cat, dd, discover, and so forth), library features (open, shut, learn, write, creat, opendir, and so forth) or equal instructions and features as supported within the instruments and languages that you just already use.

Below the covers, the Linux Digital Filesystem (VFS) interprets these operations into calls to Mountpoint, which in turns interprets them into calls to S3: LIST, GET, PUT, and so forth. Mountpoint strives to make good use of community bandwidth, growing throughput and permitting you to cut back your compute prices by getting extra work performed in much less time.

Mountpoint can be utilized from an Amazon Elastic Compute Cloud (Amazon EC2) occasion, or inside an Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (EKS) container. It can be put in in your present on-premises techniques, with entry to S3 both straight or over an AWS Direct Join connection by way of AWS PrivateLink for Amazon S3.

Putting in and Utilizing Mountpoint for Amazon S3
Mountpoint is on the market in RPM format and might simply be put in on an EC2 occasion working Amazon Linux. I merely fetch the RPM and set up it utilizing yum:

$ wget https://s3.amazonaws.com/mountpoint-s3-release/newest/x86_64/mount-s3.rpm
$ sudo yum set up ./mount-s3.rpm

For the final couple of years I’ve been commonly fetching photographs from a number of of the Washington State Ferry webcams and storing them in my wsdot-ferry bucket:

I gather these photographs with a view to monitor the comings and goings of the ferries, with a purpose of analyzing them in some unspecified time in the future to seek out one of the best occasions to trip. My purpose immediately is to create a film that mixes a complete day’s price of photographs into a pleasant time lapse. I begin by making a mount level and mounting the bucket:

$ mkdir wsdot-ferry
$  mount-s3 wsdot-ferry wsdot-ferry

I can traverse the mount level and examine the bucket:

$ cd wsdot-ferry
$ ls -l | head -10
complete zero
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2020_12_30
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2020_12_31
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2021_01_01
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2021_01_02
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2021_01_03
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2021_01_04
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2021_01_05
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2021_01_06
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 2021_01_07
$
$  cd 2020_12_30
$ ls -l
complete zero
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 fauntleroy_holding
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 fauntleroy_way
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 lincoln
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 trenton
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 vashon_112_north
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 vashon_112_south
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 vashon_bunker_north
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 vashon_bunker_south
drwxr-xr-x 2 jeff jeff zero Aug  7 23:07 vashon_holding
$
$ cd fauntleroy_holding
$  ls -l | head -10
complete 2680
-rw-r--r-- 1 jeff jeff  19337 Feb 10  2021 17-12-01.jpg
-rw-r--r-- 1 jeff jeff  19380 Feb 10  2021 17-15-01.jpg
-rw-r--r-- 1 jeff jeff  19080 Feb 10  2021 17-18-01.jpg
-rw-r--r-- 1 jeff jeff  17700 Feb 10  2021 17-21-01.jpg
-rw-r--r-- 1 jeff jeff  17016 Feb 10  2021 17-24-01.jpg
-rw-r--r-- 1 jeff jeff  16638 Feb 10  2021 17-27-01.jpg
-rw-r--r-- 1 jeff jeff  16713 Feb 10  2021 17-30-01.jpg
-rw-r--r-- 1 jeff jeff  16647 Feb 10  2021 17-33-02.jpg
-rw-r--r-- 1 jeff jeff  16750 Feb 10  2021 17-36-01.jpg
$

I can create my animation with a single command:

$ ffmpeg -framerate 10 -pattern_type glob -i "*.jpg" ferry.gif

And right here’s what I get:

As you may see, I used Mountpoint to entry the prevailing picture recordsdata and to jot down the newly created animation again to S3. Whereas this can be a pretty easy demo, it does present how you should utilize your present instruments and expertise to course of objects in an S3 bucket. Provided that I’ve collected a number of million photographs over time, with the ability to course of them with out explicitly syncing them to my native file system is a giant win.

Mountpoint for Amazon S3 Info
Listed here are a few issues to remember when utilizing Mountpoint:

Pricing – There are not any new fees for using Mountpoint; you pay just for the underlying S3 operations. You may as well use Mountpoint to entry requester-pays buckets.

Efficiency – Mountpoint is ready to make the most of the elastic throughput supplied by S3, together with knowledge switch at as much as 100 Gb/second between every EC2 occasion and S3.

Credentials – Mountpoint accesses your S3 buckets utilizing the AWS credentials which might be in impact once you mount the bucket. See the CONFIGURATION doc for extra info on credentials, bucket configuration, use of requester pays, some suggestions for using S3 Object Lambda, and extra.

Operations & Semantics – Mountpoint helps primary file operations, and might learn recordsdata as much as 5 TB in measurement. It may well record and skim present recordsdata, and it could possibly create new ones. It can’t modify present recordsdata or delete directories, and it doesn’t help symbolic hyperlinks or file locking (in the event you want POSIX semantics, check out Amazon FSx for Lustre). For extra details about the supported operations and their interpretation, learn the SEMANTICS doc.

Storage Courses – You should use Mountpoint to entry S3 objects in all storage courses besides S3 Glacier Versatile Retrieval, S3 Glacier Deep Archive, S3 Clever-Tiering Archive Entry Tier, and S3 Clever-Tiering Deep Archive Entry Tier.

Open Supply – Mountpoint is open supply and has a public roadmap. Your contributions are welcome; you should definitely learn our Contributing Tips and our Code of Conduct first.

Hop On
As you may see, Mountpoint is basically cool and I’m guessing that you’re going to discover some superior methods to place it to make use of in your purposes. Test it out and let me know what you assume!

— Jeff;