How to Sync Files from Linux to Amazon S3
AWS S3 is Amazon’s cloud storage service, permitting you to retailer particular person information as objects in a bucket. You can add information from the command line in your Linux server, and even sync total directories to S3.
If you simply need to share information between EC2 cases, you should utilize an EFS volume and mount it immediately to a number of servers, reducing out the “cloud” altogether. But you shouldn’t use it for every thing, as a result of it’s much pricier than S3, even with Infrequent Access turned on.
Limit S3 Access to an IAM User
Your server in all probability doesn’t want full root entry to your AWS account, so earlier than you do any sort of file syncing, you must make a brand new IAM consumer to your server to use. With an IAM consumer, you may restrict your server to solely managing your S3 buckets.
From the IAM Management Console, make a brand new consumer, and allow “Programmatic Access.”
You’ll be requested to select permissions for this consumer. Make a brand new group, and assign it the “AmazonS3FullAccess” permission.
After that, you’ll be given an entry key and secret key. Make a word of those; you’ll want them to authenticate your server.
You may manually assign extra detailed S3 permissions, comparable to permission to use a particular bucket or solely to add information, however limiting entry to simply S3 ought to be tremendous most often.
File Syncing With s3cmd
s3cmd is a utility designed to make working with S3 from the command line simpler. It’s not part of the AWS CLI, so that you’ll have to manually set up it from your distro’s bundle supervisor. For Debian-based methods like Ubuntu, that may be:
sudo apt-get set up s3cmd
s3cmd is put in, you’ll want to hyperlink it to the IAM consumer you created to handle S3. Run the configuration with:
You’ll be requested for the entry key and secret key that the IAM Management Console gave you. Paste these in right here. There’s a number of extra choices, comparable to altering the endpoints for S3 or enabling encryption, however you may go away all of them default and simply choose “Y” on the finish to save the configuration.
To add a file, use:
s3cmd put file s3://bucket
Replacing “bucket” together with your bucket title. To retrieve these information, run:
s3cmd get s3://bucket/remotefile localfile
And, if you would like to sync over an entire listing, run:
s3cmd sync listing s3://bucket/
This will copy the whole listing right into a folder in S3. The subsequent time you run it, it would solely copy the information which have modified because it was final ran. It gained’t delete any information except you run it with the
s3cmd sync gained’t run robotically, so in case you’d like to maintain this listing recurrently up to date, you’ll want to run this command recurrently. You can automate this with
cron; Open your crontab with
crontab -e, and add this command to finish:
0 0 * * * s3cmd sync listing s3://bucket >/dev/null 2>&1
This will sync “directory” to “bucket” as soon as a day. By the way in which, if
crontab -e acquired you caught in
vim, you may change the default textual content editor with
export VISUAL=nano;, or whichever you like.
s3cmd has a variety of subcommands; you may copy between buckets with
cp, transfer information with
mv, and even create and take away buckets from the command line with
rb, respectively. Use
s3cmd -h for a full checklist.
Another Option: AWS CLI
s3cmd, there are a number of different command line choices for syncing information to S3. AWS gives their very own instruments with the AWS CLI. You’ll want Python 3+, and might set up the CLI from
pip3 set up awscli --upgrade --user
This will set up the
aws command, which you should utilize to work together with AWS companies. You’ll want to configure it in the identical method as s3cmd, which you are able to do with:
You’ll be requested to enter the entry key and secret key to your IAM consumer.
The syntax for AWS CLI is comparable to
s3cmd. To add a file, use:
aws s3 cp file s3://bucket
To sync an entire folder, use:
aws s3 sync folder s3://bucket
You can copy and even sync between buckets with the identical instructions. You can use
aws assist for a full command checklist, or learn the command reference on their web site.
Full Backups: Restic, Duplicity
If you need to do giant backups, you might have considered trying to use one other instrument relatively than a easy sync utility. When you sync to S3 with
s3cmd or the AWS CLI, any adjustments you’ve made will overwrite the present information. Because the primary fear of cloud file storage isn’t normally drive failure, however unintended deletion with out entry to revision historical past, it is a downside.
AWS helps file versioning, which solves this concern considerably, however you should still need to use a extra highly effective backup program to deal with it your self, particularly in case you’re doing full-drive backups.
Duplicity is a straightforward utility that backs up information within the type of encrypted TAR volumes. The first archive is an entire backup after which any subsequent archives are incremental, storing solely the adjustments made for the reason that final archive.
This may be very environment friendly, however restoring from a backup is much less environment friendly, because the restoration course of can have to comply with the chain of adjustments to arrive on the closing state of the info. Restic solves this concern by storing information in deduplicated encrypted blocks, and retains a snapshot of every model for restoration. This method, the present state of the information is definitely referenceable, and every revision continues to be accessible.
Both instruments will be configured to work with AWS S3, in addition to a number of different storage suppliers. Alternatively, in case you simply need to again up EBS-based EC2 cases, you should utilize incremental EBS snapshots, although it’s pricier than backing up manually to S3.
This Web site is affiliated with Amazon associates, Clickbank, JVZoo, Sovrn //Commerce, Warrior Plus etc.