Getting Started with AWS and Python

Working with Amazon S3

Amazon S3 is one of the most foundational and basic of all the Amazon services. You can store, retrieve, delete, and set permissions and metadata on objects—Amazon S3 parlance for what you would think of as files. You can version your objects as well as prevent deletions using multi-factor authentication. This article focuses on the basics of storing and retrieving files.

Let's jump right in with some code using your favorite Python library. Start by creating a bucket in which to store files (the name of the bucket must be globally unique, or else you'll get an error upon creation):

import boto
s3 = boto.connect_s3()
bucket = s3.create_bucket('media.yourdomain.com')  # bucket names must be unique
key = bucket.new_key('examples/first_file.csv')
key.set_contents_from_filename('/home/patrick/first_file.csv')
key.set_acl('public-read')

After creating the bucket, create a key in which to store the data. This key name can be anything you decide, however, I like to use a forward slash (/) and organize my keys similar to how I create folders in a file system. Calling the new_keymethod returns a new key object, but it hasn't done anything on Amazon S3, yet. That's where the next call,set_contents_from_filename, comes in. This call opens a file handle to the specified local file, and buffers read and write the bytes from the file into the key object on Amazon S3. Finally, you set the access control list (ACL) on the file so that you can access it publicly without permissions.

Now, say that you want to download that file to another machine. You could simply use the following command:

import boto
s3 = boto.connect_s3()
key = s3.get_bucket('media.yourdomain.com').get_key('examples/first_file.csv')
key.get_contents_to_filename('/myfile.csv')

As you can see, this code is just the reverse of what you did to upload the file in the first place. But what if you wanted to move the file to a new bucket and prefix name? Simply copy and delete:

import boto
s3 = boto.connect_s3()
key = s3.get_bucket('media.yourdomain.com').get_key('examples/first_file.csv')
new_key = key.copy('media2.yourdomain.com', 'sample/file.csv')
if new_key.exists:
    key.delete()

Doing so copies the file to the new bucket called media2.yourdomain.com and gives it the key name sample/file.csv. Then, before deleting the origin, you verify that the new key exists, which tells you that the copy finished successfully.