You are here: Home Blog Automated Amazon EBS Volume Snapshots with Boto

Automated Amazon EBS Volume Snapshots with Boto

by Chris Crownhart — last modified May 02, 2013 09:40 AM
Filed Under:

The following write-up describes how to setup, configure and use an easy tool for taking regular snapshots of Amazon Elastic Block Storage (EBS) volumes.

Recently, we were asked by a client to setup regular EBS volume snapshots for their production EC2 instances. After a little research, Mike came across a fairly simple tool called aws-snapshot-tool: https://github.com/evannuil/aws-snapshot-tool.  This tool runs as a cron job and automatically creates daily, weekly, and monthly snapshots, and automatically rotates out the oldest snapshots for each time period.

The tool, written in Python, takes advantage of an excellent module called boto (https://github.com/boto/boto, http://docs.pythonboto.org/en/latest/).  Boto is a Python package that provides an interface to Amazon Web Services (AWS) allowing you to easily maintain and manage various aspects of your AWS environment with very little coding.

Install Notes

The aws-snapshot-tool github page has simple instructions to get started, but here are a couple points of interest:

install and setup boto

  1. pip install boto
  2. Setup a .boto config file to store your AWS access key (See: https://code.google.com/p/boto/wiki/BotoConfig)

Setup your Identity and Access Management (IAM) access

When you access AWS through their API, you are utilizing the web service functions to which your account has been given access.  The aws-snapshot-tool includes a sample IAM configuration, that shows which actions need to be setup:

"Action": [
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:DeleteSnapshot",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeSnapshots",
        "ec2:DescribeTags",
        "ec2:DescribeVolumeAttribute",
        "ec2:DescribeVolumeStatus",
        "ec2:DescribeVolumes"
      ],

Configuration (including rotation)

Configuration is very easy.  The script uses a config.py file that looks like:

config = {
    'ec2_region_name': 'us-east-1',
    'ec2_region_endpoint': 'ec2.us-east-1.amazonaws.com',
    'tag_name': 'tag:MakeSnapshot',
    'tag_value':    'True',
    'keep_day': 5,
    'keep_week': 5,
    'keep_month': 11,
    'log_file': '/tmp/makesnapshots.log',
}

There are three important pieces being defined here:

  • The region where your EBS volumes are defined
  • The tag name put on each volume in order to enable snapshots
  • The definition for how long to keep snapshots: 5 days, 5 weeks, and 13 months

Tag Your Volumes

Well, it couldn't be any easier.  Simply add a Tag of 'MakeSnapshot', with a value of 'True', (as defined in the above config file) to any volume you want picked up by this script.  The script loops through all volumes in the entire defined ec2_region_name field, looks for that tag, and handles the snapshot and rotation of snapshots automatically.

AWS Volume Snapshot Tag

Cron Setup & Output

We chose to run our script on another EC2 instance, but this could run anywhere.   We setup the following cron jobs to run:

$ crontab -l

# mon-fri 30 3 * * 1-5 /home/zope/aws-snapshot-tool/makesnapshots.py day

# every sat 30 3 * * 6 /home/zope/aws-snapshot-tool/makesnapshots.py week

# first sun 30 3 1-7 * 0 /home/zope/aws-snapshot-tool/makesnapshots.py month

Each snapshot is created with a description that begins like 'day_snapshot...', 'week_snapshot...', or 'month_snapshot...', like in the following screenshots:

AWS Snapshots - Day

AWS Snapshots - Week

Gotchas?

At the moment, I can only think of 3:

  • The config.py script points to a region. If we decide to start using other regions for our servers, we will need to accomodate that.
  • The cost of storage. Is cheap. However, once this has run for a year, we will have 23 snapshots per volume saved in our account.
  • These are real time snapshots. It is each server administrator's responsibility to insure the volume being backed up is in the proper state.
Document Actions

Where is the config.py file located?

Avatar Posted by Gene Bean at Sep 26, 2013 10:26 AM
Thanks for writing this how-to. One question. Where is the config.py file located? Can't find this anywhere. There is one at /usr/share/pyshared/boto/pyami/config.py but it looks like Python code and not a config file at all. Hellpppp.....!!!

See config.sample

Avatar Posted by Chris Crownhart at Sep 26, 2013 10:27 AM
Gene,

When you download the snapshot tool from https://github.com/evannuil/aws-snapshot-tool, it comes with a file called config.sample. Just rename that to config.py.

Thank you, but still...

Avatar Posted by Gene Bean at Sep 26, 2013 09:57 PM
Thank you Chris. Sorry if this is a silly question. I'm familiar with everything else but Python. where do I download aws-snapshot-tool to if I'm on an Ubuntu/Linux server?

filezilla

Avatar Posted by sangeeth at Jan 07, 2014 10:49 AM
If you have windows server on your pc, you can download the snapshot tool into windows and using Filezilla(ftp client) simply you can upload the tool in to Linux server....

OK I think I figured it out

Avatar Posted by Gene Bean at Sep 26, 2013 09:57 PM
Silly me. These are Python scripts so it doesn't matter where they're installed, as long as the cron knows where to find it. Got it thank you !!

Another question though -- what about the issue of rebooting the instance after the snapshot is made? Is it better to stop, take snapshot, then restart? Does this script handle all that?

Please explain "It is each server administrator's responsibility to insure the volume being backed up is in the proper state." -- what areas need to be addressed? Thanks for your help.

Need help

Avatar Posted by Gene Bean at Sep 27, 2013 11:14 AM
Hi it's me again :-)

I'm pretty sure I got all the config info correct. When I run it on the command line using "python makesnapshots.py day", it hangs. When I hit Ctrl-C to interrupt it, this is what I see (pasted below). The logfile for this script just says it started but doesn't get past that. What could it be?

ubuntu@ip-10-28-249-183:~/aws-snapshot-tool$ sudo python makesnapshots.py day
^CTraceback (most recent call last):
  File "makesnapshots.py", line 100, in <module>
    vols = conn.get_all_volumes(filters={config['tag_name']: config['tag_value']})
  File "/usr/lib/python2.7/dist-packages/boto/ec2/connection.py", line 1309, in get_all_volumes
    [('item', Volume)], verb='POST')
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 882, in get_list
    response = self.make_request(action, params, path, verb)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 868, in make_request
    return self._mexe(http_request)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 785, in _mexe
    time.sleep(next_sleep)
KeyboardInterrupt

Offline

Avatar Posted by Chris Crownhart at Sep 27, 2013 11:15 AM
Gene,

Let's take this offline. Email me at webrequests@coresoftwaregroup.com, and I'll see how I can help you.

Got it to work

Avatar Posted by Gene Bean at Sep 30, 2013 09:48 AM
Bingo !! Got it to work. I was tagging the EC2 Instance instead of the Volume. Also, I had tried using us-east-1a as the Region -- the subpart is not needed in the config. Thank you for providing this tutorial.

Error encountered

Avatar Posted by shong at Jun 28, 2014 02:33 PM
I encountered the error below. Anyone knows what is wrong? Many thanks!

Traceback (most recent call last):
  File "makesnapshots.py", line 130, in <module>
    vols = conn.get_all_volumes(filters={config['tag_name']: config['tag_value']})
  File "/usr/local/lib/python2.7/site-packages/boto/ec2/connection.py", line 2115, in get_all_volumes
    [('item', Volume)], verb='POST')
  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 1143, in get_list
    response = self.make_request(action, params, path, verb)
  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 1089, in make_request
    return self._mexe(http_request)
  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 922, in _mexe
    request.body, request.headers)
  File "/usr/local/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/boto/https_connection.py", line 111, in connect
    sock.connect((self.host, self.port))
  File "/usr/local/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.gaierror: [Errno -2] Name or service not known

Add comment

You can add a comment by filling out the form below. Plain text formatting.

Info
You are not logged in. You may optionally enter your username and password below. If you don't enter anything, this comment will be posted as 'Anonymous User'.
(Required)
(Required)
(Required)
Enter the word
 
 
Copyright © 2003-2017 Core Software Group | 303/809-1001 | Fort Collins, Colorado | All rights reserved.