You are here: Home Blog Authors ccrownhart

Chris Crownhart

May 02, 2013

Automated Amazon EBS Volume Snapshots with Boto

by Chris Crownhart — last modified May 02, 2013 09:40 AM
Filed Under:

The following write-up describes how to setup, configure and use an easy tool for taking regular snapshots of Amazon Elastic Block Storage (EBS) volumes.

Recently, we were asked by a client to setup regular EBS volume snapshots for their production EC2 instances. After a little research, Mike came across a fairly simple tool called aws-snapshot-tool: https://github.com/evannuil/aws-snapshot-tool.  This tool runs as a cron job and automatically creates daily, weekly, and monthly snapshots, and automatically rotates out the oldest snapshots for each time period.

The tool, written in Python, takes advantage of an excellent module called boto (https://github.com/boto/boto, http://docs.pythonboto.org/en/latest/).  Boto is a Python package that provides an interface to Amazon Web Services (AWS) allowing you to easily maintain and manage various aspects of your AWS environment with very little coding.

Install Notes

The aws-snapshot-tool github page has simple instructions to get started, but here are a couple points of interest:

install and setup boto

  1. pip install boto
  2. Setup a .boto config file to store your AWS access key (See: https://code.google.com/p/boto/wiki/BotoConfig)

Setup your Identity and Access Management (IAM) access

When you access AWS through their API, you are utilizing the web service functions to which your account has been given access.  The aws-snapshot-tool includes a sample IAM configuration, that shows which actions need to be setup:

"Action": [
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:DeleteSnapshot",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeSnapshots",
        "ec2:DescribeTags",
        "ec2:DescribeVolumeAttribute",
        "ec2:DescribeVolumeStatus",
        "ec2:DescribeVolumes"
      ],

Configuration (including rotation)

Configuration is very easy.  The script uses a config.py file that looks like:

config = {
    'ec2_region_name': 'us-east-1',
    'ec2_region_endpoint': 'ec2.us-east-1.amazonaws.com',
    'tag_name': 'tag:MakeSnapshot',
    'tag_value':    'True',
    'keep_day': 5,
    'keep_week': 5,
    'keep_month': 11,
    'log_file': '/tmp/makesnapshots.log',
}

There are three important pieces being defined here:

  • The region where your EBS volumes are defined
  • The tag name put on each volume in order to enable snapshots
  • The definition for how long to keep snapshots: 5 days, 5 weeks, and 13 months

Tag Your Volumes

Well, it couldn't be any easier.  Simply add a Tag of 'MakeSnapshot', with a value of 'True', (as defined in the above config file) to any volume you want picked up by this script.  The script loops through all volumes in the entire defined ec2_region_name field, looks for that tag, and handles the snapshot and rotation of snapshots automatically.

AWS Volume Snapshot Tag

Cron Setup & Output

We chose to run our script on another EC2 instance, but this could run anywhere.   We setup the following cron jobs to run:

$ crontab -l

# mon-fri 30 3 * * 1-5 /home/zope/aws-snapshot-tool/makesnapshots.py day

# every sat 30 3 * * 6 /home/zope/aws-snapshot-tool/makesnapshots.py week

# first sun 30 3 1-7 * 0 /home/zope/aws-snapshot-tool/makesnapshots.py month

Each snapshot is created with a description that begins like 'day_snapshot...', 'week_snapshot...', or 'month_snapshot...', like in the following screenshots:

AWS Snapshots - Day

AWS Snapshots - Week

Gotchas?

At the moment, I can only think of 3:

  • The config.py script points to a region. If we decide to start using other regions for our servers, we will need to accomodate that.
  • The cost of storage. Is cheap. However, once this has run for a year, we will have 23 snapshots per volume saved in our account.
  • These are real time snapshots. It is each server administrator's responsibility to insure the volume being backed up is in the proper state.

Apr 10, 2013

A Landscape of Python Web Development

by Chris Crownhart — last modified Apr 10, 2013 10:55 PM

Last week we had our kickoff meeting of the new Python Web Development meetup in Fort Collins, Colorado. We had 8 folks at the meeting and had great discussions about a variety of Python products and ideas for future meetings.

In preparation for our kickoff meeting of the new Python Web Development meetup, I started putting together what I called "A Landscape of Python Web Development".  It started with a few of the obvious tools and products that we use at Core Software Group, like Zope/Plone, Django, and boto and suds.

From there, it started becoming obvious that I needed to break down the ever-growing list into mutliple categories.

My list of products played out like this:

The Categories

  • Frameworks
  • CMS
  • Deployment/Hosting
  • Page Templates/Forms
  • Databases
  • Testing
  • Tools/Other
  • Static Site Generators

Frameworks

This list of frameworks includes some old timers, and some newer, 'lightweight' frameworks.  The last one listed, itty, was mentioned in one of Raymond Hettinger's pycon videos, and seemed an interesting, super small framework to check out.

  • Zope
  • Pyramid
  • Django
  • Flask
  • Bottle
  • CherryPy
  • web2py
  • itty

CMS

I'm sure I missed some, but these seemed to be the key players that we have come across, and the interwebs seemed to point to.

  • Plone
  • Kotti
  • Django CMS
  • Mezzanine

Deployment/Hosting

The first three are cloud platforms, and the last three are samples of python tools related to deployment.

  • Heroku
  • Google App Engine
  • Elastic Beanstalk
  • boto for EC2
  • fabric
  • buildout

Page Templates/Forms

This list was a late addition, but is now pretty obvious.  You can't do web development without page templates and forms.

  • Zope Page Templates (ZPT)
  • Django Templates
  • jinja2
  • chameleon
  • deform
  • z3c.form
  • WTForms

Databases

While the following are not directly related to web development, they certainly come into play when building applications that require a backend database.

  • sqlalchemy
  • psycopg2
  • pymongo
  • MySQL-python

Testing

Ok, I have to admit that I'm incredibly weak when it comes to writing tests.  But I needed a list of some of the tools out there.

  • unittest
  • doctest
  • coverage
  • selenium
  • funkload

Tools/Other

Wow. This list could go on and on and on.  I grabbed a few tools that we use, plus a few others I came across over the last month.  Hopefully, we can provide some presentations about some of these in future meetups.

  • suds
  • requests
  • twisted
  • mincss
  • xml.dom, lxml, etree
  • virtualenv
  • simplejson
  • PIL/Pillow

Static Site Generators

This list of static site generators almost has nothing to do with the above items, but I couldn't resist putting the list together.  I'm fascinated by the concept of someone willing to write blog entries as text files, which in turn get deployed as static HTML to some server on the web.  What I really want to know is: "How many folks are actually using these kind of tools for production sites?"

Hovercraft/impress.js

Finally, while building my presentation, I came across a Python package called Hovercraft!, written by Lennart Regebro, for generating impress.js presentations from a reStructuredText file.  Awesome.  And so easy to use.  You can download my kickoff.rst file and the generated output for my presentation.

Our discussion at the Python Web Development meetup was interesting, and it turns out that most of categories really start to look like the definition of a "development stack" which I suppose could be used to define a set of tools that someone might use to get started doing Python based web development.

I hope you find some value in this list.  If I missed some completely obvious products, please add a comment and let me know.

Apr 18, 2012

Plone Demonstrations on World Plone Day

by Chris Crownhart — last modified Apr 18, 2012 10:55 AM

On Wednesday, April 25, as part of a worldwide effort to promote and support Plone, we will be holding a World Plone Day event in Fort Collins, Colorado. Stop by for our brown bag lunch series featuring demos of the open source CMS.

Colorado World Plone Day LogoCore Software Group will be presenting demonstrations of Plone, an open source, python-based CMS, showing the latest features from the 4.1 release.

Demonstrations will show:

  • how content managers use the various tools in Plone
  • how administrators manage a Plone site
  • how developers can extend the CMS through building custom themes or custom products

For those interested, we can also:

  • show a variety of sites that we have built showing different solutions customers have requested
  • show how to quickly install and configure a Plone site

The presentations will last from approximately 12:00pm - 2:00pm.

For more information and to RSVP, visit the event page.

Sep 01, 2011

Fix for 404 Not Found Error for @@manage-portlets

by Chris Crownhart — last modified Sep 01, 2011 12:35 PM

In the midst of upgrading a Plone site from 2.5.x to 3.3.x (and eventually to 4.x), we discovered certain custom content types were returning a 404 error when accessing the @@manage-portlets page.

This one took a while to track down, so I thought I would write down our solution.  It turned out to be easy.  We simply added an interface to our custom content type's class as follows:

First import the necesary modules:

from zope.interface import implements
from plone.portlets.interfaces import ILocalPortletAssignable

 

Then we added an implements line to our class:

class SiteFolderCT(BaseFolder):
    """An Archetype for an SiteFolder application"""
    implements(ILocalPortletAssignable)
    <the rest of the class code...>

 

I hope this helps out others.

Apr 19, 2010

Colorado World Plone Day - 2010

by Chris Crownhart — last modified Apr 19, 2010 01:20 PM

Wednesday, April 28, Noon: Colorado World Plone Day will be held at the National Renewable Energy Laboratory's Visitors Center in Golden, Colorado

This year's Colorado World Plone Day event will be held at the National Renewable Energy Laboratory's (NREL) Visitors Center in beautiful Golden, Colorado, and will be presented as part of NREL's Power Lunch series.

Colorado World Plone Day LogoPresenters will talk about various features of Plone, demo websites using Plone, and field any questions people may have.

So far, this year's presenters are:

  • Kurt Bendl - Contractor at NREL
  • Dan Timmons - CU, Boulder
  • Chris Crownhart - Core Software Group

The presentations will last from approximately 12:00pm - 12:45pm, followed by a 15 minute question/answer period.

For those of you who need to get back to work, we will break shortly.  For those of you who wish to stick around to see some developer oriented presentations, and/or to discuss Plone's features in more depth, we will have the room for a couple more hours.

Direction to the NREL Visitors Center can be found at: http://www.nrel.gov/visitors_center/contact_visit.html

The Power Lunch series is a brown bag style presentation series, so please bring a sandwich and come hang out with us for an hour.

For more information:

 
 
Copyright © 2003-2017 Core Software Group | 303/809-1001 | Fort Collins, Colorado | All rights reserved.