Blog

Automated Amazon EBS Volume Snapshots with Boto

Chris Crownhart — Thu, 02 May 2013 09:40:00 -0600

Recently, we were asked by a client to setup regular EBS volume snapshots for their production EC2 instances. After a little research, Mike came across a fairly simple tool called aws-snapshot-tool: https://github.com/evannuil/aws-snapshot-tool. This tool runs as a cron job and automatically creates daily, weekly, and monthly snapshots, and automatically rotates out the oldest snapshots for each time period.

The tool, written in Python, takes advantage of an excellent module called boto (https://github.com/boto/boto, http://docs.pythonboto.org/en/latest/). Boto is a Python package that provides an interface to Amazon Web Services (AWS) allowing you to easily maintain and manage various aspects of your AWS environment with very little coding.

Install Notes

The aws-snapshot-tool github page has simple instructions to get started, but here are a couple points of interest:

install and setup boto

pip install boto
Setup a .boto config file to store your AWS access key (See: https://code.google.com/p/boto/wiki/BotoConfig)

Setup your Identity and Access Management (IAM) access

When you access AWS through their API, you are utilizing the web service functions to which your account has been given access. The aws-snapshot-tool includes a sample IAM configuration, that shows which actions need to be setup:

"Action": [
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:DeleteSnapshot",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeSnapshots",
        "ec2:DescribeTags",
        "ec2:DescribeVolumeAttribute",
        "ec2:DescribeVolumeStatus",
        "ec2:DescribeVolumes"
      ],

Configuration (including rotation)

Configuration is very easy. The script uses a config.py file that looks like:

config = {
    'ec2_region_name': 'us-east-1',
    'ec2_region_endpoint': 'ec2.us-east-1.amazonaws.com',
    'tag_name': 'tag:MakeSnapshot',
    'tag_value':    'True',
    'keep_day': 5,
    'keep_week': 5,
    'keep_month': 11,
    'log_file': '/tmp/makesnapshots.log',
}

There are three important pieces being defined here:

The region where your EBS volumes are defined
The tag name put on each volume in order to enable snapshots
The definition for how long to keep snapshots: 5 days, 5 weeks, and 13 months

Tag Your Volumes

Well, it couldn't be any easier. Simply add a Tag of 'MakeSnapshot', with a value of 'True', (as defined in the above config file) to any volume you want picked up by this script. The script loops through all volumes in the entire defined ec2_region_name field, looks for that tag, and handles the snapshot and rotation of snapshots automatically.

Cron Setup & Output

We chose to run our script on another EC2 instance, but this could run anywhere. We setup the following cron jobs to run:

$ crontab -l

# mon-fri 30 3 * * 1-5 /home/zope/aws-snapshot-tool/makesnapshots.py day

# every sat 30 3 * * 6 /home/zope/aws-snapshot-tool/makesnapshots.py week

# first sun 30 3 1-7 * 0 /home/zope/aws-snapshot-tool/makesnapshots.py month

Each snapshot is created with a description that begins like 'day_snapshot...', 'week_snapshot...', or 'month_snapshot...', like in the following screenshots:

Gotchas?

At the moment, I can only think of 3:

The config.py script points to a region. If we decide to start using other regions for our servers, we will need to accomodate that.
The cost of storage. Is cheap. However, once this has run for a year, we will have 23 snapshots per volume saved in our account.
These are real time snapshots. It is each server administrator's responsibility to insure the volume being backed up is in the proper state.

A Landscape of Python Web Development

Chris Crownhart — Wed, 10 Apr 2013 22:55:00 -0600

In preparation for our kickoff meeting of the new Python Web Development meetup, I started putting together what I called "A Landscape of Python Web Development". It started with a few of the obvious tools and products that we use at Core Software Group, like Zope/Plone, Django, and boto and suds.

From there, it started becoming obvious that I needed to break down the ever-growing list into mutliple categories.

My list of products played out like this:

Frameworks

This list of frameworks includes some old timers, and some newer, 'lightweight' frameworks. The last one listed, itty, was mentioned in one of Raymond Hettinger's pycon videos, and seemed an interesting, super small framework to check out.

Zope
Pyramid
Django
Flask
Bottle
CherryPy
web2py
itty

CMS

I'm sure I missed some, but these seemed to be the key players that we have come across, and the interwebs seemed to point to.

Plone
Kotti
Django CMS
Mezzanine

Deployment/Hosting

The first three are cloud platforms, and the last three are samples of python tools related to deployment.

Heroku
Google App Engine
Elastic Beanstalk

boto for EC2

fabric
buildout

Page Templates/Forms

This list was a late addition, but is now pretty obvious. You can't do web development without page templates and forms.

Zope Page Templates (ZPT)
Django Templates
jinja2
chameleon
deform
z3c.form
WTForms

Databases

While the following are not directly related to web development, they certainly come into play when building applications that require a backend database.

sqlalchemy
psycopg2
pymongo
MySQL-python

Testing

Ok, I have to admit that I'm incredibly weak when it comes to writing tests. But I needed a list of some of the tools out there.

unittest
doctest
coverage
selenium
funkload

Tools/Other

Wow. This list could go on and on and on. I grabbed a few tools that we use, plus a few others I came across over the last month. Hopefully, we can provide some presentations about some of these in future meetups.

suds
requests
twisted
mincss
xml.dom, lxml, etree
virtualenv
simplejson
PIL/Pillow

Static Site Generators

This list of static site generators almost has nothing to do with the above items, but I couldn't resist putting the list together. I'm fascinated by the concept of someone willing to write blog entries as text files, which in turn get deployed as static HTML to some server on the web. What I really want to know is: "How many folks are actually using these kind of tools for production sites?"

Klein - https://github.com/twisted/klein
Pelican - http://blog.notmyidea.org/pelican-a-simple-static-blog-generator-in-python.html
Hyde - http://ringce.com/hyde
Cactus - https://github.com/koenbok/Cactus
Nikola - https://github.com/ralsina/nikola

Hovercraft/impress.js

Finally, while building my presentation, I came across a Python package called Hovercraft!, written by Lennart Regebro, for generating impress.js presentations from a reStructuredText file. Awesome. And so easy to use. You can download my kickoff.rst file and the generated output for my presentation.

Our discussion at the Python Web Development meetup was interesting, and it turns out that most of categories really start to look like the definition of a "development stack" which I suppose could be used to define a set of tools that someone might use to get started doing Python based web development.

I hope you find some value in this list. If I missed some completely obvious products, please add a comment and let me know.

How to enable User Folders in Plone

Mike Cullerton — Tue, 09 Apr 2013 15:22:57 -0600

I needed to enable the User Folders feature of Plone from within my product. My first thought was to use a Generic Setup profile, but the solution was setuphandlers.py.

I inspected the Enable User Folders button on the Security settings page of Site Setup, and saw the attribute was named enable_user_folders. I tried to find a generic setup profile I could edit.

The closest I got was adding enable_user_folders to properties.xml. This turned the attribute on in the ZMI, but not in Site Setup and User Folders were not enabled.

Then I found a reference to

security.enable_user_folders

in a setuphandlers.py file from the Rhaptos git repo.

So, I added these lines to setupVarious in my setuphandlers.py.

from plone.app.controlpanel.security import ISecuritySchema
site = context.getSite()
security = ISecuritySchema(site)
security.enable_user_folders = True

Now, when I install my product, User Folders are enabled automatically.

Note that you need to make sure setupVarious is turned on in configure.zcml.

<genericsetup:importStep
    name="your.package"
    title="your.package special import handlers"
    description=""
    handler="your.package.setuphandlers.setupVarious"
    />

Hope this helps someone.

How to open SSH links in iTerm 2

Mike Cullerton — Mon, 08 Apr 2013 16:45:00 -0600

I know some of you have this problem. I've seen a few references to it on the net, but no solutions. iTerm 2 won't open ssh links.

I have a pyramid application that uses boto to query ec2 instances and then posts information about the instances--including an ssh link, to a web page on my laptop.

The ssh links open fine in iTerm. I couldn't get them to open in iTerm 2 though. Instead, they just opened new tabs.

I really like using iTerm 2, so to 'solve' the problem, I added the actual ssh shell command for each of the instances as text on the page.

To open an ssh connection to an instance, I simply copy the ssh command, click the link, paste the ssh command into the new terminal tab, and hit return.

Hey, it works.

Every once in a while, when it really bothers me, I try to figure out a real solution.

Today was one of those days.

The solution turned out to be pretty simple.

In the iTerm 2 Preferences, under the Profiles tab, General section, change the Command to Login shell.

Now, when I click on one of the links, it opens correctly in iTerm 2.

I hope it works for you too.

*Note that you must first have iTerm 2 set up to accept ssh connections. Preferences->Profiles->General->URL Schemes

Plone Hello World Tutorial

Mike Cullerton — Fri, 14 Sep 2012 01:25:00 -0600

About a month ago, there was a bit of a ruckus on twitter about the state of Plone and documentation and what not.

There was a specific request for a Hello World type tutorial introducing Plone development. After talking with Mikko and others on IRC, I put some ideas together and released a first attempt a few days later.

Over the last few weeks, I've put time into adding more sections, reorganizing the layout, and general cleanup. Tonight, I pushed those changes to the developer manual at http://collective-docs.readthedocs.org/en/latest/getstarted/helloworld/index.html

At this point it's starting to take shape. There are holes in it, and things I'd like to clean up, but it's a good start.

I'd like to get some feedback. What do folks think? What is missing? Are there any errors? Really, anything helpful is appreciated.

You can comment here, or reach me @cullerton on twitter.

Thanks,

Mike

Plone Demonstrations on World Plone Day

Chris Crownhart — Wed, 18 Apr 2012 10:55:00 -0600

Core Software Group will be presenting demonstrations of Plone, an open source, python-based CMS, showing the latest features from the 4.1 release.

Demonstrations will show:

how content managers use the various tools in Plone
how administrators manage a Plone site
how developers can extend the CMS through building custom themes or custom products

For those interested, we can also:

show a variety of sites that we have built showing different solutions customers have requested
show how to quickly install and configure a Plone site

The presentations will last from approximately 12:00pm - 2:00pm.

For more information and to RSVP, visit the event page.

Fix for 404 Not Found Error for @@manage-portlets

Chris Crownhart — Thu, 01 Sep 2011 12:35:00 -0600

This one took a while to track down, so I thought I would write down our solution. It turned out to be easy. We simply added an interface to our custom content type's class as follows:

First import the necesary modules:

from zope.interface import implements
from plone.portlets.interfaces import ILocalPortletAssignable

Then we added an implements line to our class:

class SiteFolderCT(BaseFolder):
    """An Archetype for an SiteFolder application"""
    implements(ILocalPortletAssignable)
    <the rest of the class code...>

I hope this helps out others.

Adding a Zope 3 permission to Plone

Mike Cullerton — Mon, 13 Sep 2010 16:15:00 -0600

Quick Overview

We recently migrated a Plone 2.1 site to Plone 3.3. While moving a skins file to a browser view, we needed a new permission to protect the browser view. Browser views need a named Zope 3 style permission, so the old Zope 2 style permission wouldn't work. It ended up taking 3 steps; declare the permission in zcml, assign the permission to a role in rolemap.xml, and use the permission to protect your view.

Step-by-step

Step 1: Declare the permission in ZCML

Most people declare the permissions in permissions.zcml and include that file in configure.zcml. But you can declare the permission in configure.zcml and just include permissions.zcml. Note that id is the zope 3 style identifier, and title is the zope 2 style identifier. The title shows up on the security tab in the ZMI.

src/your.product/your/product/browser/permissions.zcml:

    <permission 
       id="myproduct.myNewPermission"
       title="MyProduct: My New Permission"
       />

Step 2: Assign the permission to a role

Assign the permission to a role in rolemap.xml. Here, I assign my new permission to the manager role.

src/your.product/your/product/profiles/default/rolemap.xml:

    <permission name="MyProduct: My New Permission">
       <role name="Manager" />
    </permission>

Step 3: Protect your view with the new permission

In configure.zcml protect your view with the new permission. Here, I'm protecting the class MyClass in the file common.py. We chose to put the permissions from Step 1 into permissions.zcml, so we include it here. Note that the permission must be the same as the id in Step 1.

src/your.product/your/product/browser/configure.zcml:

<include file="permissions.zcml" />
<browser:page
    for="Products.CMFPlone.interfaces.IPloneSiteRoot"
    name="myName"
    class=".common.MyClass"
    permission="myproduct.myNewPermission"
    />

As usual, the community developer manual at http://plonemanual.twinapex.fi/ was very helpful.

Injecting Plone variables into javascript

Mike Cullerton — Tue, 22 Jun 2010 09:00:00 -0600

Step 1: Create a class to output the javascript

I created a file in src/my.product/my/product/browser called CommonUtils.py and added a class GlobalJS with a __call__ method that returns a string containing the javascript. __call__ sets the content-type, creates the string containing the variable, and returns the string.

class GlobalJS(BrowserView):
 
    def __call__(self,REQUEST,RESPONSE):
        RESPONSE.setHeader('Content-Type', 'application/javascript')
        js_string = "var portal_url = '%s';" % (self.context.portal_url())
        return js_string

Step 2: Create a browser view pointing to the class

In src/my.product/my/product/browser/configure.zcml, I added a browser view pointing to GlobalJS, gave it a name, and added a permission. Note that the name of the file containing GlobalJS is CommonUtils.py

<browser:view 
     for="*"
     name="global_js.js" 
     class=".CommonUtils.GlobalJS"
     permission="zope2.View"
     />

Step 3: Register the view in javascript registry

Here, I register the browser view in src/my.product/my/product/profiles/default/jsregistry.xml. I needed access to portal_url before jquery was loaded, so I added the insert-before property. Note that the id is the same as the name registered in the browser view.

<javascript 
    id="global_js.js"
    cacheable="True" compression="safe" cookable="True"
    enabled="True" expression=""  inline="True" insert-before="jquery.js"/>

Now, global_js.js is loaded on every page and I have access to portal_url from within javascript.

Hope this helps someone.

Colorado World Plone Day - 2010

Chris Crownhart — Mon, 19 Apr 2010 13:20:00 -0600

This year's Colorado World Plone Day event will be held at the National Renewable Energy Laboratory's (NREL) Visitors Center in beautiful Golden, Colorado, and will be presented as part of NREL's Power Lunch series.

Presenters will talk about various features of Plone, demo websites using Plone, and field any questions people may have.

So far, this year's presenters are:

Kurt Bendl - Contractor at NREL
Dan Timmons - CU, Boulder
Chris Crownhart - Core Software Group

The presentations will last from approximately 12:00pm - 12:45pm, followed by a 15 minute question/answer period.

For those of you who need to get back to work, we will break shortly. For those of you who wish to stick around to see some developer oriented presentations, and/or to discuss Plone's features in more depth, we will have the room for a couple more hours.

Direction to the NREL Visitors Center can be found at: http://www.nrel.gov/visitors_center/contact_visit.html

The Power Lunch series is a brown bag style presentation series, so please bring a sandwich and come hang out with us for an hour.

For more information:

See: http://www.coresoftwaregroup.com/colorado-world-plone-day-2010
Contact Chris Crownhart at 303/809-1001

Overriding the Title tag in Plone 3

Mike Cullerton — Tue, 26 Jan 2010 12:20:00 -0700

Quick Overview

You need to override the plone.htmlhead.title viewlet and create a class to render your title. The default version of the viewlet is defined in [eggs]/plone/app/layout/configure.zcml, and the class is defined in common.py. You'll need a few more things from common.py too. Override the viewlet in your own [theme]/browser/configure.zcml file, and add the location of your new class.

Getting Started

I needed to override a viewlet, but didn't know which one. A quick grep found a reference to the plone.htmlhead viewlet manager, and then one to the plone.htmlhead.title viewlet. Viewlets are found in the plone/app/layout/viewlets/configure.zcml file within the eggs area of your buildout. Here is the specific code in configure.zcml.

...
    <browser:viewlet
       name="plone.htmlhead.title"
       manager=".interfaces.IHtmlHead"
       class=".common.TitleViewlet"
       permission="zope2.View"
       />
...

In this case, the plone.htmlhead.title viewlet references a class called .common.TitleViewlet. This means a file--in the same directory as configure.zcml, named common.py contains a class named TitleViewlet. Here is the TitleViewlet code from common.py.

class TitleViewlet(ViewletBase):
 
 ...

 def index(self):
     portal_title = safe_unicode(self.portal_title())
     page_title = safe_unicode(self.page_title())
     if page_title == portal_title:
         return u"<title>%s</title>" % (escape(portal_title))
     else:
         return u"<title>%s &mdash; %s</title>" % (
                  escape(safe_unicode(page_title)),
                  escape(safe_unicode(portal_title)))

In the index method at the bottom, Plone sets the value of the <title> tag.

If the page title and the portal title are the same, the portal title is used. If not, both are used, with page title on the left and portal title on the right.

We needed to switch it, so that portal title was on the left, and page title was on the right.

What we did

To override the title viewlet, I created an entry in my theme's browser/configure.zcml file.

<!-- Our custom title viewlet --> 
 <browser:viewlet
 name="plone.htmlhead.title"
 manager="plone.app.layout.viewlets.interfaces.IHtmlHead"
 class=".myTitleViewlet.TitleViewlet" 
 layer=".interfaces.IThemeSpecific"
 permission="zope2.View" 
 />

Notice that there is a new layer statement, and that the manager line includes the complete path to IHtmlHead. The class statement now points to our new class containing code to override the title.

Here is the code from myTitleViewlet.py.

from zope.interface import implements
from zope.component import getMultiAdapter
from zope.viewlet.interfaces import IViewlet
from zope.deprecation.deprecation import deprecate

from Products.CMFPlone.utils import safe_unicode
from Products.Five.browser import BrowserView
from cgi import escape

from plone.app.layout.viewlets.common import ViewletBase

class TitleViewlet(ViewletBase):

 ...

 def index(self):
 portal_title = safe_unicode(self.portal_title())
 page_title = safe_unicode(self.page_title())
 if page_title == portal_title:
 return u"<title>%s</title>" % (escape(portal_title))
 else:
 return u"<title>%s | %s</title>" % (
 escape(safe_unicode(portal_title)),
 escape(safe_unicode(page_title)))

I needed some of the imports from the top of common.py, and added one of my own for ViewletBase. The only code that really changed is the else clause, where the order of portal_title and page_title are switched, and the em dash is replaced by a pipe (|).

A great reference for overriding the <title> tag is HTML Head Title page which is part of the Plone Theme Reference.

Installing plone 3 using paster

Mike Cullerton — Thu, 10 Sep 2009 17:30:00 -0600

We build most of our sites using paster (versus using the Unified Installer). Here are some simple notes we had for doing that (more information on using paster with plone can be found in the Plone documentation):

This how-to assumes you have ZopeSkel installed.

Create a directory to house the plone installation.

mkdir /path/to/plone/install/directory

Use paster to install buildout files.

paster create -t plone3_buildout /path/to/plone/install/directory

Change your working directory to the new zope install.

cd /path/to/zope/install/directory

Bootstrap the system.

python bootstrap.py

Run buildout.

./bin/buildout

That's it. Now, just start zope/plone.

./bin/instance start

Or run in it the foreground (and in debug mode).

./bin/instance fg

Migrating portal_status_message to addPortalMessage for Plone 3

Chris Crownhart — Fri, 28 Aug 2009 09:30:00 -0600

This is a pretty simple change, but I could not find much documentation on it around the web. We had quite a few controller python scripts that had code in them setting a portal_status_message. The code looked like:

return state.set(portal_status_message='Your comment has been sent to the web support team.')

This old code is completely ignored now, so you have to convert these to the new format.

In Plone 3, portal_status_message is now handled using a cookie, and there are special classes and methods for adding your messages to the cookie.

Now, to set a status message, do the following:

from Products.CMFPlone import PloneMessageFactory as _
context.plone_utils.addPortalMessage(_(u'Your comment has been sent to the web support team.'))

Additionally, you can pass flags to the method to indicate:

'info'
'warning'
'error'

like:

context.plone_utils.addPortalMessage(_(u'Your comment has been sent to the web support team.'), 'info')

For more information about the classes, interfaces, and methods, see this PLIP: http://plone.org/products/plone/roadmap/111

Simple way to add lightbox/thickbox support to Plone 3 site

Chris Crownhart — Wed, 26 Aug 2009 12:15:00 -0600

On the old Core Software Group site, I downloaded thickbox and manually added the css and js files to the portal_css and portal_js registries. It worked, but was kind of kludgy.

Thanks to Steve McMahon's Products.pipbox, it's clean and easy, and gives you a massive amount of flexibility and control. Here's all I did:

Added Products.pipbox to my eggs section of my buildout.cfg, then ran bin/buildout
Installed pipbox via quickinstaller
pipbox adds a property sheet to portal_properties, which for now, you have to edit via the ZMI
I added the following to /portal_properties/pipbox_properties -> selector specs:

{type:'overlay',subtype:'image',selector:'img.image-right,img.image-left,img.image-inline', urlmatch:'/image_.+$',urlreplace:''}

That's it. Now our portfolio page uses the lightbox popups to display screenshots of Plone sites we have built.

Next up:

Figure out how to control the position of the popup box.
Figure out how to take advantage of AJAX form support in pipbox.
Figure out how to implement AJAX tabs using pipbox.

Welcome to our new blog

Chris Crownhart — Sat, 15 Aug 2009 12:45:00 -0600

We at Core Software Group have been working with Zope for almost 10 years now, and with Plone for nearly 7 years! Man, that is awesome! I have always believed that Plone is the killer app for Zope, and have thoroughly enjoyed providing Plone consulting services.

During our journey down the Plone road, we have learned a ton, and it is my hope that we can share what we have learned in the past, and what we will learn in the future, on this blog, so other folks can benefit.

We have worked on Zope-only applications, built CMF-only applications, supported a variety of Plone versions, and most recently have been migrating old Plone sites to the newer Plone 3 platform.

There are an enormous amount of resources on the web for helping developers with Plone, but it always seems like the one thing you are working on does not have much documentation. Additionally, there are just some things that require the "secret decoder ring"!

When we come across oddities that require you to pat your head and rub your belly at the same time to get Plone to work just right, we will tag them with "secret decoder ring".

I hope that we can provide useful information to other developers. If you like what we've done, or have questions about what we can do, please feel free to contact us.

Blog

Automated Amazon EBS Volume Snapshots with Boto

Install Notes

Configuration (including rotation)

Tag Your Volumes

Cron Setup & Output

Gotchas?

A Landscape of Python Web Development

The Categories

Frameworks

CMS

Deployment/Hosting

Page Templates/Forms

Databases

Testing

Tools/Other

Static Site Generators

Hovercraft/impress.js

How to enable User Folders in Plone

How to open SSH links in iTerm 2

Plone Hello World Tutorial

Plone Demonstrations on World Plone Day

Fix for 404 Not Found Error for @@manage-portlets

Adding a Zope 3 permission to Plone

Quick Overview

Step-by-step

Step 1: Declare the permission in ZCML

Step 2: Assign the permission to a role

Step 3: Protect your view with the new permission

Injecting Plone variables into javascript

Step 1: Create a class to output the javascript

Step 2: Create a browser view pointing to the class

Step 3: Register the view in javascript registry

Colorado World Plone Day - 2010

Overriding the Title tag in Plone 3

Quick Overview

Getting Started

What we did

Installing plone 3 using paster

Migrating portal_status_message to addPortalMessage for Plone 3

Simple way to add lightbox/thickbox support to Plone 3 site

Welcome to our new blog