You are here: Home Blog archive



Aug 14, 2013

Public Data Hacking with Python in Larimer County

by Mike Cullerton — last modified Aug 14, 2013 12:25 AM

Inspired by Dave Beazley's Public Data Hacking tutorial, I decided to investigate my local public data. It turns out, Larimer County provides API access to property and property tax records. The access is limited, but does provide some beginner experience with public data.

At our recent Python Web Dev meetup [1], we worked on Dave Beazley's Learn Python Through Public Data Hacking tutorial [2]. In the tutorial, Dave introduces python with a series of coding challenges involving public data resources like transit information and health inspections.

Many governments and municipalities now provide access to public records over the Internet, both as part of open information acts and to lower their costs of providing that information. Some even provide direct access to their data through APIs.

Working on the problems in the tutorial got me wondering about the public data available here in Larimer County, CO.

Larimer County provides access to a wide variety of data through the web [3], including property records, voter registration, and restaurant health inspections.

The county also allows direct access to property and property tax information through APIs. The APIs are implemented as SOAP web services.

The county has sample code for languages they call ColdFusion, C#, and PHP.

Hello, the 1990's called. They want their manuals back. [4].

I offer these Python examples.


The API's are somewhat limited. To use them you need to know the schedule number or parcel number for a property. They don't allow you to easily grab a chunk of data to play around with while learning.

# import suds client for SOAP layer
from suds.client import Client

# get a tax result
property_tax_url = ''
property_tax_client = Client(property_tax_url)
property_tax_result = property_tax_client.service.GetPropertyTax(scheduleNum='', parcelNum=1234567890, taxYear=2012)

# get an info result
property_info_url = ''
property_info_client = Client(property_info_url)
property_info_result = property_info_client.service.GetPropertyInfo(scheduleNum='', parcelNum='1234567890')

The property information client has additional web services available. The county has documentation, code examples, etc. [5]


It's cool the county provides access to this information online. But, it is very limited in how you can access it.

Hopefully, in the future we'll see more APIs that allow us greater flexibility in our searches.

It was a good learning experience.


[1] Python Web Dev Meetup

[2] Learn Python Through Public Data Hacking

[3] Larimer County Public Records Databases

[4] I'm joking.

[5] Larimer County Public Records Databases API

Jul 25, 2013

First impressions with Kotti

by Mike Cullerton — last modified Jul 25, 2013 03:15 PM
Filed Under:

The boss is away this week, so I'm learning about Kotti. Here's how I solved the problem of limiting a content type to the root of the site.

The Basics

My application is very simple. There are two content types; Forum and Idea. A forum is added to the root of the site, and then ideas are added to the forum.

I was already familiar with kotti's documentation and the tutorial there, so getting the basic application running was very easy.

First, I set up a virtualenv, and installed kotti along with a skeleton for my package.

$ virtualenv --no-site-packages kotti_agora-27
$ cd kotti_agora-27/
$ ./bin/pip install -r
$ wget
$ ./bin/pip install kotti_paster
$ ./bin/paster create -t kotti_addon kotti_agora
$ cd kotti_agora/
$ ../bin/python develop

The pip install step takes a little while.

From there, I was able to add code to kotti_agora, including my two content types, their views, and some kotti plumbing to wire it all up. For the most part, I stole everything from the kotti tutorial mentioned above and from the source code for kotti_blog.

Limiting Ideas to a Forum

The first problem I had to solve was limiting ideas to a forum. Kotti has an addable_to attribute when creating content types. I set the addable_to attribute of Idea to Forum.

class Idea(Content):
    id = Column(Integer(), ForeignKey(''), primary_key=True)
    body = Column(Text())
    date = Column('date', UTCDateTime())

    type_info = Content.type_info.copy(

Limiting Forum to the Site Root

At this point, I had an application that worked. I could add a forum to the root of my site, and then add ideas to it.

But, you could add a Forum to any document in the site, and you could add more than one Forum.

I asked a couple questions about this on the #kotti irc channel and was pointed to an example in kotti_media. The solution to both problems is a custom TypeInfo. This one checks whether we have the site root for our context, and if a Forum already exists.

class ForumTypeInfo(TypeInfo):

    def addable(self, context, request):
        """only add once, and only at the root"""
        addable = context == get_root()
        child_type_already_added = self in [
                c.type_info for c in context.children]
        return addable and not child_type_already_added

I then use ForumTypeInfo for Forum's type_info. Note how this is different from the Idea class above.

class Forum(Content):
    id = Column(Integer(), ForeignKey(''), primary_key=True)

    type_info = ForumTypeInfo(

Add Forum Automatically

For a little icing on the cake, I decided to add the Forum to the site automatically, using a populator. I put mine in If forum does not exist, the populator adds it.

def populate():
    root = get_root()
    if 'forum' not in root.keys():
        root['forum'] = Forum(title=u'Forum')

Then add configuration to your .ini file.

kotti.populators = kotti_agora.populate

Pretty slick.

Now, there is one Forum and it only exists at the root of the site. And, if it doesn't exist, it is created when the site starts.


There are so many options for python web programmers these days. My expertise is with Plone, but after using Pyramid for a few small projects I wanted to try out Kotti CMS. Something about Pyramid seems right to me, and Kotti fits the same mold. Kind of like working on a bicycle or a VW bus. I can see it all. It fits in my head nicely, and the problems are approachable. 

Kotti has great documentation, which helps you get started. The project is relatively young and there are some holes in the narrative documentation, but it has evolved in the short time I've used it.

And, a big shout out to the #kotti irc channel. Very friendly. Very helpful.


Jul 09, 2013

Setuptools and a failing bootstrap

by Mike Cullerton — last modified Jul 09, 2013 12:15 PM
Filed Under:

After distribute and setuptools merged back together, I had a problem running bootstrap. Upgrading setuptools fixed the problem for me.

I used zopeskel 2.21.2 to install a development version of Plone, but got an error when I tried to bootstrap it.

michaelc@Cullerton$ ../bin/python
Traceback (most recent call last):
  File "", line 117, in <module>
    ws.require('zc.buildout' + VERSION)
  File "/Users/michaelc/plone_dev/decsys-26/lib/python2.6/site-packages/distribute-0.6.27-py2.6.egg/", line 690, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/michaelc/plone_dev/decsys-26/lib/python2.6/site-packages/distribute-0.6.27-py2.6.egg/", line 588, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: setuptools>=0.7


Upgrading setuptools fixed the problem for me.

easy_install --upgrade setuptools


Hope this helps someone.

Reinout has a related post.

May 02, 2013

Automated Amazon EBS Volume Snapshots with Boto

by Chris Crownhart — last modified May 02, 2013 09:40 AM
Filed Under:

The following write-up describes how to setup, configure and use an easy tool for taking regular snapshots of Amazon Elastic Block Storage (EBS) volumes.

Recently, we were asked by a client to setup regular EBS volume snapshots for their production EC2 instances. After a little research, Mike came across a fairly simple tool called aws-snapshot-tool:  This tool runs as a cron job and automatically creates daily, weekly, and monthly snapshots, and automatically rotates out the oldest snapshots for each time period.

The tool, written in Python, takes advantage of an excellent module called boto (,  Boto is a Python package that provides an interface to Amazon Web Services (AWS) allowing you to easily maintain and manage various aspects of your AWS environment with very little coding.

Install Notes

The aws-snapshot-tool github page has simple instructions to get started, but here are a couple points of interest:

install and setup boto

  1. pip install boto
  2. Setup a .boto config file to store your AWS access key (See:

Setup your Identity and Access Management (IAM) access

When you access AWS through their API, you are utilizing the web service functions to which your account has been given access.  The aws-snapshot-tool includes a sample IAM configuration, that shows which actions need to be setup:

"Action": [

Configuration (including rotation)

Configuration is very easy.  The script uses a file that looks like:

config = {
    'ec2_region_name': 'us-east-1',
    'ec2_region_endpoint': '',
    'tag_name': 'tag:MakeSnapshot',
    'tag_value':    'True',
    'keep_day': 5,
    'keep_week': 5,
    'keep_month': 11,
    'log_file': '/tmp/makesnapshots.log',

There are three important pieces being defined here:

  • The region where your EBS volumes are defined
  • The tag name put on each volume in order to enable snapshots
  • The definition for how long to keep snapshots: 5 days, 5 weeks, and 13 months

Tag Your Volumes

Well, it couldn't be any easier.  Simply add a Tag of 'MakeSnapshot', with a value of 'True', (as defined in the above config file) to any volume you want picked up by this script.  The script loops through all volumes in the entire defined ec2_region_name field, looks for that tag, and handles the snapshot and rotation of snapshots automatically.

AWS Volume Snapshot Tag

Cron Setup & Output

We chose to run our script on another EC2 instance, but this could run anywhere.   We setup the following cron jobs to run:

$ crontab -l

# mon-fri 30 3 * * 1-5 /home/zope/aws-snapshot-tool/ day

# every sat 30 3 * * 6 /home/zope/aws-snapshot-tool/ week

# first sun 30 3 1-7 * 0 /home/zope/aws-snapshot-tool/ month

Each snapshot is created with a description that begins like 'day_snapshot...', 'week_snapshot...', or 'month_snapshot...', like in the following screenshots:

AWS Snapshots - Day

AWS Snapshots - Week


At the moment, I can only think of 3:

  • The script points to a region. If we decide to start using other regions for our servers, we will need to accomodate that.
  • The cost of storage. Is cheap. However, once this has run for a year, we will have 23 snapshots per volume saved in our account.
  • These are real time snapshots. It is each server administrator's responsibility to insure the volume being backed up is in the proper state.

Apr 10, 2013

A Landscape of Python Web Development

by Chris Crownhart — last modified Apr 10, 2013 10:55 PM

Last week we had our kickoff meeting of the new Python Web Development meetup in Fort Collins, Colorado. We had 8 folks at the meeting and had great discussions about a variety of Python products and ideas for future meetings.

In preparation for our kickoff meeting of the new Python Web Development meetup, I started putting together what I called "A Landscape of Python Web Development".  It started with a few of the obvious tools and products that we use at Core Software Group, like Zope/Plone, Django, and boto and suds.

From there, it started becoming obvious that I needed to break down the ever-growing list into mutliple categories.

My list of products played out like this:

The Categories

  • Frameworks
  • CMS
  • Deployment/Hosting
  • Page Templates/Forms
  • Databases
  • Testing
  • Tools/Other
  • Static Site Generators


This list of frameworks includes some old timers, and some newer, 'lightweight' frameworks.  The last one listed, itty, was mentioned in one of Raymond Hettinger's pycon videos, and seemed an interesting, super small framework to check out.

  • Zope
  • Pyramid
  • Django
  • Flask
  • Bottle
  • CherryPy
  • web2py
  • itty


I'm sure I missed some, but these seemed to be the key players that we have come across, and the interwebs seemed to point to.

  • Plone
  • Kotti
  • Django CMS
  • Mezzanine


The first three are cloud platforms, and the last three are samples of python tools related to deployment.

  • Heroku
  • Google App Engine
  • Elastic Beanstalk
  • boto for EC2
  • fabric
  • buildout

Page Templates/Forms

This list was a late addition, but is now pretty obvious.  You can't do web development without page templates and forms.

  • Zope Page Templates (ZPT)
  • Django Templates
  • jinja2
  • chameleon
  • deform
  • z3c.form
  • WTForms


While the following are not directly related to web development, they certainly come into play when building applications that require a backend database.

  • sqlalchemy
  • psycopg2
  • pymongo
  • MySQL-python


Ok, I have to admit that I'm incredibly weak when it comes to writing tests.  But I needed a list of some of the tools out there.

  • unittest
  • doctest
  • coverage
  • selenium
  • funkload


Wow. This list could go on and on and on.  I grabbed a few tools that we use, plus a few others I came across over the last month.  Hopefully, we can provide some presentations about some of these in future meetups.

  • suds
  • requests
  • twisted
  • mincss
  • xml.dom, lxml, etree
  • virtualenv
  • simplejson
  • PIL/Pillow

Static Site Generators

This list of static site generators almost has nothing to do with the above items, but I couldn't resist putting the list together.  I'm fascinated by the concept of someone willing to write blog entries as text files, which in turn get deployed as static HTML to some server on the web.  What I really want to know is: "How many folks are actually using these kind of tools for production sites?"


Finally, while building my presentation, I came across a Python package called Hovercraft!, written by Lennart Regebro, for generating impress.js presentations from a reStructuredText file.  Awesome.  And so easy to use.  You can download my kickoff.rst file and the generated output for my presentation.

Our discussion at the Python Web Development meetup was interesting, and it turns out that most of categories really start to look like the definition of a "development stack" which I suppose could be used to define a set of tools that someone might use to get started doing Python based web development.

I hope you find some value in this list.  If I missed some completely obvious products, please add a comment and let me know.

Copyright © 2003-2018 Core Software Group | 303/809-1001 | Fort Collins, Colorado | All rights reserved.