Get Docker running on AWS OpsWorks

bhcmIBcI’ve spent the past couple of weeks at my new job doing a couple of things: hiring kick ass Python and UI engineers and getting some build-and-deploy infrastructure set up so the team can hit the ground running.

Long story short: I wanted a way to deploy pre-built Docker images from any repository to hosts running in OpsWorks.

I chose Docker because it would let me get a repeatable, consistent environment locally and on various non-production and production environments. And I’d get there a lot quicker than writing Puppet or Chef recipes and using Vagrant.

Screen Shot 2014-12-05 at 9.32.37 PMWhen it came time to get a non-local environment spun up I turned to AWS due to some networking and security issues around my team’s first project.

Time was of the essence, so I first turned to Beanstalk but found its Docker support problematic. Amazon announced but hasn’t yet released their Elastic Container Service. I ended up picking OpsWorks.

I couldn’t find a lot of advice on the 21st century version of man pages, so I’m writing this up in the hope it helps others, and that wiser folks tell me what I can do better!

Brief OpsWorks primer

Screen Shot 2014-12-05 at 9.34.47 PMOpsWorks is an engine for running Chef recipes based on lifecycle events in the course of a machine’s life.

You start by defining a layer, which is a group of machines that do similar tasks like serve your Web app, run memcache, or host Celery workers.

Then for that layer you define which recipes fire whenever a machine is setup, or an app is deployed to it, or it’s shutdown, etc.

AWS OpsWork and Docker deployment strategy

The best strategy I could find was on an AWS blog post.

Chris Barclay sets up a layer with recipes that install Docker. Application deployments require the OpsWorks instance to pull your code, including its Dockerfile from a git repo and build it locally before running it.

I didn’t like building the Docker images locally from git sources. It ruled out using pre-built community images and opened the door to random build issues on a subset of the boxen.

What I wanted was a way to deploy pre-built Docker images from any repository to hosts running in OpsWorks.

Improved OpsWorks and Docker deployment

I took the code from Chris Barclay and adopted it. You set some key environment variables in your OpsWork application definition and that tells the chef recipe what registry, image and tag to pull and, optionally, the registry username and password to authenticate with.
Here’s the instructions and source to get up and running:

  1. Set up a new stackinOpsWorks. Under Advanced set the following:
    • Chef version: 11.10
    • Use custom Chef cookbooks: https git url to a repo containing the recipes
    • Manage Berkshelf: Yes
    • Berkshelf version: 3.1.3
  2. Add a layer
    • Type: Other
    • Recipes
      • Setup: owdocker::install
      • Deploy: owdocker::docker-image-deploy
  3. Add an App
    • Type: Other
    • Repository type: Other
    • Environment variables:
      • registry_image: The path portion of a docker pull command ala: docker pull $registry_image
      • registry_tag: The tag of the image that should be pulled from the registry ala quay.io/yourusername/yourimage:$registry_tag
      • layer: The shortname of the layer the image should be deployed to
      • service_port: The port on the HOST that will be connected to the container
      • container_port: The port on the CONTAINER that will be connected to the service port
      • registry_username: OPTIONAL username to login to the registry
      • registry_password: OPTIONAL password to login to the registry
      • registry_url: OPTIONAL url to a non hub.docker.com registry ala quay.io

Posted in Programming, Python, Technology | Leave a comment

DotCloud: Try ALL THE PaaSes

For fun, I’m writing a series of blog posts breaking out what it takes to deploy this app to a variety of Platforms as a service. All of my sanitized config files are on GitHub.

Today I’ll cover deploying twitter-dedupe to DotCloud

DotCloud

0. General thoughts

DotCloud, like Heroku is easy to grok if you’re familiar with the 12 factor app pattern.

I didn’t find the documentation easy to navigate. I spent more time looking for what I needed than I did with Heroku.

DotCloud stores configuration in a JSON file on your container rather than exporting it as environment variables. That required a minor script wrapped around my daemon code.

I was surprised that DotCloud didn’t offer a way to run or test your application locally. This is the company that brought us Docker so I figured I’d get to use it locally to set up my image.

As you’ll see below, it’s surprisingly not easy to run a staging and production version of your app in DotCloud.

1. Provision redis

Adding Redis to my application was super easy. I added two lines to my dotcloud.yml file and I had a redis stack.

data:
    type: redis

2. Deploy the daemon

  1. You configure what to run using a Supervisord config file. The one I used for twitter-dedupe was pretty simple.
  2. You deploy your code using aDotCloud’s command line tool:
    dotcloud push

DotCloud has git and hg integrations but I couldn’t tell from the documentation if I could select which branch gets pushed to DotCloud each time I invoke dotcloud push.

3. Access the logs

During development and for live troubleshooting there’s a handy command to tail the logs live:

dotcloud logs

There weren’t any built-in connections between DotCloud and Loggly.

That meant diving in and configuring syslog on my DotCloud container and wiring it up to Loggly’s syslog endpoint, or wiring Loggly into my application itself. Neither seemed appealing so I skipped it.

4. Do it all again for a staging environment

I couldn’t find any documentation or best practices for running multiple copies of the same application on DotCloud.

Each folder my computer could be tied to one, and from what I can tell, only one DotCloud application.

So to duplicate my application and having a staging environment I followed all the steps to set up my application again in a different folder.

I ended up with something like this:

dotcloud/
├── slateliteprd
│   ├── dcdaemon.py
│   ├── dotcloud.yml
│   ├── requirements.txt
│   └── supervisord.conf
└── slatelitetst
    ├── dcdaemon.py
    ├── dotcloud.yml
    ├── requirements.txt
    └── supervisord.conf
Posted in Python | Leave a comment

Heroku: Try ALL THE PaaSes

For fun, I’m writing a series of blog posts breaking out what it takes to deploy twitter-dedupe to a variety of Platforms as a service. All of my sanitized config files are on GitHub.

Today I’ll cover deploying twitter-dedupe to Heroku

Heroku

0. General thoughts

Heroku is easy to grok if you’re familiar with the 12 factor app pattern.

I had that in mind when I was writing twitter-dedupe so it wasn’t surprising that I picked it to power @slatemaglite

The Heroku docs had answers for all the questions I had.

The Heroku toolbelt provides nice tools like .env files and foreman to manage and run your app in your local environment.

1. Provision redis

Provisioning redis was super easy. I added the Redis To Go add-on to my account.

I added some code to my app to look for the REDISTOGO environment variable set to a redis://url and I was off to the races.

I was a little frustrated by the need to put a relatively proprietary environment variable name into my code. Other Redis add on providers used similar patterns for their name. I don’t know why REDIS_URL wouldn’t suffice for them all.

Update: [Folks at Heroku agree this should be changed and are working on it](https://twitter.com/dgouldin/status/527154532909056000).

2. Deploy the daemon

Deployment was a three step process: configure, deploy and then scale.

  1. You configure what to run using a Procfile. The one I used for twitter-dedupe was very simple.
  2. You deployyourcodetoHeroku using a Git-based workflow:
    git push heroku
  3. Somewhat confusingly on a new project, you need to scale from 0 to 1 or more instances after your deploy:
    heroku ps:scale daemon=1

3. Access the logs

During development and for live troubleshooting there’s a handy command to tail the logs live:

heroku logs --tail

There a lot of Logging addons as well. I decided I wanted to try Loggly on this project.

Heroku has the concept of Syslog drains which will send your log output to any syslog capable system.

Loggly has an easy integration with Heroku. It’ll give you the exact command to add the appropriate drain. It looks something like this:

heroku drains:add https://{{a url here}} --app {{ your app name here }} 

4. Do it all again for a staging environment

Heroku has the concept of forking applications.

So once I had my initial app up and running the way I wanted, I ran:

heroku fork -a myfirstapp mysecondapp

That copied all my addons and configuration. Then I did some get setup so I could push to both:

git remote add test git@heroku.com:mysecondapp.git
git push test master # Deploy
git remote rename heroku prod

After a deploy I needed to scale up test:

heroku ps:scale daemon=1 --app mysecondapp

And I had a running test environment. Deploying to it, testing and then deploying to prod looks like this:

heroku push test master
heroku logs --tail --app mysecondapp
# Do some verification
heroku push prod master
Posted in Programming, Python, Technology | Tagged | 1 Comment

Try ALL THE PaaSes

I chose to deploy twitter-dedpue to Heroku to power @slatemaglite.

For fun, I’m writing a series of blog posts breaking out what it takes to deploy this app to a variety of Platforms as a service. I intend to keep my (sanitized) config files on GitHub and probably some raw notes of what it took to get things set up.

For each service, my goal is to:

  1. Provision redis
  2. Deploy the daemon
  3. Access the logs
  4. Do it all again for a staging environment

I’ll be trying out these services:

  1. Heroku
  2. Dotcloud/Cloudcontrol
  3. Elastic Beanstalk
  4. Gondor.io
  5. Google Compute Engine
  6. Anything else someone recommends to me :-)
Posted in Programming, Python | Leave a comment

RACI for new leaders

Understanding roles is a perennial issue, but especially as a company scales and small-group communication breaks down it becomes more and more of an issue….

  • Responsible. The people who do the actual work.
  • Accountable. The one person on the hook. ‘The Decider’ as Bush II puts it.
  • Consulted. Opinion contributors.
  • Informed. One way updates.

(Via. RACI — Just about anything)

First off, I love the RACI pnemonic and I’ve used it for years with my teams. In my experience mentoring new leaders there are couple of things to watch out specific to RACI:

  1. Make sure the person you’re making Accountable understands that it isn’t automatically assumed they’re Responsible as well. A lot of new leaders take on the assignment and don’t enlist their team for help performing the work.
  2. Ask them to think thoroughly about whom should be Consulted. Enthusiasm for a new assignment can lead them to run off with their team leave out some key stakeholder. Ask them up front who needs to be Consulted and provide some guidance based on your knowledge of the organization if they’re far afield.
Posted in Management | Leave a comment

OKRs: Adopting Objectives and Key Results

I’ve been looking for a way to up my and my team’s game as competitive pressures, deadlines and customer demand increases. 

OKRs got their start and Intel and made their leap to Google, LinkedIn and other valley companies. In a nutshell: An OKR is a qualitative objective that’s inspirational and some challenging quantitative key results that measure your progress against that objective.

It’s a simple concept, but there’s a lot of nuance to make it something that drives your team forward. Otherwise it’s just another way of doing MBOs.

Here’s my highlights thus far:

  • Objectives should be qualitative and inspirational, time bound (a quarter, a year, etc.), and can be achieved independently by the team.
  • Key results quantify the inspirational language, they are measurable
  • Key results should be hard — the sweet spot is when the team is 50% confident they can achieve it. More confident and you are not driving growth, less confident and your team will give up.
  • Key results should be something that happened because of what you did, not what you did. Good: Customer satisfaction score increases by 10% vs. Bad: Meet with customers, devise features that will please them, implement them.
  • OKRs should result in failure — If the team is achieving all the key results then they were set too low. Most companies and teams are extremely failure averse. If you adopt OKRs, you have to set the stage culturally with the team and with your management above you that you are going to try hard things and miss
  • OKRs are not a performance review. The quickest way to keep people from aiming high is to punish them for missing.
  • OKRs shouldn’t change during the Objective’s time box. OKRs should help focus the team.
  • Start small: One OKR per company with supporting OKRs per team
  • “OKRs are not the only thing you do, they are the one thing you must do.  Expect people to keep the ship running.”
  • OKRs at every level should be available publicly.

Here’s some sources to learn more about OKRs

Posted in Management | Leave a comment

Keep VirtualBox guest additions up to date in your Vagrant boxes

vagrant plugin install vagrant-vbguest

“And that’s it. From now on every vagrant up will check & install the correct guest additions right after booting:”

Every time I ever booted a vagrant box and it yelled at me my guest additions are out of sync, I wish I had known about this. Know I do, and so do you :-)

(Via. kvz.io)

Posted in Programming | Leave a comment

Focus meetings just on decisions, wither understanding?

How Larry Page Changed Meetings At Google After Taking Over Last Spring

Every meeting must have one clear decision maker. If there’s no decision maker — or no decision to be made — the meeting shouldn’t happen.

No more than 10 people should attend.

Every person should give input, otherwise they shouldn’t be there.

No decision should ever wait for a meeting. If a meeting absolutely has to happen before a decision should be made, then the meeting should be scheduled immediately.

I really like these guidelines, and I’m tempted to adopt them whole cloth with my team.

My team meeting agenda is mostly updates on what’s happening within the teams and within the larger organization. There’s a lot of value in seeing someone’s reaction to news and having a chance to answer questions and stop misinterpretations before they start in a setting like that.

On weeks when we haven’t been able to meet, I’ve often sent out my notes of news from the organization I wanted to share. Inevitably the next week I take a few minutes to breeze through it and uncover that someone didn’t understand or made a faulty assumption about what I wrote.

(Via. Business Insider)

Posted in Management | 2 Comments

django-configurations: Sanest approach I’ve seen to having multiple DJANGO_SETTINGS_FILE setups for your various environments.

Plus it looks like it’d work really well with envdir which lets you store environment variables in files in a folder and execute programs using that folder’s files as the context. Example from the docs:

envdir envs/prod/ python manage.py runserver

(Via Glenn Siegman)

Link | Posted on by | Leave a comment

Applying “patches welcome” to management

Refactoring organizations to increase your impact: “So we changed the meetings to focus on proposals instead of problems. Stating and identifying problems is fine, and sometimes it’s all we can do when the solution eludes us. But proposals are the heavy artillery in meetings. They force us to focus on possibilities, and they force us to do our homework and weigh the alternatives.”

Things I love about this post:

  • Proposals as the management equivalent of “pull requests” or “patches welcome”
  • It’s a more sophisticated version of the old management trope “Bring me solutions, not problems”
  • It’s another post about management from folks in the technology industry, something I wish there was more of
Posted in Management | 1 Comment