I’ve spent the past couple of weeks at my new job doing a couple of things: hiring kick ass Python and UI engineers and getting some build-and-deploy infrastructure set up so the team can hit the ground running.
Long story short: I wanted a way to deploy pre-built Docker images from any repository to hosts running in OpsWorks.
I chose Docker because it would let me get a repeatable, consistent environment locally and on various non-production and production environments. And I’d get there a lot quicker than writing Puppet or Chef recipes and using Vagrant.
When it came time to get a non-local environment spun up I turned to AWS due to some networking and security issues around my team’s first project.
Time was of the essence, so I first turned to Beanstalk but found its Docker support problematic. Amazon announced but hasn’t yet released their Elastic Container Service. I ended up picking OpsWorks.
I couldn’t find a lot of advice on the 21st century version of man pages, so I’m writing this up in the hope it helps others, and that wiser folks tell me what I can do better!
Brief OpsWorks primer
OpsWorks is an engine for running Chef recipes based on lifecycle events in the course of a machine’s life.
You start by defining a layer, which is a group of machines that do similar tasks like serve your Web app, run memcache, or host Celery workers.
Then for that layer you define which recipes fire whenever a machine is setup, or an app is deployed to it, or it’s shutdown, etc.
AWS OpsWork and Docker deployment strategy
The best strategy I could find was on an AWS blog post.
Chris Barclay sets up a layer with recipes that install Docker. Application deployments require the OpsWorks instance to pull your code, including its Dockerfile from a git repo and build it locally before running it.
I didn’t like building the Docker images locally from git sources. It ruled out using pre-built community images and opened the door to random build issues on a subset of the boxen.
What I wanted was a way to deploy pre-built Docker images from any repository to hosts running in OpsWorks.
Improved OpsWorks and Docker deployment
I took the code from Chris Barclay and adopted it. You set some key environment variables in your OpsWork application definition and that tells the chef recipe what registry, image and tag to pull and, optionally, the registry username and password to authenticate with.
Here’s the instructions and source to get up and running:
- Set up a new stackinOpsWorks. Under Advanced set the following:
- Chef version: 11.10
- Use custom Chef cookbooks: https git url to a repo containing the recipes
- Manage Berkshelf: Yes
- Berkshelf version: 3.1.3
- Add a layer
- Type: Other
- Recipes
- Setup: owdocker::install
- Deploy: owdocker::docker-image-deploy
- Add an App
- Type: Other
- Repository type: Other
- Environment variables:
- registry_image: The path portion of a docker pull command ala: docker pull $registry_image
- registry_tag: The tag of the image that should be pulled from the registry ala quay.io/yourusername/yourimage:$registry_tag
- layer: The shortname of the layer the image should be deployed to
- service_port: The port on the HOST that will be connected to the container
- container_port: The port on the CONTAINER that will be connected to the service port
- registry_username: OPTIONAL username to login to the registry
- registry_password: OPTIONAL password to login to the registry
- registry_url: OPTIONAL url to a non hub.docker.com registry ala quay.io
https://gist.github.com/cmheisel/e887a10c2007956e43f6
So.. I tried to run this on amazon linux and then realized that it was geared for Ubuntu.. So I change the owdocker_install.rb to this…
case node[:platform]
when “ubuntu”,”debian”
package “docker.io” do
action :install
end
when ‘centos’,’redhat’,’fedora’,’amazon’
package “docker” do
action :install
end
end
service “docker” do
action :start
end
That works fair enough.. Docker does in fact get installed..
After that the images were not being downloaded nor deployed… Nothing in the log files about that good bad or otherwise..
I did some troubleshooting and figured out that since we do not have a signed cert for artifactory yet. Docker refuses to connect to the host.. I dug for many hours to no avail when a co-worker pointed out that the following line had to be in /etc/sysconfig/docker file.. (OPTIONS=”–insecure-registry registry_name.domain”) So I put it there and I was able to manually download the docker image! Cha-Ching! Not so fast..
I had to create a template.. in git.. owdocker/templates/default/docker_config.erb
I placed the line from above in it…
OPTIONS=”–insecure-registry registry_name.domain”
Then I added the following lines to the owdocker_install.rb file…
= action :install
= end
++ template ‘/etc/sysconfig/docker’ do
++ mode 00644
++ source ‘docker_config.erb’
++ end
= end.
Still not so fast..
The config file deployment works as expected.
Still the image doesn’t deploy and still there is nothing in the log files for chef good bad or otherwise related to the deployment of the image. The instance even shows it completed with success as if nothing failed…
I have the following setting for the deploy app
registry_image http:///
registry_tag :1.2
layer deploy_container
service_port 8080
container_port 8080
Any thoughts? Any help is greatly appreciated..
J
The registry_image got eaten.. But it says http://domain.com/image_name
I apologize in advance for babbling all over your comments section here.. 😀
I did find an error that pointed to the deploy process being skipped because of a layer naming problem.. Once I fixed that I was confronted with this even better error…
[2015-03-20T04:31:46+00:00] INFO: IMAGE: Pulling http://domain.io/image_name:1.1
================================================================================
Recipe Compile Error in /var/lib/aws/opsworks/cache.stage2/cookbooks/owdocker/recipes/owdocker_docker-image-deploy.rb
================================================================================
NoMethodError
————-
No resource or method named `docker_image’ for `Chef::Recipe “owdocker_docker-image-deploy”‘
Cookbook Trace:
—————
/var/lib/aws/opsworks/cache.stage2/cookbooks/owdocker/recipes/owdocker_docker-image-deploy.rb:58:in `block in from_file’
/var/lib/aws/opsworks/cache.stage2/cookbooks/owdocker/recipes/owdocker_docker-image-deploy.rb:7:in `each’
/var/lib/aws/opsworks/cache.stage2/cookbooks/owdocker/recipes/owdocker_docker-image-deploy.rb:7:in `from_file’
Relevant File Content:
———————-
/var/lib/aws/opsworks/cache.stage2/cookbooks/owdocker/recipes/owdocker_docker-image-deploy.rb:
51: password deploy[:environment_variables][:registry_password]
52: email deploy[:environment_variables][:registry_username]
53: end
54: end
55:
56: # Pull tagged image
57: Chef::Log.info(“IMAGE: Pulling #{deploy[:environment_variables][:registry_image]}:#{deploy[:environment_variables][:registry_tag]}”)
58>> docker_image “#{deploy[:environment_variables][:registry_image]}” do
59: tag deploy[:environment_variables][:registry_tag]
60: end
61:
62: dockerenvs = ” ”
63: deploy[:environment_variables].each do |key, value|
64: dockerenvs=dockerenvs+” -e “+key+”=”+value unless key == “registry_password”
65: end
66: Chef::Log.info(“ENVs: #{dockerenvs}”)
67:
Any thoughts? 😀
One last annoying post for the night..
So I did notice this…
[2015-03-20T05:26:43+00:00] INFO: Entering docker-image-deploy [2015-03-20T05:26:43+00:00] WARN: Skipping deploy::docker application deploy_container as it is not deployed to this layer [2015-03-20T05:26:43+00:00] INFO: Exiting docker-image-deploy
So I changed the layer to “docker” and tried that.. It would fail as it would always try to run the deploy before it ever ran docker.. I tried a depends statement and that didn’t work either… So I switch it back to docker_container and it fails every time stating that it is not deployed to this layer…
😕
I understand if you delete all of these.. 😀
If you would like to delete all the previous messages.. I took it to an AWS thread.. https://forums.aws.amazon.com/thread.jspa?threadID=174095 Your review would be greatly appreciated.
It looks like you need a Berksfile, I added that to my gist and the blog post. I also replied over on the AWS forums.
I’m kind of a Chef/Ruby noob, and I’ve been having trouble following your instructions. I’ve put the two recipes into my own cookbook (which I’m about 75% sure I structured correctly), and OpsWorks seems to be attempting to read from them.
The problem is that I’m not sure exactly where to put the Berksfile?
I’m kind of a Chef/Ruby noob, and I’ve been having trouble following your instructions. I’ve put the two recipes into my own cookbook (which I’m about 75% sure I structured correctly), and OpsWorks seems to be attempting to read from them.
The problem is that I’m not sure exactly where to put the Berksfile?
You should put it in directory that contains your recipes folder like so:
mypackage/
———– Berksfile
———– metadata.rb
———– recipes/
——————– your_recipe.rb