Docker Volumes

Getting Started with Docker on Azure

Intro & Prereqs
We’re seeing a pretty significant uptick in the use of Azure as the cloud provider of choice among clients. As organizations move to hybrid and/or multi-provider clouds, Docker plays a key role in abstracting underlying platform configuration details away from implementations, allowing developers to build consistently-functioning solutions that can be tested and run in identical configurations, and that can be reliably deployed to disparate environments. In this post we’ll cover running Docker containers on Azure using Docker Machine and using Docker storage volumes for persistent storage.

Since Azure doesn’t yet have a dedicated container service like AWS and GCP, we’ll need to rely on Docker Machine to get the job done. Docker Machine lets us install and control the Docker service on local and remote VMs. We’ll configure a Docker host and use the same Dockerfile we’ve used in previous posts to test our solution. Before we jump in, we’ll need to have the following items installed:

Docker:

Mongo: https://docs.mongodb.com/manual/installation/

  • (the mongod command should be available from your CLI)

MEANjs.org Yeoman Generator: http://meanjs.org/generator.html

  • (the yo command should be available from your CLI, and you should have a generator named

Setting up the project
To get started, navigate to an empty directory and generate a new MEAN-stack app using the MEANjs.org Yeoman generator and the following command and options:

Command:

yo meanjs

Options:

You're using the official MEAN.JS generator.
? What mean.js version would you like to generate? 0.4.2
0.4.2
? In which folder would you like the project to be generated? This can be changed later. mean
Cloning the MEAN repo.......
? What would you like to call your application? mean-test
? How would you describe your application? Full-Stack JavaScript with MongoDB, Express, AngularJS, and Node.js
? How would you describe your application in comma seperated key words? MongoDB, Express, AngularJS, Node.js
? What is your company/author name? Justin Rodenbostel
? Would you like to generate the article example CRUD module? Yes
? Would you like to generate the chat example module? No
Running npm install for you....
This may take a couple minutes.

------------------------------------------
Your MEAN.js application is ready!

To Get Started, run the following command:

cd mean && grunt

Happy Hacking!
------------------------------------------

Next, we need to create directories to house our Mongo data, and we need to start the Mongo server. Navigate to your project directory (using the names above, it should be in the ‘mean’ directory relative to where you ran the last command) and create directories so that your project contains the following folders:

  • <project dir>/data
  • <project dir>/data/db

While in your project directory, start the Mongo server using the following command:

mongod --dbpath data/db

To confirm your database has properly started, you should see output similar to the output below:

2016-08-24T22:03:19.039-0500 I JOURNAL  [initandlisten] journal dir=data/db/journal
2016-08-24T22:03:19.040-0500 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
2016-08-24T22:03:19.054-0500 I JOURNAL  [durability] Durability thread started
2016-08-24T22:03:19.054-0500 I JOURNAL  [journal writer] Journal writer thread started
2016-08-24T22:03:19.054-0500 I CONTROL  [initandlisten] MongoDB starting : pid=7300 port=27017 dbpath=data/db 64-bit host=Justins-MacBook-Pro.local
2016-08-24T22:03:19.054-0500 I CONTROL  [initandlisten]
2016-08-24T22:03:19.054-0500 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
2016-08-24T22:03:19.054-0500 I CONTROL  [initandlisten] db version v3.0.7
2016-08-24T22:03:19.054-0500 I CONTROL  [initandlisten] git version: nogitversion
2016-08-24T22:03:19.054-0500 I CONTROL  [initandlisten] build info: Darwin elcapitanvm.local 15.0.0 Darwin Kernel Version 15.0.0: Wed Aug 26 16:57:32 PDT 2015; root:xnu-3247.1.106~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
2016-08-24T22:03:19.054-0500 I CONTROL  [initandlisten] allocator: system
2016-08-24T22:03:19.054-0500 I CONTROL  [initandlisten] options: { storage: { dbPath: "data/db" } }
2016-08-24T22:03:19.060-0500 I INDEX    [initandlisten] allocating new ns file data/db/local.ns, filling with zeroes...
2016-08-24T22:03:19.093-0500 I STORAGE  [FileAllocator] allocating new datafile data/db/local.0, filling with zeroes...
2016-08-24T22:03:19.093-0500 I STORAGE  [FileAllocator] creating directory data/db/_tmp
2016-08-24T22:03:19.190-0500 I STORAGE  [FileAllocator] done allocating datafile data/db/local.0, size: 64MB,  took 0.096 secs
2016-08-24T22:03:19.215-0500 I NETWORK  [initandlisten] waiting for connections on port 27017

Now that the database server is running we can start our new application using the following command (again from the project directory):

grunt

To confirm the application is up and running, open a browser window and navigate to http://localhost:3000, where you should see something similar to the screenshot below:

Screen Shot 2016-08-24 at 10.06.04 PM

Now that we have a running application, we can start to build out the docker machine we’ll deploy it to.

Provisioning the Azure VM
To start using Azure with Docker, we need to create a Docker Machine configuration that uses a virtual machine in Azure. Using the docker-machine command line tools, we’ll create an Azure Resource Group (complete with a virtual machine, NIC, Firewall, Public IP, Virtual Network, Storage Account and availability set) using the “Azure driver”. It’s pretty simple. Just use the command below:

docker-machine create -d azure \
  --azure-ssh-user ops \
  --azure-subscription-id <azure subscription id> \
  --azure-open-port 3000 \
  --azure-image canonical:UbuntuServer:14.04.3-LTS:latest \
  azure-test

This will take a few minutes!

Some items to note in the command above:

  • You must replace <azure subscription id> in the command below with your own Azure subscription key
  • We’ve chosen to leave port 3000 open for our application’s development mode
  • We’ve chosen to use the long term support version of Ubuntu 14.04 for our host machine
  • We’ve named the Docker Machine ‘azure-test’

When the machine creation is complete use the following command to verify the Docker Machine is available for configuration.

docker-machine ls

Also make sure your newly created machine is the one that’s currently running with the following command:

eval $(docker-machine env azure-test)

Deployment
Before we go further with Docker, let’s a make a few changes to our app’s Dockerfile so we can run Mongo in our container, and start both the MEANjs.org app and Mongo with one command. In your project’s Dockerfile, replace this line:

FROM node:0.12

which indicates which base image to start with, with these lines:

FROM node:0.12.14-wheezy

# Install Mongo
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
RUN echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2 main" | tee /etc/apt/sources.list.d/mongodb-org-3.2.list
RUN apt-get update

RUN apt-get install -y mongodb-org

# Install gem sass for  grunt-contrib-sass
RUN apt-get install -y supervisor

In this block, we’re performing the following activities:

  • Adding the apt-get repo and relevant keys for the Mongo binary repository.
  • Installing Mongo DB using apt-get
  • Installing Supervisord (a lightweight process control system that we’ll configure later) using apt-get

Near the middle of the same file, replace this line:

WORKDIR /home/mean

With these:

WORKDIR /home/mean

RUN mkdir -p /var/log/supervisor
RUN mkdir -p /data/db
RUN mkdir ./logs

COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf

In this block, we’re performing the following activities:

  • Creating the log directory for Supervisord
  • Creating the data directory for Mongo DB
  • Creating the logs directory for the MEANjs.org application

Near the bottom of the same file, replace these lines:

# Port 3000 for server
# Port 35729 for livereload
EXPOSE 3000 35729
CMD ["grunt"]

With these:

EXPOSE 3000
CMD ["/usr/bin/supervisord"]

In this block, we’re performing the following activities:

  • Telling Docker to open port 3000 for this image
  • Running Supervisord to “start” the image

You may have noticed that one of the commands we added earlier is copying a file named supervisord.conf from the root of our local project directory to the docker container. Let’s create that file, and add the following content:

[supervisord]
nodaemon=true

[program:mongod]
command=mongod --dbpath /data/db
redirect_stderr=true

[program:grunt]
command=grunt prod --force
redirect_stderr=true

Supervisord is a lightweight process control system commonly used in Linux environments. In this file, we’re instructing supervisord to start mongo and our MEANjs node app in background processes, redirecting their output to the stderr log file.

With a configured docker-machine, we’re ready to build and deploy our containers. Start by using the following command to build your docker image:

docker build -t mean-test .

Next, tag the current build with the following command, replacing <username> with your Docker hub user name. This will assign the image we just assembled the ‘latest’ tag.

docker tag mean-test <username>/mean-test:latest

Next, push the newly tagged image into your repo at Docker hub using the following command:

docker push <username>/mean-test:latest

Now, we can create our machine and storage volume. Use the following command to create a named storage volume using the image you’ve just pushed to Docker Hub, again replacing <username> with your Docker Hub username:

docker create -v /data --name mean-test-storage <username>/mean-test:latest /bin/true

Create the machine, linking to the previously created storage volume and mapping our previously-opened http port (3000), using the command below, again replacing <username> with your Docker Hub username:

docker run -d -p 3000:3000 --volumes-from mean-test-storage --name mean-test <username>/mean-test:latest

Test to see if your app is up and running by navigating to it on the web. If you are not sure what the public IP of your machine you can print the configuration details of your machine in the console by running the following command:

docker-machine env azure-test

Navigate to your site using the public IP and port 3000 and you should see the same screen we saw when you ran the app locally. Pretty easy!

Conclusion
I hope this post has provided a simple overview of how to get started with Docker Machine on Azure, and I hope the use of a full-stack application as an example provides insight beyond the basic tutorials available elsewhere on the web.

Code for this tutorial is available on Github at: https://github.com/jrodenbostel/getting-started-with-docker-on-azure

Time to wrap up some side projects and get back to learning more about using the ELK stack, Elm, and functional programming patterns! Stay tuned.

 

Advertisements

Configuring Persistent Storage with Docker and Kubernetes

With DevOps becoming one of the most widely-used buzzwords in the industry, automated configuration management tools like Docker, Ansible, and Chef have been rapidly increasing in popularity and adoption. In particular, Docker, which allows teams to create containerized versions of their applications that can be run without additional virtualization overhead, and is widely supported by PaaS providers. Rather than revisit the value and challenges of using Docker, which is widely written about on the web (good example here: http://venturebeat.com/2015/04/07/docker-in-the-enterprise-the-value-and-the-challenges/), I’ll talk about a specific aspect of using Docker that can be tricky depending on where your Docker container is running – Persistent Storage.

If you’ve worked with Docker in the past or followed the link above, you’d know that one of the big advantages of using Docker is the ability to deploy managed, self-contained deployments of your application. In this scenario, services like Kubernetes, Docker Swarm, and Apache Mesos can be used to build elastic infrastructure – infrastructure that scales automatically when under peak loads, and that contracts when idle, thereby meeting the demands of customers while utilizing infrastructure in a very efficient manner. One thing to note when using Docker is that while it’s very easy to roll out upgrades and changes to containers, when a container is upgraded, it is recreated from scratch. This means anything that is saved to disk that is not part of the Docker manifest is deleted. Depending on the container manager you’re using, the configuration required to enable persistent storage can very greatly. In a very simple example, I’ll detail how to enable persistent storage using standalone Docker, as well as while using Kubernetes on the Google Cloud Platform. This example assumes you have Docker installed, and have a basic understanding of Docker and Kubernetes concepts.

For this post, we’ll start with a simple Dockerfile based on the standard httpd image. The code for this example can be found on Github at: https://github.com/jrodenbostel/persistent-storage-with-docker.

Docker

If you’re starting from scratch, create a simple Dockerfile in your project directory:

Dockerfile

FROM httpd:2.4
COPY ./public-html/ /usr/local/apache2/htdocs/

RUN mkdir /data

You can see this will create an image based on the standard httpd image, copy the contents of the local /public-html folder into the htdocs directory, and then create a folder at the OS root called data.

From our project directory, we can build an image based on this Dockerfile named “docker-storage-test” using the following command:

docker build -t docker-storage-test .

We can create a container using that image and run it on the fly using the following command:

docker run -t -i --name docker-storage-test-container docker-storage-test

That will create a container named “docker-storage-test-container” using our image named “docker-storage-test”. Because the -i flag puts us in interactive mode, after executing that command, we should be greeted with a command prompt on our host machine. At that prompt, if we navigate to /data, we should find an empty directory.

root@c1522a53c755:/# cd data
root@c1522a53c755:/data# ls -a
.  ..
root@c1522a53c755:/data#

Let’s say we wanted to create some files in that /data folder and preserve them when upgrading our image. We’ll simulate that by doing the following:

root@c1522a53c755:/data# touch important-file.txt
root@c1522a53c755:/data# ls -a
.  ..  important-file.txt
root@c1522a53c755:/data#

To preserve our important files between upgrades, we’ll need to create persistent storage for our image. One way to do that with standalone Docker is to create data volume container. We’ll reuse the same image from our original container, and create a data volume container named “docker-storage-test-volume” mapped to the /data folder using the following command:

docker create -v /data --name docker-storage-test-volume docker-storage-test /bin/true

Before we can use our new data volume, we have to remove our old container using the following command:

docker rm docker-storage-test-container

To attach that data volume container to a new instance of our base container and attach, we use the following command:

docker run -t -i --volumes-from docker-storage-test-volume --name docker-storage-test-container docker-storage-test

Same as before, we can navigate to our /data directory and create our important file using the following commands:

root@b170d2f08ff3:/# cd /data/
root@b170d2f08ff3:/data# touch important-file.txt
root@b170d2f08ff3:/data# ls -a
.  ..  important-file.txt

Now, we can upgrade the docker-storage-test image and create new containers based off it, and that file will be preserved:

docker rm docker-storage-test-container
docker run -t -i --volumes-from docker-storage-test-volume --name docker-storage-test-container docker-storage-test
root@00f17622393f:/# cd /data
root@00f17622393f:/data# ls -a
.  ..  important-file.txt

Kubernetes

Google Cloud Platform’s Container Engine can be used to run Docker containers. As Google’s documentation states, the Container Engine is powered by Kubernetes. Kubernetes is an open-source container cluster manager originally written by Google. As previously mentioned, Kubernetes can be used to easily create scalable container based solutions. This portion of the example assumes you have a Google Cloud Platform account with the appropriate gcloud and kubectl tools installed. If you don’t, directions can be found at the links below:

https://cloud.google.com/sdk/

https://cloud.google.com/container-registry/docs/before-you-begin

For this example, I’ll be using a project called “docker-storage-test-project”. I’ll call out where project names are to be used in the examples below. To enable persistent storage on the Google Cloud Platform’s Container Engine, we must first create a new container cluster.

From the Google Cloud Platform Container Engine view, click “Create Cluster”.

Screen Shot 2016-06-05 at 8.31.07 PM

For this example, my cluster’s name will be “docker-storage-test-cluster”, with a size of 1, using 1 vCPU machines.

After creating the cluster, we’ll prepare our image for upload to Google Cloud Platform’s private Container Registry by tagging it using the following command:

docker tag docker-storage-test gcr.io/docker-storage-test-project/docker-storage-test

After tagging, push the image to your private Google Cloud container registry using the following command:

gcloud docker push gcr.io/docker-storage-test-project/docker-storage-test

Create a persistent disk named “docker-storage-test-disk” using the gcloud SDK command below:

gcloud compute disks create --size 10GB docker-storage-test-disk

Verify the kubectl tool is configured correctly to connect to your newly create cluster. To do this, I used the following command:

gcloud container clusters get-credentials docker-storage-test-cluster

Run the image we’ve uploaded in our newly created cluster:

kubectl run docker-storage-test --image=gcr.io/docker-storage-test/docker-storage-test:latest --port=80

At this point, a Kubernetes deployment file is created for us automatically. To mount the persistent disk we created earlier, we have to edit the deployment. The easiest way to do this is to edit the file with vim and create a local copy of it using the current file’s contents. To do that bring up the contents of the current deployment file using the following command:

kubectl edit deployment docker-storage-test

Copy and past that content into a new file. For this example, I’ve pasted the contents of this file into a file named “kubernetes_deployment.yml” on my project folder.

Add a volumes entry to the spec config – this should be at the same level as “containers:”. I added mine at the bottom. Note that “pdName” must equal the name of the persistent disk you created earlier, and “name” must map to the section we’ll create next:

volumes:
  - name: docker-storage-test-disk
    gcePersistentDisk:
      # This disk must already exist.
      pdName: docker-storage-test-disk
      fsType: ext4

Now add a volumeMount entry to the container config:

        volumeMounts:
          # This name must match the volumes.name below.
          - name: docker-storage-test-disk
          mountPath: /data
        resources: {}

Delete and recreate our deployments, this time using the new kubernetes deployment file we’ve created, by using the following commands:

kubectl delete service,deployments docker-storage-test
kubectl create -f kubernetes_deployment.yml

Now, let’s test our configuration by attaching to docker-storage-test container in the pod we’ve just created, create a file in the /data directory, recreate the deployment, and check for the file’s presence by using the following commands:

First, get your pod name:

kubectl get pods

Then attach to the pod and container. My pod’s name is “docker-storage-test-846338785-jbjk8”

kubectl exec -it docker-storage-test-846338785-jbjk8 -c docker-storage-test bash

root@docker-storage-test-846338785-jbjk8:/usr/local/apache2# cd /data
root@docker-storage-test-846338785-jbjk8:/data# touch important-file.txt
root@docker-storage-test-846338785-jbjk8:/data# ls -l
total 16
-rw-r--r-- 1 root root     0 Jun  6 04:04 important-file.txt
drwx------ 2 root root 16384 Jun  6 04:02 lost+found
root@docker-storage-test-846338785-jbjk8:/data# exit

We’ve got an important file – now delete the container, and recreate it, this is simulating the effect upgrading your container’s image would have:

kubectl delete service,deployments docker-storage-test
kubectl create -f kubernetes_deployment.yml

Get your pod name again. Mine is “docker-storage-test-846338785-u2jji”. Connect to the pod and browse to the data directory. We’ll see if our file is there:

kubectl exec -it docker-storage-test-846338785-u2jji -c docker-storage-test bash

root@docker-storage-test-846338785-u2jji:/usr/local/apache2# cd /data
root@docker-storage-test-846338785-u2jji:/data# ls -l
total 16
-rw-r--r-- 1 root root     0 Jun  6 04:04 important-file.txt
drwx------ 2 root root 16384 Jun  6 04:02 lost+found
root@docker-storage-test-846338785-u2jji:/data#

Conclusion

This is just two of the many ways to configure persistent storage using Docker, and container-related technologies. These are just the two that I had to figured out in my recent explorations. Many more can be found in both the Docker and Kubernetes documentation.
The next post may not be out for a while, but based on the trends of my current work, it’s sure to be IoT-based. Stay tuned for more.