Building Internal Web Applications with Docker

Create a template for building internal web applications with Docker+Django+PostgreSQL+NGINX.

One of the most powerful things about being an engineer is taking a simple idea, exploring it and bringing it to reality. From a little kid building my own toys to spending weekends creating new frameworks, Ive always enjoyed this. As I grew older I wanted to solve different types of problems and this is what brought me here.

Almost 4 years ago I became a team lead/supervisor and I became obsessed with using data to make decisions and building tools to make my team more efficient. Some of my tools just had a user of 1 and others gained more popularity. I researched better ways to develop and deploy applications and after many Stack Overflow posts I found Docker and pieced together this template. This post is a collection of my thoughts on this journey which is not over by any means. I want to lay out a very simple, reusable and scalable template meant for building internal applications.

Opinion: Companies need to invest in building internal applications to make their employees happier and more efficient. Few things saddens and angers me more than seeing human potential wasted on bad processes. Horrible tools leads to apathy and even causes people to doubt their competency and sanity.

Why Internal Applications?

One of the things I cannot stress enough is scoping. You should always be clear about the problem you are solving and up front about when assumptions break down. Internal applications are applications that are meant to solve a particular company problem or to aid some process. Even though this template utilizes NGINX as a proxy, the application is not going to be fault tolerant, nor optimized for region/geography and it is depending (not completely) on being behind the company's firewall for protection. I would point out though, that NGINX and Django provides some security. In a nutshell, internal applications are scoped to relatively few users and the data generated is of some business value.

Why a template and Docker?

Since I started programming I always looked for ways to be more efficient and not have the blank screen stare of "How do I start?". Templates and frameworks gives you a starting point and should make it easy to do the right thing and do the same thing over and over. A template and Docker aids in both of these. I use Docker because I can easily scaffold out a new project and begin quickly working on the business of the application - designing models, views, urls, etc. If you are not familiar with Docker, strap in because your world will change. There are tons of material on the interwebs for Docker; but think of Docker containers as magical boxes that each do something specific and you can assemble them to build the backend of your application.

Design Decision: I chose Docker because it gives me the ability upgrade or swap out a container with little impact to other parts of my application aka loose coupling and high cohesion.

Deliverables

1. Instructions (this blog post) on how to use the template and what is going on in each file.

2. Link to the Github repo where you can find the necessary files to build the template I use on internal projects.

3. Joy and Happiness

What are we building?

The template will launch four Docker containers:

• NGINX: will be our reverse proxy or handle client traffic and hand off to the web server. It can also serve static files (css, javascript, images, etc)

• Django/Gunicorn: Django is the Python web framework that has a huge following and lots of stack overflow posts. I use it to build models, views, and url routes. Gunicorn is a HTTP server that natively supports Django and is a drop in replacement for the Django development server.

• There are two containers for the database. The persistent store is used to hold the table-space and the process container is the actual Postgres server.

Design Decision: The reason I created two containers for the database is so I can upgrade the Postgres process without affecting the data. The persistent store actually mounts to a folder on disk.

I know what you are thinking - "Brandyn I thought we were building simple web application, this seems like overkill!". I know I know I know. I cannot tell you how many times I've said the same thing. I started building something small with the best intentions of it not growing and inevitably it always grew 99.9999999999999% of the time. Why not make it easy to do the right thing from the beginning?

Let the Games Begin!

I probably shouldn't say this, but if your are not interested in the details you can skip to the end where I give instructions on what files you need to download and what steps you need to take to setup the template.

But if you do care about the details- great!

Design Decision: The software stack I used for this is Github/Github Desktop, Atom and the Command Line.

Components of the Template

The folder structure is:

The template consists of 5 files which are going to go through in excruciating detail:

1. Requirements.txt

2. .env

3. Dockerfile

4. docker-compose.yml

5. Makefile

As well as 3 folders:

1. config (Used to store the NGINX config file and any other file you may need in the future)

2. db (Mount point for the database)

3. src (where the source code ie. Django stuff will be. Note there is a sub folder called static)

Requirements.txt

This file is used by the Dockerfile when building the Django container. We will start off with a Python image and then pip install these libraries.

Design Decision: Be very specific with version numbers. This makes sure your environment is reproducible and there are not any bugs introduced because of library updates. This principle goes for docker images as well.

.env

I use environment files to provide a single touch point for a user to change parameters so they do not have to mess with any other files unless they want to. In this file we specify the Database name, user, password and port to pass to the docker-compose file for the database container; and the Djangoapp_name is for the web container.

Dockerfile

The purpose of the dockerfile is to define the parameters of a docker image you want containers to be based off of. Think of the dockerfile/image as the specification and the container is the instantiation of what you specified.

Update: Updated the dockerfile in Github to use the python:3.6.3-slim image. It is a lot smaller, ~156MB compared to ~700MB.

FROM python:3.6.3 - Specify the python image you want to base this custom image off of.

ENV PYTHONBUFFERED 1 - By default Python stdout is buffered. Unbuffering simply means getting an output during execution of python code instead of at the end.

RUN mkdir /src - Create a folder called /src. Note that folders in containers do not reflect folders on disk unless you mount them or specifically add files.

WORKDIR /src - Make the src folder the working directory.

ADD requirements.txt /src/ - Add the requirements file to the src folder.

RUN pip install -r requirements.txt - pip install all the libraries from the requirements file.

Further docker documentation: Here

Docker-compose.yml

Docker compose is a tool for defining and running multiple services. These services can be launched with a single command: docker-compose up. The docker compose template provided has four services defined that correspond to the 4 containers we will launch.

Database Service:

db and db-data are the names of the name of the services. Technically I could have used the env variable I created (DB_SERVICE) but just did not feel like using it. <- Im human. If you are going to change the name in the env file, change the name of the service in this file.

restart:always - This instructs the container to restart if it crashes.

image: postres:10.0 - Use the postgres image to build the container.

container_name: postgresdb_01 - give the container a name so it can be referenced.

volumes_from: -db-data - reference any volumes created and mounted in the db-data container.

environment - read the variables from the .env file and pass them as environment variables to the container.

ports and expose - the ports command also exposes the port, but decided to include both to be explicit.

For the db-data, its purpose is to just maintain the tablespace and mount the postgres data to the db folder on disk. The command true is a no-op. The behavior you will notice is the container restarting over and over. This is ok.

Web Service:

build: . - This command builds the dockerfile at the same location as the compose file.

command: will override any commands in dockerfile, but we do not have any. This commands tells gunicorn to be the http server for Django, create 2 workers and map to port 8000.

env_file - pass in the .env file to be used as environment variables in the container.

volumes: mount the src volume created in the container to the src folder on disk.

ports and expose: map the container ports to the port 8000 of the computer running the application.

depends_on - makes sure the db service is up before launching this. This is because Django needs to connect to connect to the db service.

Hacking: You may not need to use NGINX. If you do not want to use it, you can simply comment out the NGINX service. You can access the Gunicorn server by using the configured port, in this case 8000. One caveat is you will have to figure out serving static files with Gunicorn.

NGINX :

Nothing crazy is going on here, Im just mapping the config folder on disk to the conf.d folder in the nginx container. But at this point I guess you are wondering, how does the NGINX proxy server pass requests onto the Django/Gunicorn HTTP server? This is defined through a conf file that is in the config folder.

In a nutshell there are two different location blocks. One serves the static files and the other simply passes all traffic that comes to the computer's IP address to the Gunicorn HTTP server at web:8000. (Remember web is the name of the service and it serves as the hostname of that service).

Makefile

This is the magic. This file is what makes it easy and repeatable to scafold out a new project. The makefile creates simple commands you run in the command line. To use a command, simply type: make <command>

include .env - Bring in the .env variable file so variables can be used.

setup - creates the necessary folders.

create_project - creates a Django project based on the Django app name in the .env file.

build - creates the necessary docker images.

create_app - creates a Django app with a name specified by the user. ( make create_app app=<User Defined Name>)

migrate_data - this runs Django commands to push any model changes to the database.

collectstatic - runs the Django command to collect static files and move them to a particular directory to be served.

unit_test - runs the Django command to execute any unit test defined in Django.

up - creates containers for each of our services and begins running them.

down - shuts down all running containers that were spun up by the up command.

start - starts all containers after they have been stopped by the stop command.

stop - stops all running containers associated with the docker compose file.

bash-web - opens bash for the web container and allows a user to run command line

bash-db - opens bash for the db container and allows a user to run command line.

log-web - shows the logs for the web container.

How to use the template?

1. Create a project folder on your computer.

2. Clone the template repo and rename it with the specific name you want.

3. Clone the remote repo you created in step 2 to your local project folder.

4. Using Atom (or your preferred text editor), update the variable values in the .env file.

5. Using Bash/Command Line tool navigate to your folder and run make setup

6. Build the docker image by running make build

7. Create the Django project by running make create_project (Note that you may get an Error 1, Im working on fixing this, you can ignore it).

8. Open up the Django settings file and update the database settings. Replace :

with:

and add to the end of the file:

10. Move the static files to be served by running make collectstatic

12. Now everything is ready. Launch by running make up

13. You can access the application by simply entering in the IP address of the computer/server/VM running this application.