The Most Complicated Way to Build a Site

Update: I’ve changed my setup. See below how & why.

It feels like I’ve been changing how my website works for as long as it’s been alive. It started as a WordPress site, then ran Octopress for a while, but three years ago I changed it over to Jekyll, running the Skinny Bones theme. I’m pretty happy with that choice.

Unlike a traditional CMS, Jekyll builds my site offline. There is no back-end and the site is just pure HTML from the user’s perspective. Because there is no back-end, Jekyll isn’t susceptible to viruses or hackers, like a traditional CMS. I don’t need to constantly update the site just to keep it safe and working.

Which doesn’t mean that I don’t. Learning to use Jekyll has been part of the fun, but it sure takes some learning. Originally, I built my site offline and used FTP to move it to my university’s server. I did this for a while before realizing there was a much easier option: GitHub Pages. GitHub Pages supports Jekyll by default, so I didn’t have to do any extra work: I pushed a new version to GitHub and it built a new version of the site. It was great. I bought a domain, pointed it at GitHub and I had free hosting.

This worked for a while, but then an update to Chrome changed how it handled security and suddenly everyone visiting my site got a scary warning. It was hosted on GitHub, but the address wasn’t a GitHub address so Chrome warned everyone of my clever misdirection. Getting security warnings isn’t an optimal way to start a visit to a site, so I had to do something.

I tried some other alternatives and ran into some problems, but eventually I figured out a way to make everything work. The setup works, but is incredibly complex for a simple personal website. It does have it’s perks: there is no CMS, changes are saved by git so I can’t lose anything important even if I mess up, I can work with the site offline – or use an online editor, if I want to.

Here’s the setup I ended up with:

I have local copies of the site on all computers I usually work with, but the site is actually stored on GitHub. This means that when I want to make changes – like post a new blog post – I usually work with an offline copy and push the changes I made to GitHub. Not always, though. There is a front-end for editing Jekyll-based websites on GitHub that I’ve used on several occasions: Prose. Using Prose is a bit like writing on a WordPress site, but just on GitHub.

After I’ve written a post or changed something, the changes need to actually happen on the live website. This happens in two stages.

First, CircleCI checks out the latest version of the site from GitHub automatically. It runs the same build commands I would use offline in a virtual machine and builds a new version of the site. The result is run through a light checkup to see that nothing went wrong – that there are no broken links, for example. If something goes wrong, CircleCI stops and I get a warning in my email.

If everything went well, CircleCI uses a simple rsync command and copies the resulting website to my hosting provider. What ends up there is a simple list of HTML files, checked for errors and less than 10 MB total in size. The journey from command line to GitHub to CircleCI to my hosting provider is quite a round-trip, but at least it works the way I wanted it to. The same setup could be used for a much more complicated site and it would benefit from the same speed, light weight and error-checking as this site.

Setup guide

Here’s a short guide of what you need to do in order to get the same setup working. I’m assuming you know how to install things on the command-line and use git, so I won’t explain the details here.

1. Get Jekyll

Technically, you don’t need a local install of Jekyll. You can build the site on a remote machine that already has Jekyll, but it’s probably easier if you first test that everything works locally. Installing Jekyll could be a bit easier, but basically you need Ruby, RubyGems and some build tools.

2. Build your site

Get yourself a local copy of Skinny Bones, or some other Jekyll template. If you want to build the site yourself, Jekyll can also create a simple skeleton for you to build upon, but working with a template makes things a bit easier.

Skinny Bones has an excellent “Getting Started” guide for the steps you need to take in order to get a working site. The short version is that you need to edit _config.yml. You probably also want to add some pages and write some posts to _posts. There are a few different ways to do it and the Getting Started guide tells you more about it.

At this point you probably want to test your site locally to see that everything looks right. Run bundle exec jekyll serve and you get a nice live preview of what the site will look like.

3. Push to GitHub

Create a repository and push your site to GitHub. You can use GitHub Pages to have GitHub build your page and present it either as a user page or a project page. Now you have a working webpage, if you don’t care what the URL is.

4. Get hosting

Next, you need a host for your site. You need to get DNS services and a virtual machine for hosting the contents of your site. It’s not necessary, but probably easier, if you get both from the same provider. The provider should at least allow connecting through a command-line SSH connection. It doesn’t need to be very fast or have a lot of space, since you are only going to transfer HTML files there.

Next steps will depend on your provider, but the basic steps needed are:

1. Change the name servers to your hosting providers name servers.
2. Change the DNS records to the hosting providers IP address.
3. Check that you can log in and see something in the URL you got.
4. Create SSH keys and copy the public key to the virtual machine.

5. Setup continuous integration

You could also use some other continuous integration service, but CircleCI provides one free virtual machine for building your projects and is pretty easy to use. You can log in with your GitHub account and connect CircleCI to your website repository. CircleCI searches for a circle.yml file from the root of your repository and reads it for build instructions. They provide guidelines for writing a config file, but you can get by with a very simple one. Here’s an example:

machine:
  environment:
      NOKOGIRI_USE_SYSTEM_LIBRARIES: true # speeds up installation of html-proofer

      dependencies:
        post:
	    - bundle exec jekyll build

	    test:
	      post:
	          - bundle exec htmlproofer ./_site --allow-hash-href --disable-external --empty-alt-ignore

		  deployment:
		    prod:
		        branch: master
			    commands:
			          - rsync -rv ./_site/ USER@MACHINE:/home/USER/WEBSITE-FOLDER

This config builds your site, runs a simple check to see that the HTML is valid and then uses rsync to move your site over to the hosting providers machine.

This won’t work as is: you need to change the USER to whatever your username is, MACHINE to whatever the address of the virtual machine is and WEBSITE-FOLDER to whatever the default folder for the server is (probably something like public_html). You also need to add your private SSH key to CircleCI so that it has access to the virtual machine without providing a password.

If everything is correct, CircleCI will now build and deploy your site whenever you push new changes to GitHub. If something goes wrong, the site won’t be deployed and you will get a warning message telling you what went wrong.

Leave me a comment if you have suggestions on how to make this setup better or if you encounter any problems.

Updated setup

CircleCI updated their workflow and the new config turned into a more complicated mess than the one I was using. Instead of figuring it out, I decided to change my site setup.

The new setup I have is only slightly less complicated. I used this opportunity to move from a webhotel to a virtual machine – it was only slightly more expensive and since I don’t have the usual Wordpress setup, it allowed more flexibility in how I setup things.

I’ve wanted to try Caddy for a while now, so I installed it on my virtual server. It supports webhooks so instead of having a build server, I set Caddy to pull from Github every time I pushed a new commit, and to automatically build my site with the latest commit.

This has the upside of getting rid of one extra step because there is no longer a separate build server, but it also puts strain on the virtual machine, because building the site with Jekyll can be more resource intensive than serving a few HTML pages. Since my site only serves a few visitors at a time, it seems unlikely that I run out of resources.

Caddy seems pretty great. The config file was only a few lines long before I added the git plugin and even most of those lines were optional. The major upside to Caddy seems to be that it simply works. Another great feature was enabling HTTPS automatically, which is pretty useful.

I originally became interested in Caddy because I noticed that it had a plugin for handling Jekyll sites. It was basically a CMS for Jekyll sites, allowing everything to be changed through a browser window. That seemed like the best of both worlds, but eventually I decided against using that plugin. I’m so used to writing my posts on my own computer and pushing the changes to Github that I’ll stick to that workflow for now.

I liked my experiment with CircleCI, but having a build server for my HTML website was probably a bit overkill. Let’s see how long Caddy lasts before I decide to change everything again.

More simplification

Apparently the answer to the question in the previous paragraph was “about a year.” I since simplified the setup even more, replacing the automatic git pull with a simple script called deploy.sh, which is little more than a wrapper for rsync. The end result is still served by Caddy, but it no longer handles other parts of the deployment.

For some reason the deployment process failed a couple of times, forcing me to ssh to the Caddy server and do the deployment manually. That was way too much work and I couldn’t figure out why the process was failing. A simple script with rsync is much easier to debug, so that’s what I’m using now.