Kid On Point

Tom Granot-Scalosub's Personal Website

Continuous Integration & Deployment For Android Using CircleCI - Top To Bottom Tutorial

March 5, 2019

A More Picture Like Explanation Of Version Control Software

February 15, 2019

Premise

As part of my work for Smart Receipts - an expense tracking app company, I was recently tasked with setting up a full CI (Continuous Integration) & CD (Continuous Deployment) pipeline for the company’s Android & iOS apps. This post deals with the Android part, and a follow-up on iOS will be published sometime in the near future.

Contact Me

There’s a comment section at the bottom here, but feel free to reach out to my email with questions and thoughts - use tom @ this lovely website for that. If your comment merits an edit, I’ll probably make a revision and link back to your site. Ah, link juice! :)

Some Context

Running a mobile app business requires a lot of technical logistics. Other than integrating the code written by your developers into a single place (like a git repo), you’ll need a lot of other things done in the process:

And other things that pop up along the way.

It used to be that everyone either did all of that by hand, or ran a dedicated CI server like Jenkins that took care of most of the work for them. Recently, though, a few hosted services became available on the cheap - most notably Travis-CI, CircleCI and JetBrains’ TeamCity (the guys behind IntelliJ IDEA and PyCharm, among other things).

For us, there was a manual process that we did each time we had a new release that we wanted to automate. The process was divided roughly to the following steps:

  1. New Code - A push is made to a certain release_VERSION branch in our GitHub repo. This can be a new release, a bugfix or a cosmetical patch, but either way it needs to go live.
  2. Version Bumping - We bump the version number in our app/build.gradle file, and push that change to the repo.
  3. Handling Multiple Changes - In practice, we collected a lot of these changes before we started the manual process - so in each iteration of the process, we had a few commits lined up for deployment.
  4. Building - We create a release build for all build variants of our app.
  5. Testing _ We run tests on all of the variants.
  6. Publishing _ We then publish all of the variants to the various app stores.
  7. Notifying - Finally we notify a certain e-mail address, reserved for this purpose, about the success or failure of the process. This email should preferably include a complete list of commits that occurred in the current branch, but not in the previous one - to give a quick sense of the release notes that we need to develop

In practice, I achieved full automation on about 90% of this, but I’m happy with the results nonetheless.

Prep Work

Choosing a CI Server/Hosted Service

After a few hours of playing around with Jenkins, I figured out most of what needs to be done aside from publishing to the app store. During the work, though, I realised that we have a cost problem here - Jenkins can rack up quite a bill if left unmonitored.

After looking around for a bit - and noting that Smart Receipts is completely open source - I figured out that CircleCIhas the best offering for our use case. 4 free linux containers (which are great for Android projects) are included in their OSS plan, and that put the nail in the coffin.

Thinking Automatically

The process outlined above could almost be automated as-is. I’ll re-write it again, this time as an automated process.

  1. Listen for a GitHub Push - Specifically, a push to a certain release_VERSION branch in our GitHub repo.
  2. Bump Version - Bump version in app/build.gradle file, and push that change to the repo.
  3. Wait for 15 Minutes - To accept multiple commits that are all part of the same release. This is specific to our build process, and can be removed if you don’t need it.
  4. Build & Test - Build and test all app flavours.
  5. Publish - Publish all variants to Play Store.
  6. Notify - Send notice to e-mail.

We now have a nice, automatic process template we can start to implement.

Enter CircleCI

Setting Up Your Project

CircleCI is, at the end of the day, a server that spins containers. It waits for something to happen (for example, a push to a GitHub repo the user has privileges in), then spins a containers and runs a bunch of pre-defined tasks in it.

In our case, we’ll be waiting for GitHub push, spin up a Linux container that has all the necessary goodies for an Android build, perform all the actions listed above and close the container.

A “Project” is just the name for your GitHub repo on the CircleCI service. When you sign up with GitHub, CircleCI will load up all of your repositories into a projects page, which will be the first thing you see when you sign up. You can click the blue ‘Set Up Project’ button to start configuring your repo.

Writing a config.yml File

Setting up a project in CircleCI includes adding a .circleci folder, and inside it a .circleci/config.yml file. It will determine some important variables about your CI process, set restrictions and detail all the steps that need to happen during a successful build.

If you’re running into permission problems, then clone your repo to your computer, open the repo’s folder and do chmod -R 777 .circleci/. This makes sure the config file is accessible to everyone, including CircleCI.

CircleCI recognises your project as an Android/iOS project, and then suggests you use a pre-made template config.yml file, designed for your mobile operating system. Here’s the one for Android - you can use it, if you wish.

Important Note: Regardless of which branch you want the build to run from, the original config.yml must be added to the master branch. If you’re having problems with it leave a comment below or e-mail me and I’ll elaborate further. This helped me quite a bit with understanding how to work with branches in CircleCI.

I chose to modify it quite a bit, and you can see below the full one I use for the Smart Receipts Android app (don’t worry, there’s a part by part breakdown below).

Note that once you’re done writing your config.yml file, you need to push it into your repo - this should have no effect on your circleci build. You can then press “start building” in the CircleCI interface, to actually trigger your first build. It Will Probably Fail. Don’t worry - there’s a lot of massaging that needs to be done to get your first green build. Drop me a line if you really get stuck below.

Part-By-Part Breakdown

Once you’re done, you can add, commit and push that file into your repo. Then, make sure to click “Start Building” in the CircleCI interface to trigger your first build. Feel free to cancel it - we’ll check it in the next step anyways.

Testing The Setup - Moving Between Branches

We can now test the setup. This is a bit tricky: Note that we’d need to manually edit a branch to trigger a build (since we decided that only specific branches - like a release_XXXXX branch - merit a build). For now, we’ll do it manually - and after this is live, we’ll know the functionality is there.

I’m working with a release_4.18.0 branch on my repo, that was laying there before I started working on this. You can of course choose whichever regex you like and whatever branch you like, this is just for the sake of example.

So I’ll checkout that branch, and then copy over the files I’ve created from the master branch so everything will work smoothly: Make sure you’re in the main directory after you checkout the branch!!

git checkout release_4.18.0
git show master:scripts/version-bumper.sh > scripts/version-bumper.sh
chmod -R 777 scripts/
mkdir .circleci
git show master:.circleci/config.yml > .circleci/config.yml
git add .
git commit -m"adding circleci config file and version bumping to release branch"
git push

This should indeed trigger a build, and the build should fail - that’s perfectly fine, we’re just trying to make sure our initial setup is up and running.

Setting environment variables

As you noticed before, we have a few variables of the form ${VAR_NAME} in .circleci/config.yml. These are just regular environment variables, that are configured inside CircleCI’s interface. These are all the ones I’ve used, so you can check your interface to see you have all of them.

We now have everything set up, except for the publishing part of the app. Let’s configure it now. On to Deployment!

Final Touches - Getting Ready To Publish

We’re almost done - now we just have to configure some things for the Gradle Play Publisher, and sign our app (so the Play Store will not reject it).

Setting Up Gradle Play Publisher

The GPP has great documentation - most of this is taken directly from here.

First, adding the following to the top of your app/build.gradle file (or in the build.gradle file inside your main repo - depending on the way you set up your project):

plugins {
    id("com.android.application")
    id("com.github.triplet.play") version "2.1.0"
}

Note that I used GPP version 2.1.0, and that this is one of the ways you can set plugins for Gradle - there are two possible ways, both of which should work fine.

Now, inside the android block, add the following inner block:

    signingConfigs {
        release {
            // You need to specify either an absolute path or include the
            // keystore file in the same directory as the build.gradle file.
            storeFile file("../upload_key.jks")
            storePassword System.getenv("STORE_PASSWORD")
            keyAlias System.getenv("KEY_ALIAS")
            keyPassword System.getenv("KEY_PASSWORD")
        }
    }

This gets the environment variables required for the signing, as well as the signing key.

Then, at the bottom of the file (but before any preBuild statements you may have) put the following:

play {
    serviceAccountCredentials = file("GPLAY_KEY.json")
    track = "beta"
    userFraction = 1
    releaseStatus = "completed"
}

Remember we saved the JSON key as an environment variable? This uses that key, and places it inside the app folder as needed.

In addition, you can pick which track you want your app to go live on, which release status to assign to it and what percentage of users should get to see it. I chose 100 percent on beta with a release status of completed, by the way - just like in the example above.

If you want to find out more about the nuts and bolts of this, you can read about signing via gradle in the Android docs here.

Protecting The Upload Key

The thing that actually sings your app is a .jks file you received while opening the Google Play Service Account. Since this is a binary, we can’t store it inside an environment variable.

Since I already have a tarball containing other secrets, I chose to add this file into that tarball, and just decrypt it with the rest of them.

These are the steps to stash away your upload key, while making sure it’s available in the main folder when needed. I’ve skipped the commands since dealing with GPG keys is out of the scope of this article - see the guide here for more details:

  1. Add the upload_key.jks to the repo’s main folder, but do not add it to Git just yet.
  2. Decrypt your secrets.tar archive (see above in the .config.yml for an example).
  3. Add the file to the archive.
  4. Encrypt the archive using your GPG key.
  5. Add a line in .gitignore with the file’s name - upload_key.jks.
  6. Add, commit and push all the files to the remote repo.

This should take care of signing the app.

Final testing

Push a minor change to the release_VERSION branch, and make sure your build is green.

One interesting note - if everything seems to work but you’re getting a 500 Internal Server Error from the Google Play Store API, make sure there isn’t an app already in the track. If there is, remove it and try again - this solved it for me.

Conclusion

This was a long process. It’s not easy as I thought, and I’ve followed a six or so different guides until I’ve found my mix that’s written here. Please, feel free to leave a comment or contact me with questions - I hope I’ll be able to help.

Happy CIing!

I’ve found many, many great resources online. A lot of them are listed above with context, and here are a few assorted ones I looked into while writing this: