Recently I’ve noticed a lot of GitHub projects containing .travis.yml files, indicating that the owner uses the Travis continuous integration service to periodically build their project and run unit tests. Out of curiosity, I thought I might I try out some of the hosted offerings that are popular in the open source and startup communities. In my professional Java developer life, the industry standard for years now has been Jenkins (formerly Hudson prior to the fork). Can Travis CI or any of its similar competitors, mostly geared toward Ruby, Python, and JavaScript, measure up for Java continuous integration?
For this experiment, I used my FitnessJiffy project hosted on GitHub. FitnessJiffy is a fairly straightforward diet and exercise tracker web application. My own take on the classic “Java Pet Store“, I’ve written multiple implementations of this app whenever I want some experience with a new framework or language. The linked version is a Spring Boot application, and has a library dependency on my fitnessjiffy-etl project (code for importing/exporting data, and migrating between various database types).
A flexible build process for standalone projects…
The most immediately impressive thing about Travis is its flexibility. Much like a continuous integration version of Heroku, the infrastructure is designed to run pretty much anything you throw at it. There is currently language-specific documentation for 17 different programming languages. Within each, you can specify environment parameters (e.g. which JDK implementation and version to use). You can also simply override altogether the script command that is executed (e.g. use Gradle or Ant instead of Maven).
… but lacking support for dependencies between projects
However, I quickly ran into problems with the dependency relationship between my fitnessjiffy-spring and fitnessjiffy-etl projects. With Jenkins, it is fairly simple to configure relationships between projects, so that when one is successfully built it causes the other build to kick off. Travis seems to lack this concept. Each git repository is basically atomic, in its own little world. If you want separate projects to build in a particular order, then you have to work this out yourself through scheduling or through git push commands being run in the right order.
As it turns out, the problem is even more complicated than that. Even if you have your separate projects building in the correct order, how does the dependent project receive the dependency artifact created by the first build? Travis pulls Maven dependencies only from Maven Central, and your custom snapshot artifacts most certainly aren’t being published there.
To work around this, I tried setting up my own custom Maven repository on Bintray. I updated my pom.xml files, so that fitnessjiffy-etl would deploy snapshots to this repo, and fitnessjiffy-spring would look there for the dependency. No good! After much digging, I discovered that not only does Travis pull from Maven Central alone, but it is actually configured to ignore any additional repos that you add to your POM files. It looks like you simply can’t have Java projects dependent on separate git repositories, unless you pursue additional tricks (which at this point I was no longer motivated to try).
**UPDATE** It appears that Travis does support the concept of private dependencies, but only for paid customers of their travis-ci.com commercial offering (note the .com rather than the usual .org). That’s fine, but the paid offering starts at $129 a month, dramatically more expensive than every other option discussed in this article. There is no way to try out this feature for free while evaluating their service.
A slick continuous integration interface…
My next stop was Codeship, a very slick-looking alternative. My favorite thing about Codeship right of the bat was that there is no need for you to include a special config file in your actual repository (as Travis does by looking for a .travis.yml). Instead, your continuous integration config happens on the continuous integration system, and your source code repository isn’t touched. Coming from the Jenkins world, this just seems obvious to me.
Codeship doesn’t support quite as many languages as Travis does, but most of the major players are represented (and for my purposes we’re only looking at the JVM anyway). For Java builds, Codeship gives you complete ability to script whichever command(s) you want run at build-time.
… but a bit pricey, and lacking in manual operation
Thankfully, Codeship was able to support my custom POM files. However, there is still no native support for multi-project applications. I had to kick off a build of fitnessjiffy-etl, wait for it to finish and deploy the artifact to my private Maven repo, and then kick off a build of fitnessjiffy-spring. Definitely a bit clunky for the typical Java use case. Also, there is no way to manually kick off a build. Builds are only triggered by a push to the git repo, and so the only way to kick off an extra build on-demand is to push an empty commit. I’ve had to deal with such workarounds in my professional life when I’m only given limited access to the continuous integration server. However, if I’m paying for the build server, then I expect a damn “Build” button!
Although paid Codeship plans are more affordable than Travis, they start at $49/month and quickly climb to a hundred dollars and beyond for the ability to run more than one build at a time. At that price range, it’s cheaper to run your own Jenkins setup on a beefy Digital Ocean droplet. Not to mention more flexible and powerful for Java projects.
Half the price of Codeship, and with a “Build” button…
Drone.io is somewhat comparable to Codeship. The list of supported languages is roughly the same, and continuous integration config takes place on the continuous integration server rather than within your source code repository. Moreover, the pricing plans are roughly half as expensive as Codeship’s, and you can launch manual builds!
It’s worth discussing the deployment capabilities of these offerings. A typical continuous integration task is to deploy your application to a server after compiling and testing it. Drone.io supports deployment to Heroku, as well as “beta” level deployment options to Google App Engine or dotCloud. You can also apparently SSH into a remote box and run any custom commands that you script. I have read some online reviews arguing that Codeship’s deployment options are a bit nicer than Drone’s. However, it’s all academic in my case. I run FitnessJiffy on a private DigitalOcean droplet, with my only two environments being “production” and “my laptop”. So I prefer to handle actual deployments myself rather than deploying every successful build automatically. However, it’s an important feature, and worth taking into consideration.
… but still no decent multi-project support
All of the previous caveats regarding multi-project dependencies still apply with Drone.io. You may be able to hack two or more projects together using customized POM’s and a private Maven repo, but there is no true integration between the projects that Drone.io understands. It’s all manual. In the end, I decided to refactor my fitnessjiffy-spring application to remove the dependency on fitnessjiffy-etl. I then chose Drone to provide continuous integration verification for both projects (if you look at their README files, you’ll see the badge logo indicating whether or not tests are passing). So I guess you could say that Drone “wins”, at least for my open-source needs at the moment… but I would not choose to purchase a paid plan for a professional project.
An amazing option…
The final choice I tested was Shippable, which bills itself as the “Truly Fastest CI in the Cloud”. It has the slickest interface out of all the competitors discussed here, even though it follows the Travis model of requiring a YAML file in your source code repository to customize the build in any meaningful way.
Shippable is aggressively generous in their pricing and in what they give away for free. Every other provider’s free plan covers only publicly-visible source code repositories. Shippable, however, gives you free unlimited builds for both public and private projects! There are automated deployment options for Heroku, Amazon Elastic Beanstalk, Google App Engine, and OpenShift. Their paid plan, supporting up to 5 concurrent builds, is only one dollar a month. Compared to the other options in this article, that’s insane!
… if it worked
However, I fear that they’re being a bit too generous, to compete in a marketplace that is seriously over-saturated. Despite their “fastest” monicker, in my experience the service has been ridiculously slow. There are extreme delays before a triggered build actual executes, and I have yet to see a single build and test suite run all the way to completion. I suspect that Shippable simply doesn’t have the horsepower to service all of the freeloaders and dollar-a-month people using their system.
Conclusion
As long as you can refactor or hack your way around cross-project dependencies, any option in this article would be sufficient for putting a nifty little “Build Passing” icon on your GitHub project’s README page. However, it is extremely common for Java applications to have cross-project dependencies. It is extremely common for Java applications to be deployed to a standalone servlet container or JEE app server, rather than to Heroku or another PaaS. There are a hundred other things that aren’t thought about in the Ruby/Python/JavaScript world, but which are taken for granted in a Java shop.
Quite simply, this is what Jenkins does. Open source personal tinkering is one thing, but if I were working on a paid professional project then there’s nothing I’ve seen so far that I would accept as a substitute. What’s more, the pricing for hosted Jenkins solutions really isn’t out of line with the options discussed above. A CloudBees plan for 5 to 20 developers is $60 to $100, with Jenkins priced at 42-cents per hour of build-time on top of that. If you’re already using JIRA for project tracking, then you might like Atlassian’s Bamboo (a commercial CI offering comparable to Jenkins). Decent hosting options are at $50 and $100 a month, priced by the number of concurrent builds rather than the number of developers.
However, having relied on third-party hosted solutions in the past, I still personally believe that there is no better option than self-hosting. A $40 to $80 a month Digital Ocean droplet would probably be more than adequate for one-at-a-time builds of fairly complex Java applications. If you need to get in the range of 16+ GB of memory, for enormous builds or multiple builds run in parallel, then Google Compute Engine or Amazon EC2 starts to make the most financial sense.
Yeah, when you self-host, you are on the hook for solving any problems that come up. However, I have yet to encounter a Jenkins problem that took more than a few minutes of Google-searching to resolve. I’ll gladly take that over the helpless feeling of being down for hours while a third-party provider resolves issues beyond my control.
