Using CI Tools for Continuous Delivery?

The views presented here are my own and not of my employer

CI and CD: Introduction

I was reading through the conversation at DevOps tool-chain group. The problem posted was difficulty of using Jenkins as a delivery pipeline. Specially when it comes to complex workflows for example diamond dependencies, fan out & fan in etc. My thoughts after reading post were (Which you will see being different later):

  • Though you can use Jenkins to build continuous deliver pipelines, there are better suited and specialized “deployment automation” tools for the job:
  • I was intrigued and started exploring the tools mentioned in the post. Should one use a CI tool like Jenkins to build delivery pipelines? Or use specialized tools mentioned earlier for continuous delivery? Or choose one good tool and use for both purposes? Are there tools which can do both well and how to choose one – these are questions we are trying to answer in this post.

Continuous Integration

Continuous integration primarily focuses on building code, running tests and ensuring the code is in “clean” state. Since latest state of software is primary purpose – older builds and changes are lesser important as you move forward.

Continuous delivery/deployment

Continuous delivery focuses on deploying a set of changes to one/more environments. The process is typically a set of steps with decision points, parallel flows and roll forward and roll back steps. Basically continuous delivery is comparatively complex workflow with some specialized actions like rolling deployments, talking to servers etc.

In contrast to a continuous integration process, in continuous delivery -

  • Not every build will be deployed – more often than not a deployment candidate might be sum of last few builds. (Ex. Build 450-458 forming deploy version 1.7.4 of software).
  • Changes done for a deployment candidate are sum of all changes done for all builds since previous deployment. Moreover a continuous delivery process has to have knowledge of which build was deployed when and where.
  • The level & granularity of access control needed for a CI process is vastly different than for a CD process. You might want most people to be able to kick off a build but not deploy the artifact to certain environments.

The gap!

As we can see CI and CD process use vastly varied semantics, they aim to achieve different objectives. They handle different units of work and the complexity of process are different too! So should you use one tool to do two things? Let’s try it out. We will test drive Go (From ThoughtWorks), TeamCity (from JetBrains),   Bamboo (From Atlassian), Jenkins (OpenSource) and and see how do they fit in.

Parameters for using a CI tool for CD

1) Support for CI+CD Semantics: Ability to suppport dual semantics for example information from multiple builds into a release candidate

2) Workflow designer: Visual workflow – to build workflow which is easy to see and modify over time

3) Features: Standard actions available for various CD tasks  - like deploy to tomcat, connect to server XYZ etc.

4) Extensible: Ability to write custom actions/plugins for doing adhoc jobs – because not all actions provided in tool might fit your needs.

5) Miscellaneous factors: Can you build rolling deployments? Is it agentless or needs agents on target machines? Does tool support horizontal scalability for handling huge volumes (Which is tied to agent based model to some extent) etc.

So let’s test drive one by one!

Go- ThoughtWorks

Go almost felt like a tool which is malleable and would fit well in your organizations’s existing knowledge (Scripts, Tools etc.). It’s minimalist interface coupled with a very good visual workflow builder gives you a perfect candidate which you can use to build continuous delivery pipelines.

TeamCity- JetBrains

TeamCity – which is an excellent CI tool had to be fit in to do continuous delivery tasks. It can be a good fit for doing simpler continuous delivery pipelines if you have a wide variety of technologies- handling which is a unique strength of TeamCity.

Bamboo – Atlassian

Bamboo came out as one of best candidates for doing both Continuous Integration and continuous delivery at the same time. Product understands the two paradigms separately and handles them that way.

Jenkins – Open Source

Though it is possible to build continuous delivery pipelines in Jenkins – it felt a little unnatural. We did not explore all possibilities with many plugins available, but at the surface it looked feasible – but not natural.

Conclusion

We used continuous integration tools to also do continuous delivery – and surprisingly some tools like Go and Bamboo did a real good job of it. Does this means using specialized tools for continuous delivery is not needed? Definitely no! We used a rather simplistic scenario and did not deploy at a scale which is common in many companies today. So that conclusion would be half baked.

Nevertheless I believe that companies which already have these tools can start implementing continuous delivery pipelines – they may not be perfect but you will understand what works well. Some of your pipelines might as well be better residents within CI tools. You will also gain valuable insights into your delivery pipelines and will aid in better decision while buying the continuous delivery tool! Secondly if the budgets for CD tool is not very high – it might make sense to get started with these tools and eventually make decisions on buying CD tools.

Would love to understand the CI and CD scenario of your team!

16 thoughts on “Using CI Tools for Continuous Delivery?

  1. Tnx for a great post! I’ve been building pipelines in Go, Jenkins and Bamboo and basically agree with your conclusions on those and it was interesting to see TeamCity in comparison, haven’t worked with that one yet.

    I’m curious about the phrase “specialized tools for continuous delivery”, is it your opinion is that tools like Nolio, LiveRebel and DeployIt are superior for complex CD workflow over traditional CI tools? I haven’t worked with those tools but to me they seam more like specialized deployment tools and it would be interesting to see how they would implement a complex CD scenario.

    Basically I think all tools in this segment claims a little bit of CD but in my opinion each tools should rather be used for what is is good at and the integrations of tools is what makes up the complete pipeline, there is no tool to rule them all. Unfortunately, when chaining and integrating several tools to implement the workflow you will easily loose visibility and accessibility of the process, which is why many companies that are good at CD builds this layer them selves, i.e. orchestration, dashboards, integrations etc. Flow (http://startflowing.net/) is an interesting new alternative that has this mindset but still in my opinion lacks a lot for a full scale enterprise CD implementation.

    Tools or services like Travis that take a data driven approach I think are very exciting and refreshing because it is far to easy to gradually build up a maintenance hell in manual clicking intensive tools like Bamboo or Jenkins if you run CD at scale.

    • Thanks a lot for detailed response Andreas. Your comments have sparked some interesting thoughts in my mind for a potential future post- specially around the CD tools.

      WRT “specialized tools for continuous delivery” – yes these tools have capabilities which are suited better for CD – they support complex workflows, release templates and actual releases etc (For example Nolio has built in actions and drag and drop type workflow capability). Though each tool has it’s own set of features which solve certain problems. And I agree to your point that each tools in this landscape claims to do same/similar thing.

      I have not explored much of startflowing.net or Travis CI but I will definitely check them out. Thanks again.

  2. While TeamCity does cater for hybrid composites by build triggers and dependent builds, it does not have builtin semantics for promotion of artifacts so you need to roll your own for several missing features, though less than with jenkins. This means resolving to Nexus and/or Yum to store the blessed artifacts of the builds. And implementing promotion, pull from
    repo, and delete-from-pipe yourself. I noticed that Quickbuild does cater for these constructs. Artifactory seems more feature-complete than Nexus too.
    What remains is full-stack dependency management, i.e. SDN, base OS, middleware, database changes forwards/backwards, et al. These should be journalled in a coherent set of ‘plans’ so you can create any version in a test environment to debug on a specific replica. Ansible was not mentioned, but is very capable to do all kinds of plumbing, including orchestration, ad-hoc commands and idempotent declarative infrastructure updates, also of non-Java stuff.

    • Thanks for insightful comments Bas. Though I have never heard of QuickBuild before – it looks interesting from features and screen tour. As far Ansible – I think of it as a separate and a topic which deserves it’s own space. Though Ansible/Salt/Puppet/Chef do similar things and provide bottom up stack building capability – what is to be seen in near future is how they co-exist with container virtualization technologies like Docker etc. I personally think that the needs served by CI/CD tools are rather different than needs served by CM tools like Puppet/Ansible etc. But may be a deep dive and comparison in that space for some other day!

  3. Thought-provoking article. I like the way you used embedded slides to illustrate. Nicely done.

    I’m not sure I agree that CI and CD really represent two different problem domains. Deployment is a standard part of the software life-cycle: develop, test, and deploy. Why arbitrarily separate deployment?

    Context is important here. Are we talking about a piece of custom software developed and deployed in-house? Or a shrink wrap product developer? Or deploying an existing application like mySQL? The context will, I think, be the determining factor rather than the phase of the life-cycle.

    • Thanks Ken for info. Was happy to read email from Thoughtworks two days ago about the news. Eagerly waiting for the source code and to play around with it!

  4. Create article, but left out two key things:

    1. Pre-commit builds – if a developer can check in code that breaks the central build for everyone else, then your pipeline is not truly “continuous”.

    2. If your individual pipeline tasks are tightly bound to the structure of the pipeline itself, then you can’t “orchestrate” pipelines… another crucial feature of Continuous Delivery. For example, can the tasks (such as build project x with Maven) be re-used in other pipelines without duplication? How easy is it to re-arrange the pipeline, branch off to run additional test/deployment tasks and then re-join for a final promotion/deployment step?

    Point 1. is solved by TeamCity or using Jenkins in combination with Gerrit. Don’t know what Go and Bamboo can achieve in this space, and would be interested to find out.

    Point 2. is solved by the very new “Build Flow Plugin” in Jenkins, which allows you to let go of Jenkins’ notion of upstream/downstream build jobs, and orchestrate the whole pipeline from a single Groovy-like DSL. Again, I don’t have first hand experience or doing this in Go or Bamboo, but I’m pretty sure this is impossible in TeamCity, where each step/project can only use the output of a single preceding step/project.

    • Thanks for your comment Rick. You bring up two excellent points, which I must admit are not well covered in the article. I must focus on some of those in my next article!

  5. Hi Vishal,

    I loved your detailed analysis of the different tools. What I am really interested in is your opinion on how these compare with the specialized CD tools. You mention some differences in your replies to earlier comments, but I would love to see a full analysis of what for instance Nolio or DeployIT can offer that Go does not. Working in a very complex landscape, we use Nolio for our deployments, but I would of course always prefer an open source version if it could support the same flows.

    Thanks,

    Wouter

    • Hi Wouter, thanks for your kind comments. I would also love to do a analysis of commercial CD tools against open source CD/CI tools. There are two challenges though 1) Getting licenses for this kind of test on my experiments. I have worked on Nolio – but at work and don’t have exposure to other commercial once. 2) Scale: This is something hard to simulate in sandbox that I operate in – and that’s where I think enterprises who have scaled CD can add some valuable inputs. But I would definitely love to analyze this area further, thanks for reading.

  6. Hey Vishal, your blog and this post was mentioned in a delivery meeting today by Ned. Thanks for this writeup, I learned a lot! I don’t usually interact much with these tools since I work in the UI space, so it was really nice to learn about them here, particularly the CD part, which I have very little knowledge of.

    PS. I hope you’re still getting out a lot to photograph the world :-)

    • Hey Marshall, I am glad that you enjoyed reading posts. I am doing a series on DevOps and will keep you posted when that finishes.

      On photography front, I had slowed down due to other commitments but picking up pace again. Nice to hear from you!

  7. Really liked your slides. Currently using Jenkins but am disappointed with the lack of coherency between plug-ins for configuring pipelines (ala your example of the manual step with the Build Pipeline Plugin.) Very frustrating.

    I have used the Promoted Builds Plugin as a way to model a build lifecycle including “manual” steps (like blessing a build for production) which can easily be set via its REST API, which kicks off the next step in the process. This makes integrating with external systems very easy.

    I have tried to model the same thing with TeamCity, but don’t see the same concept as a promotion to hook into the pipeline from external tools. Any ideas?

    I look forward to your future articles. Please post a comment here when you publish your next article so I get the email notification! Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>