Moving towards full automation for build, test and deploy

A couple of topics which have interested me for a while now is CI (Continuous Integration) and automation. I’m introducing tools to provide developer support using these approaches for my team to enhance the experience of new application builds and regression testing. The latter is of particular important as this removes the need to hassle the original developers of applications whenever a change has to be made, to a large extent. There are also benefits in the area of handing over support more effectively to first and second line support teams. However, achieving CI and automation is a complex challenge because our applications are loosely coupled n-tier, spread across three core platforms: Liferay Portal 6.1 EE, Orbeon forms 4.0 PE and Weblogic 12c middleware (coming soon!).

For those who are not familiar with CI, this is the practice of integrating code changes frequently into a shared code branch, often your mainline (a.k.a. trunk or master). The theory is that by integrating frequently, you will be integrating smaller changes, and in larger teams this means a decreased likelihood of code branch merge conflicts. As it turns out, this is true. It is made easy with support from powerful SCM (Source Code Management) solutions like Git where branching, and more importantly merging, is a low cost activity. To get the full benefit it is recommended that you integrate your SCM with an issue tracker and branch for every issue, no matter how small, and make include issue IDs in commit messages when you commit the code changes that represent fixes. This facilitates automatic generation of release notes for builds and issue trackers are often able to tell you which commits resolved which issues.

Automation is a natural next step once you have achieved a level of CI. This works best if the automation happens on a server rather than the developer’s desktop and the builds / tests / deploys are triggered as a consequence of new code being committed to nominated branches in a common repository. With this approach you are guaranteeing that all source code is committed to the repository before any deploy takes place to a testing or production environment. This is of course if all developers agree not to deploy any code without using configured jobs on the automation server. I haven’t discussed automated testing in this post, but implementing automated testing is crucial to realising many of the benefits of automation. Especially in the areas of regression testing and effective support handover to first and second line support teams.

Some resources

I would suggest watching “Jumping from Continuous Integration to Continuous Delivery with Jenkins Enterprise” by Mark Pritchard (@mqprichard) and Andrew Philips (@xebialabs)  if you’re interested in learning more about how this can be implemented in software.

If you are new to CI and automation, a good place to start is Wikipedia.

On to discussions!

There are a variety of possible solutions for a CI + automation approach, and the solution largely depends on the platforms you are building on. Automation in particular is an intrusive change to the development lifecycle, which can easily become seen as obstructive without the right level of buy in from the core stakeholders: your developers! So the only real way to implement it is through a team collaborative exercise. So my focus is on putting in place a framework that enables us to have a baseline for on-going conversation and improvement. And this is also why I am writing this blog post to a large extent. It would be great to hear from other technical leads who have attempted or achieved CI and/or automation already, particularly for the platforms mentioned. It would be great to hear about your experience, so please leave a comment!

If you work as a technical lead or are simply a motivated developer working for a local authority then it would be great to have you as a member of the IT Solutions Architecture group on the Knowledge Hub


6 thoughts on “Moving towards full automation for build, test and deploy

  1. The techniques look good. I was just wondering what applications are currently being joined in LAs. Might there be other LAs attempting the same who could collaborate on Github?

  2. I’m hoping there are. And if you come across any, please let me know! The great thing about using Git for SCM is that it is distributed. So even if our main source code repository exists within our firewall, we can push and pull code from repositories hosted on GitHub or indeed in public facing DMZs of other LAs. So collaborating at the code/product level has never been easier. Also, on a related point, we’ve implemented a completely open source web service presentational architecture at DCC. This brings with it licensing arrangements suitable for duplicating the architecture at other LAs should this be desirable. The presentational services running on this architecture will be capable of connecting to existing SOA webservices available at other LAs, making this approach very scalable and a good collaboration baseline. In our case our SOA webservices are hosted on Weblogic 12c.

    • I tried, and failed, to post this as a KHub response.

      “Merely curious. You are undoubtedly doing the right thing. CI would seem to be commonplace in more advanced development shops. I was just wondering how many of the smaller unitaries and districts have the capacity to convert their portals to more business transaction input. I have a feeling that we would need a good standard set of APIs to keep the overall costs low. Have you checked out Layer7? I met them at conference a couple of weeks ago. ”

      Why do we (the Government) need to develop a bespoke service when WordPress does an adequate job for zero development cost?

  3. I hadn’t come across Layer7 before. They have some very relevant information on their website. Thanks!

    The benefits that come from delivering web business transactions that deploy to a standards based web architecture need to be made very clear to LA management. Otherwise it will simply not be resourced properly and could become a considerable development overhead with benefits yielded to a small subset of the LA only. But if resourced properly, well informed standard based architecture will flourish, and business transactions can be developed largely in isolation, such as at other LAs. This is when the capacity concern is diminished, because you are pooling the resources of LAs around the country!

    Further savings are made in areas such as realising channel shift, because the architecture should offer user access control, authentication and authorisation as a service to the deployed business transactions. This means LAs don’t compromise on the unified web services delivery needed in order to make their web channel more enticing to the customer.

    What kind of business transactions do you envisage WordPress offering at zero development cost?

  4. Sorry if I was misunderstood. It is the centrally funded KHub development I was questioning. It is unreliable, whereas WordPress works as a reliable, open source, free service for information exchange. I go to many meetings where KHub is promoted, but my experience has turned me off attempting to use it. The LeGSB group is not even sufficiently trusted by the Web filtering systems of all Board members.

    There are probably other open source business transaction services around – but I haven’t researched the market. It’s the sort of thing I think SOCITM should be commissioned to research.

  5. I think the team behind the KHub had the same ethos. The development was built on an earlier version of the open source Liferay Portal platform that we at DCC use. It’s a good platform today, but the KHub implementation from what I gather is a community edition fork from 2-3 years ago which has meant it is now difficult to upgrade to benefit from the latest features of the platform. I too encounter a lot of bugs with KHub and it can get quite frustrating. Hopefully they will resolve the version lock issue at some point and maybe that can be achieved by the KHub pooling our team’s resources even for replacing certain buggy features that also are important to us.

    SOCITM could/should definitely have a role to play if such collaborative working is achieved on a national level. Possibly by helping put pressure on suppliers of the G-Cloud to provide open APIs to their offerings.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s