Announcement blog post

Rollback faster without deploy access!

Announcing forto's open-source project helm-rollback-web

Wouldn't it be great if Product Managers/Owners had the ability to hit the rollback button themselves if they realise the release their team did had unintended consequences?

How about the developers themselves? Not the on-call dev, but any developer who saw the problem and could confirm it was because of their change?

Well, here tends to come a problem - not all product folk have the technical skill nor the access required to use helm rollback at the commandline. Neither do all engineers (the engineer on-call has production access but not every engineer is on call at the time a release goes out).

Some companies solve this by having a "held" rollback step in their CI pipeline (build->test->(sometimes hold)->deploy->(hold)->rollback) but that is messy too because you're going to rollback far less than every release. Worse still, that might mean your pipeline never "goes green".

Enter: helm-rollback-web, a tool we developed at forto to enable any subset of users to rollback a release without cluster access. It also picked up a few neat features along the way.

Here's a screenshot of how it looks running right now :

{{screenshot of main screen}}

Note the blue color of the interface - that means we're operating on the sandbox environment here, if you access it on our production environment, it's a nice red color!

On the left you can see a list of namespaces that are in the cluster and the most recent release in each of those namespaces. On the right, the top 10 most recently deployed releases.

If you click a the namespace in the left hand side you get a full list of deployments within that namespace :

{{screenshot of namespace screen}}

And here's what happens if you click on a release itself :

{{screenshot of helm-rollback-web release list screen}}

Yes, helm-rollback-web can rollback releases of helm-rollback-web!

Now, some of you might be wondering what database the tool is using to track releases so it can display in a nice list like this (which one of our engineers made mobile-friendly because if someone calls you and you want to rollback, you then can). Here's the clever bit, there is no database. This is using the standard helm3 secrets in the cluster. In fact to allow for maximum compatibility, it actually calls the helm binary.

So, in an emergency, you can rollback a bad release at the click of a button, authorised by the groups you belong to (in forto's case that's Product Managers and Engineers). Here's the rollback screen in full glory :

{{screenshot of actual rollback screen}}

Note the little info panel on the right with a direct link to datadog where the application logs are, that's handy! It's also possible to link to the release list with a url like /rollback/{namespace}/{release-name} which allows for things to direct link to the rollback tool, for example, CI slack notifications. Did I mention helm-rollback-web also pings slack so people have visibility?

And so, as this tool is really useful for us, and we think it would be really useful for any company using helm, we're open-sourcing it, giving back to the community that gives us great tools, by building on the shoulders of giants.