FluxCD: Unbreak Helm Release

  • Mon, Dec 18, 2023
  • 2-minute read

Table of Contents

Sometimes when something goes wrong, your deployment will flux will fail with this error:

NAME                    AGE    READY   STATUS
my-helm-release         166d   False   Helm upgrade failed: another operation (install/upgrade/rollback) is in progress

Quick fix

List old Helm releases

❯ helm -n namespace history my-helm-release
REVISION	UPDATED                 	STATUS    	CHART                       	APP VERSION	DESCRIPTION
282         Mon Dec 18 15:14:39 2023   deployed   my-helm-release-1.0.17 0.O.0          Upgrade complete  
283         Fri Dec 15 09:00:27 2023   failed     my-helm-release-1.0.19 0.O.0          Upgrade "my-helm-release" failed: failed to create resource: admission webhook "validate.kyverno.svc-fail" denied the request: failed create policy context: failed to add image information to the policy rule context: invalid image 'gcr.io/cloud-sql-connectors/cloud-sql-proxy:' (bad image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:: invalid reference format)  
284         Fri Dec 15 09:16:12 2023   failed     my-helm-release-1.0.19 0.O.0          Upgrade "my-helm-release" failed: failed to create resource: admission webhook "validate.kyverno.svc-fail" denied the request: failed create policy context: failed to add image information to the policy rule context: invalid image 'gcr.io/cloud-sql-connectors/cloud-sql-proxy:' (bad image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:: invalid reference format)  
285         Fri Dec 15 09:43:13 2023   failed     my-helm-release-1.0.20 0.O.0          Upgrade "my-helm-release" failed: post-upgrade hooks failed: timed out waiting for the condition  
286         Fri Dec 15 09:48:18 2023   failed     my-helm-release-1.0.20 0.O.0          Upgrade "my-helm-release" failed: post-upgrade hooks failed: timed out waiting for the condition  
287         Fri Dec 15 10:42:26 2023   failed     my-helm-release-1.0.20 0.O.0          Upgrade "my-helm-release" failed: post-upgrade hooks failed: timed out waiting for the condition  
288         Fri Dec 15 11:14:33 2023   failed     my-helm-release-1.0.20 0.O.0          Upgrade "my-helm-release" failed: post-upgrade hooks failed: timed out waiting for the condition  
289         Fri Dec 15 13:51:09 2023   failed     my-helm-release-1.0.20 0.O.0          Upgrade "my-helm-release" failed: post-upgrade hooks failed: timed out waiting for the condition

Suspend the helm release

❯ flux -n namespace suspend hr my-helm-release

Rollback to the latest good one

❯ helm -n namespace rollback my-helm-release 282

Verify rollback

❯ helm -n namespace history my-helm-release
...
290     	Mon Dec 18 15:46:07 2023	superseded	my-helm-release-1.0.17	0.O.0      	Rollback to 282

Don’t forget to check your application / pods

Resume the helm release

❯ flux -n namespace resume hr my-helm-release

Issue should be fixed now !

kubernetes fluxcd