5.23 (Kubernetes)

Upgrading from a version earlier than v5.20?

v5.23 itself adds no new components, but two optional services were introduced in recent releases. If you are upgrading from an older version and want them, install them as part of this upgrade:

• Cinchy MCP Server, added in v5.19. See New Component: Cinchy MCP Server.
• Cinchy Email Notification Service, added in v5.20. See Notification Service Setup.

Both services are optional. Skip this note if you are already on v5.20 or do not need them.

Release Notes

v5.23 does not add any new platform components or require new deployment.json fields. The significant change for Kubernetes deployments is infrastructure:

• Argo CD upgrade: Argo CD moves from v2.7.6 to v3.5.2. This is required, not optional, and it changes the command used to deploy Argo CD. See Upgrade Argo CD to v3.5.2.
• AKS default version: moves to 1.36.3.
• Model upgrade: On first start of the new version, the platform automatically applies model version 1.0.143, which renames [Cinchy].[Users].[Can Design Queries] to Can Save Queries and adds the new Can Write Queries column. See Query privileges split. No manual database changes are required, the upgrade runs automatically.
• Identity Provider startup check: From v5.23, CinchySSO verifies token signing at startup and will not start if its signing certificate is unusable. A deployment with a subtly bad signing certificate that previously started and then failed token requests will now fail to start instead. If the pod does not become ready after the upgrade, check its logs for the token signing verification line and confirm the signing certificate and its private key are correctly mounted.

For the full list of changes, see the 5.23 Release Notes.

If you are upgrading from a version earlier than v5.17.2, this release also carries forward the following infrastructure changes first introduced in v5.17:

• Kafka Cluster: Upgraded to version 4.0.0
• Strimzi Kafka Operator: Upgraded to version 0.49.0
• Amazon Machine Image (AMI): AL2023_x86_64_STANDARD support updated for EKS 1.33
• Hashicorp AWS Provider: Upgraded to version ~> 5.0
• Hashicorp Helm Provider: Resolved breaking change compatibility with version 3.0 (applies to versions <= 2.17.0)

New components from earlier releases

If you are upgrading from before v5.19, the Cinchy MCP Server (cinchy-mcp) is a new component, see New Component: Cinchy MCP Server. If you are upgrading from before v5.20, the Cinchy Notification Service (cinchy-notification) is a new component, see Notification Service Setup. Both components are optional.

Breaking Changes

Argo CD deployment command has changed

deploy_argocd.sh must now run with --server-side --force-conflicts. The regenerated script includes these flags. If you maintain a customized copy of the script, you must add them yourself, otherwise the deployment fails. See Upgrade Argo CD to v3.5.2.

warning

If upgrading from before v5.17.2, the Kafka 4.0.0 upgrade introduces breaking changes that require a complete Kafka cluster redeployment. You must remove the existing Kafka operator and Kafka cluster before proceeding, see Remove Existing Kafka Components.

If you are upgrading from v5.17.2 or later, these breaking changes do not apply.

Supported Kubernetes Versions

• Amazon Elastic Kubernetes Service (EKS): 1.34
• Azure Kubernetes Service (AKS): 1.36.3

note

The patch version (the third number, e.g., .3 in 1.36.3) for AKS may differ from what is listed above. Azure controls when patch releases are made available, and the exact version in your region may be slightly ahead or behind. Use the closest available patch for your target minor version.

Prerequisites

info

If you have made custom changes to your deployment file structure, contact Support before upgrading your environments.

• Download the latest Cinchy artifacts from the Cinchy Releases Table > Kubernetes Artifacts column. For this upgrade, download Cinchy Kubernetes Deployment Template v5.23.0.zip.

Depending on your current version, you may also need to run one or both of the following upgrade scripts:

• 5.2 Upgrade Script
• 5.5 Upgrade Script

Current Version	Run the 5.2 Upgrade Script	Run the 5.5 Upgrade Script
5.0	Yes	Yes
5.1	Yes	Yes
5.2	X	Yes
5.3	X	Yes
5.4	X	Yes
5.5	X	X
5.6	X	X
5.7	X	X
5.8	X	X
5.9	X	X
5.10	X	X
5.11	X	X
5.12	X	X
5.13	X	X
5.14	X	X
5.15	X	X
5.16	X	X
5.17	X	X
5.18	X	X
5.19	X	X
5.20	X	X
5.21	X	X
5.22	X	X

Configure for the Latest Version

Clean Existing Repositories

1. Navigate to your cinchy.argocd repository and remove all existing folder structures except for the .git folder and any custom modifications you have made.

2. Navigate to your cinchy.kubernetes repository and remove all existing folder structures except for the .git folder.

   caution

   If cinchy.kubernetes\cluster_components\servicemesh\istio\istio-injection\argocd-ns.yaml exists and is not commented out, do not modify it. Doing so will delete your ArgoCD namespace, requiring a full removal of all Kubernetes resources and redeployment.

3. Navigate to your cinchy.terraform repository and remove all existing folder structures except for the .git folder.

4. Navigate to your cinchy.devops.automations repository and remove all existing folder structures except for the .git folder and your deployment.json configuration file.

Extract Kubernetes Template

1. Extract Cinchy Kubernetes Deployment Template v5.23.0.zip and copy the contents into their respective repositories: cinchy.kubernetes, cinchy.argocd, cinchy.terraform, and cinchy.devops.automations.

Configure Secrets

v5.23 does not introduce any new secrets.

1. If your environments were not previously configured with Azure Key Vault or AWS Secrets Manager and you are enabling them during this upgrade, ensure the required secrets are created. Otherwise, no action is needed.

   • Create Secrets in AWS
   • Create Secrets in Azure

Update Deployment Configuration

note

Repeat these steps each time you perform an AKS/EKS version upgrade using the procedure below.

1. Compare the new aws.json / azure.json configuration files with your existing deployment.json and incorporate any additional fields. Update the component image tags to v5.23.0.

2. If upgrading AKS/EKS, update the Kubernetes version in your deployment.json. To upgrade AKS/EKS to the versions supported by this release, follow the sequential upgrade path, advancing through each minor version incrementally. For example, upgrading AKS from 1.34 requires sequential upgrades through 1.35 and then 1.36.

3. Open a terminal from the cinchy.devops.automations directory and run:

   dotnet Cinchy.DevOps.Automations.dll "deployment.json"

   This regenerates deploy_argocd.sh with the --server-side --force-conflicts flags required by Argo CD v3.5.2.

Remove Existing Kafka Components

info

Skip this section if you are upgrading from v5.17.2 or later. Kafka 4.0.0 was introduced in v5.17.2 and the removal was already performed during that upgrade.

warning

Due to breaking changes in Kafka 4.0.0, you must remove the existing Kafka operator and cluster before proceeding.

This process requires downtime. Plan your upgrade accordingly and perform this step during a maintenance window.

info

This is a one-time step. You only need to remove the Kafka components once, during your first upgrade that includes Kafka 4.0.0, regardless of how many Kubernetes version hops you perform.

1. Remove the Kafka operator:

   kubectl delete app strimzi-kafka-operator -n argocd

   Expected output:

   application.argoproj.io "strimzi-kafka-operator" deleted

2. Remove the Kafka cluster:

   kubectl delete app kafka-cluster -n argocd

   Expected output:

   application.argoproj.io "kafka-cluster" deleted

3. Verify that both strimzi-kafka-operator and kafka-cluster have been removed from ArgoCD:

   kubectl get app -n argocd

   Confirm that neither application appears in the output.

Upgrade Argo CD to v3.5.2

v5.23 upgrades Argo CD from v2.7.6 to v3.5.2. Perform this before deploying cluster and Cinchy components.

Why this is required

Argo CD v2.7.6 embeds a Kubernetes OpenAPI schema that predates .status.terminatingReplicas, a field added to Deployment and ReplicaSet status in Kubernetes 1.33. On clusters running 1.33 or later, the API server populates that field unconditionally, so Argo CD cannot build a typed value from the live resource:

ComparisonError: error calculating structured merge diff: error building typed
value from live resource: .status.terminatingReplicas: field not declared in schema

Every Application that combines ServerSideApply=true with a Deployment in its resource tree is left at Sync Status Unknown, which silently disables drift detection, selfHeal, and prune. In a standard Cinchy deployment this affects kube-prometheus-stack, keda, and logging-operator. Health reporting continues to show these Applications as Healthy, so the problem is easy to overlook.

The schema is compiled into the Argo CD binary, so upgrading Argo CD is the only remedy.

Server-side apply is required

Argo CD v3.x CRDs exceed the 262144-byte limit on the kubectl.kubernetes.io/last-applied-configuration annotation that client-side apply writes (applications.argoproj.io is roughly 412 KB and applicationsets.argoproj.io roughly 1.4 MB). Without --server-side, the deployment fails with:

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes

This applies to fresh installs as well as upgrades, because kubectl writes that annotation on create as well as on update.

Procedure

1. Confirm you are targeting the correct cluster:

   kubectl config current-context

2. Back up your Argo CD Applications and configuration:

   kubectl -n argocd get applications,appprojects -o yaml > argocd-apps-backup.yaml
   kubectl -n argocd get cm,secret -o yaml > argocd-config-backup.yaml

3. Apply the upgrade:

   bash deploy_argocd.sh

   note

   deploy_argocd.sh is generated by cinchy.devops.automations and now contains kubectl apply -k argocd --server-side --force-conflicts. If you are running a customized copy of this script, add those flags before running it.

   You may see warnings of the form failed to migrate kubectl.kubernetes.io/last-applied-configuration for Server-Side Apply. These are non-fatal, the objects are still applied.

4. Verify that all Argo CD pods are up and running:

   kubectl get pods -n argocd

5. Open the Argo CD UI and confirm you can sign in and view all of your applications as expected.

Rollback

The v2.7.6 manifest is retained in the cinchy.argocd repository. To roll back, point argocd/kustomization.yaml at non-ha-install-v2.7.6.yaml and re-run:

kubectl apply -k argocd --server-side --force-conflicts

Note that rolling back to v2.7.6 reintroduces the terminatingReplicas comparison failure on Kubernetes 1.33 and later.

Commit and Deploy

1. Commit all changes to Git across the relevant repositories.

2. Redeploy Argo CD if you have not already done so as part of Upgrade Argo CD to v3.5.2. Launch a terminal from the root of the cinchy.argocd repository and run:

   bash deploy_argocd.sh

3. Verify that all ArgoCD pods are running successfully.

4. Deploy or update cluster components:

   bash deploy_cluster_components.sh

5. Deploy or update Cinchy components (Required):

   bash deploy_cinchy_components.sh

6. Refresh applications in the ArgoCD console as needed.

7. All users must log out and log back in to the Cinchy environment for changes to take effect.

Post-Upgrade Verification

1. Confirm all Cinchy pods are running:

   kubectl get pods -n <cinchy-namespace>

2. Confirm no Argo CD Application is in an Unknown sync state:

   kubectl get applications -n argocd -o wide

3. Open your Cinchy URL in a browser and confirm the version reads v5.23 under System Information.

Kafka UI Connectivity Issues

After upgrading, the Kafka UI pod may retain stale connections to the previous cluster configuration, resulting in connection failures or missing topics and messages.

1. Verify the status of all Kafka pods:

   kubectl get pods -n kafka

2. If the Kafka UI cannot connect to the upgraded cluster, restart the pod to establish a fresh connection:

   kubectl delete pod -n kafka -l app.kubernetes.io/name=kafka-ui

3. Kubernetes will automatically recreate the pod. Verify it is running:

   kubectl get pods -n kafka -l app.kubernetes.io/name=kafka-ui

Upgrade AWS EKS and Azure AKS

The following procedure upgrades AWS EKS to 1.34 or Azure AKS to 1.36.3.

Upgrade one minor version at a time

Both EKS and AKS must be upgraded sequentially through each minor version. For example, upgrading AKS from 1.34 to 1.36 requires going through 1.35 first. Do not skip minor versions.

Prerequisites

• AWS: Export the required credentials before proceeding.
• Azure: Run az login to authenticate, if required.

Procedure

Your deployment.json is the copy you created from the template azure.json (Azure) or aws.json (AWS) in the cinchy.devops.automations repository. The Kubernetes version is set in that file, and the Terraform configuration is generated from it, so you update the version there rather than editing the Terraform files directly.

Repeat all of the steps below once per minor version until you reach your target version.

1. Update the Kubernetes version in your deployment.json:

   • Azure: set kubernetes_version and orchestrator_version under the aks section to the next minor version (for example 1.35.x).
   • AWS: set cluster_version under the eks section to the next minor version (for example 1.35).

   Set these to the next minor version, not your final target. For example, going from 1.34 to 1.36 means running this procedure once for 1.35 and again for 1.36.

2. Regenerate the deployment from cinchy.devops.automations:

   dotnet Cinchy.DevOps.Automations.dll "deployment.json"

   This writes the updated Kubernetes version into the Terraform configuration in cinchy.terraform.

3. Change into the Terraform directory for your cluster:

   • AWS: the eks_cluster folder under Terraform > AWS, in the subdirectory named after the cluster.
   • Azure: the aks_cluster folder under Terraform > Azure, in the subdirectory named after the cluster.

4. Initiate the upgrade. Review the proposed changes carefully, then confirm with yes to proceed. This performs an in-place update of the Kubernetes version without deleting or destroying data.

   bash create.sh

5. Wait for the cluster upgrade to finish and confirm your nodes and pods are healthy before starting the next minor version.

warning

Before confirming, verify that the proposed changes meet your expectations and protect your database and other critical resources. This command may create, update, or destroy virtual networks, subnets, AKS/EKS clusters, and node groups. Review all changes thoroughly before proceeding.

Argo CD and Kubernetes 1.33+

If you are upgrading your cluster to Kubernetes 1.33 or later, ensure the Argo CD v3.5.2 upgrade is completed. On 1.33+ an Argo CD v2.7.6 control plane cannot compute a diff for Applications that use server-side apply, leaving them stuck at Sync Status Unknown.