Skip to main content

5.20 (Kubernetes)

Release Notes​

v5.20 introduces a new platform component, updates MCP secret management, and requires changes to your deployment.json configuration:

  • Cinchy Notification Service: A new cinchy-notification component is added to the platform, providing transactional email delivery. On AWS it is backed by Amazon SES; on Azure it is backed by Azure Communication Services (ACS). A new notification_enabled flag controls whether the service and all associated cloud resources (SES/ACS, IRSA, ArgoCD app, VirtualService route) are deployed — set it to "false" to skip the notification service entirely. See New Component: Cinchy Notification Service for deployment details.
  • Cinchy MCP (Model Context Protocol) Server: The mcp-app component is updated to use the standard appsettings secret pattern (K8s Secret volume mount). If you use AWS Secrets Manager or Azure Key Vault, a new secret must be created for MCP — see Configure Secrets below. See Cinchy MCP — Secret Management for deployment details.

If you are upgrading from a version earlier than v5.17.2, this release also carries forward the following infrastructure changes first introduced in v5.17:

  • Kafka Cluster: Upgraded to version 4.0.0
  • Strimzi Kafka Operator: Upgraded to version 0.49.0
  • Amazon Machine Image (AMI): AL2023_x86_64_STANDARD support updated for EKS 1.33
  • Hashicorp AWS Provider: Upgraded to version ~> 5.0
  • Hashicorp Helm Provider: Resolved breaking change compatibility with version 3.0 (applies to versions <= 2.17.0)

Breaking Changes​

warning

If upgrading from before v5.17.2, the Kafka 4.0.0 upgrade introduces breaking changes that require a complete Kafka cluster redeployment. You must remove the existing Kafka operator and Kafka cluster before proceeding — see Remove Existing Kafka Components.

If you are upgrading from v5.17.2 or later, these breaking changes do not apply.

Supported Kubernetes Versions​

  • Amazon Elastic Kubernetes Service (EKS): 1.33 and 1.34
  • Azure Kubernetes Service (AKS): 1.33.7 and 1.34.3
note

The patch version (the third number — e.g., .5 in 1.33.5) for AKS may differ from what is listed above. Azure controls when patch releases are made available, and the exact version in your region may be slightly ahead or behind. Use the closest available patch for your target minor version.

Prerequisites​

info

If you have made custom changes to your deployment file structure, contact Support before upgrading your environments.

  • Download the latest Cinchy artifacts from the Cinchy Releases Table > Kubernetes Artifacts column. For this upgrade, download Cinchy Kubernetes Deployment Template v5.20.0.zip.

Depending on your current version, you may also need to run one or both of the following upgrade scripts:

Current VersionRun the 5.2 Upgrade ScriptRun the 5.5 Upgrade Script
5.0YesYes
5.1YesYes
5.2XYes
5.3XYes
5.4XYes
5.5XX
5.6XX
5.7XX
5.8XX
5.9XX
5.10XX
5.11XX
5.12XX
5.13XX
5.14XX
5.15XX
5.16XX
5.17XX
5.18XX
5.19XX

Configure for the Latest Version​

Clean Existing Repositories​

  1. Navigate to your cinchy.argocd repository and remove all existing folder structures except for the .git folder and any custom modifications you have made.

  2. Navigate to your cinchy.kubernetes repository and remove all existing folder structures except for the .git folder.

    caution

    If cinchy.kubernetes\cluster_components\servicemesh\istio\istio-injection\argocd-ns.yaml exists and is not commented out, do not modify it. Doing so will delete your ArgoCD namespace, requiring a full removal of all Kubernetes resources and redeployment.

  3. Navigate to your cinchy.terraform repository and remove all existing folder structures except for the .git folder.

  4. Navigate to your cinchy.devops.automations repository and remove all existing folder structures except for the .git folder and your deployment.json configuration file.

Remove Existing Kafka Components​

info

Skip this section if you are upgrading from v5.17.2 or later. Kafka 4.0.0 was introduced in v5.17.2 and the removal was already performed during that upgrade.

warning

Due to breaking changes in Kafka 4.0.0, you must remove the existing Kafka operator and cluster before proceeding.

This process requires downtime. Plan your upgrade accordingly and perform this step during a maintenance window.

info

This is a one-time step. You only need to remove the Kafka components once — during your first upgrade that includes Kafka 4.0.0, regardless of how many Kubernetes version hops you perform.

For example, if you are upgrading your Kubernetes cluster from 1.32 → 1.33 → 1.34, perform this Kafka removal only when upgrading to 1.33. When you subsequently upgrade to 1.34, skip this section entirely.

  1. Remove the Kafka operator:

    kubectl delete app strimzi-kafka-operator -n argocd

    Expected output:

    application.argoproj.io "strimzi-kafka-operator" deleted

  2. Remove the Kafka cluster:

    kubectl delete app kafka-cluster -n argocd

    Expected output:

    application.argoproj.io "kafka-cluster" deleted

  3. Verify that both strimzi-kafka-operator and kafka-cluster have been removed from ArgoCD:

    kubectl get app -n argocd

    Confirm that neither application appears in the output.

Extract Kubernetes Template​

  1. Extract Kubernetes Deployment Template v5.20.0.zip and copy the contents into their respective repositories: cinchy.kubernetes, cinchy.argocd, cinchy.terraform, and cinchy.devops.automations.

Configure Secrets​

  1. If your environments use AWS Secrets Manager or Azure Key Vault, two new secrets must be created for the components added or updated in v5.20:

    ComponentSecret name (both AWS SM and Azure KV)
    Cinchy MCPmcp-secret-appsettings-<cinchy_instance_name>
    Cinchy Notification Servicenotification-secret-appsettings-<cinchy_instance_name>

    AWS Secrets Manager — create both secrets and note the ARN returned for each:

    aws secretsmanager create-secret \
      --name mcp-secret-appsettings-<cinchy_instance_name> \
      --secret-string '{"mcpsecretappsettings":"{}"}' \
      --region <your-region>

    aws secretsmanager create-secret \
      --name notification-secret-appsettings-<cinchy_instance_name> \
      --secret-string '{"notificationsecretappsettings":"{}"}' \
      --region <your-region>

    Add both ARNs to the aws_secrets_manager_arn_list array in your deployment.json.

    Azure Key Vault — the two app-level secrets (mcp-secret-appsettings-<cinchy_instance_name> and notification-secret-appsettings-<cinchy_instance_name>) are created and updated automatically by the DevOps pipeline. No manual creation is required for these.

    Upgraders: add notificationpassword to your existing environment-settings secret

    The notification@cinchy.com service account requires a notificationpassword entry that does not exist in environment-settings secrets created before v5.20. Add it before running the upgrade:

    AWS Secrets Manager — append the new key to the existing cinchy-environment-settings-<cinchy_instance_name> secret:

    SECRET_ID=cinchy-environment-settings-<cinchy_instance_name>
    EXISTING=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID --query SecretString --output text)
    UPDATED=$(echo $EXISTING | jq '. + {"notificationpassword":"<new-unique-password>"}')
    aws secretsmanager update-secret --secret-id $SECRET_ID --secret-string "$UPDATED"

    Azure Key Vault — add a new individual secret:

    az keyvault secret set --vault-name <key_vault_name> \
      --name notificationpassword-<cinchy_instance_name> \
      --value "<new-unique-password>"
    Fresh install only

    On a fresh Azure Key Vault install (not an upgrade from a previous version), the pipeline also creates the seven cinchy-environment-settings individual secrets (encryptionkey-<instance>, connectionspassword-<instance>, etc.) automatically. If you are performing a manual fresh install, see Enable Azure Key Vault secrets for the full creation commands.

    The encryptionkey-<instance> value is generated once and must not be changed after the database is initialized. The service account password secrets can be rotated per your password policy.

    Ensure the AKS managed identity has get, list, and set permissions on the Key Vault before running the pipeline.

    If you are not using AWS Secrets Manager or Azure Key Vault, no action is needed — Kubernetes Secrets are managed directly.

    For full secrets setup instructions, see:

Update Deployment Configuration​

  1. Compare the new aws.json / azure.json configuration files with your existing deployment.json and incorporate any additional fields including the new Notification Service fields described in New Component: Cinchy Notification Service and the MCP fields described in Cinchy MCP — Secret Management below.

  2. Open a terminal from the cinchy.devops.automations directory and run:

    dotnet Cinchy.DevOps.Automations.dll "deployment.json"

Commit and Deploy​

  1. Commit all changes to Git across the relevant repositories.

  2. If changes were made to your cinchy.argocd repository, redeploy ArgoCD. Launch a terminal from the root of the cinchy.argocd repository and run:

    bash deploy_argocd.sh

  3. Verify that all ArgoCD pods are running successfully.

  4. Deploy or update cluster components:

    bash deploy_cluster_components.sh

  5. Deploy or update Cinchy components:

    bash deploy_cinchy_components.sh

    This deploys the new cinchy-notification and updated mcp-app components alongside all existing platform components.

  6. Refresh applications in the ArgoCD console as needed.

New Component: Cinchy Notification Service​

v5.20 adds cinchy-notification as a new platform component for transactional email delivery. It is deployed by default alongside all existing platform components. The setup steps differ by cloud provider.

Auto-provisioned on upgrade

The v5.20 model upgrade automatically creates the objects the service depends on — the notification@cinchy.com user, the cinchy_notification integrated client, the Notifications Enabled System Property (global kill switch), and the Notification Outbox, Notification Config, and Notification Branding tables. No manual setup of these objects is required. For configuration and usage, see the Email Notification Service guide.

  • AWS: Email is sent via Amazon Simple Email Service (SES). Terraform creates the SES domain identity and an IAM send policy. After terraform apply, you must create the IRSA binding for the notification-sa Kubernetes ServiceAccount using eksctl, copy the SES policy ARN back into your deployment.json, and re-run the DevOps Automations tool before committing — see Step 2 for the commands.
  • Azure: Email is sent via Azure Communication Services (ACS). Terraform provisions the ACS resource and outputs a connection string and endpoint. You must copy those values back into your deployment.json and re-run the DevOps Automations tool, which generates the Kubernetes Secret containing the ACS configuration.

Notification Service Setup​

Optional

The Email Notification Service is optional. If you do not plan to use email notifications, you can skip this entire section. To disable it, set notification_enabled to "false" in your deployment.json and the service will not be deployed.

Email is sent via Amazon Simple Email Service (SES). Terraform creates the SES domain identity and an IAM send policy. After terraform apply you must create the IRSA binding, paste the policy ARN back into deployment.json, and re-run the DevOps Automations tool.

Step 1: Configure deployment.json (AWS)​

Under terraform_config, add a notification block:

"notification": {
  "notification_enabled": "true",
  "notification_provider_type": "AwsSes",
  "notification_email_domain": "<your-verified-ses-domain>",
  "cinchy_ses_send_policy": "",
  "notification_from_address": "noreply@<your-verified-ses-domain>"
}
FieldDescription
notification_enabled"true" deploys the service and provisions SES, IRSA, and all associated resources. "false" skips it entirely — no cloud resources, no ArgoCD app, no VirtualService route.
notification_provider_typeEmail backend. Default: "AwsSes". See Choose your email provider below to switch to SMTP or Microsoft Graph.
notification_email_domainDomain to verify in SES for outbound email. Must be a domain you control.
cinchy_ses_send_policyLeave empty — populated from Terraform output in Step 2.
notification_from_addressSender address in the From header. Must be within the verified SES domain.

Choose your email provider​

No additional fields required. The notification_provider_type: "AwsSes" above is sufficient. The service uses IRSA for authentication — no static credentials needed.

For full provider configuration details, see Provider setup.

Under image_repo_uris, add:

"notification_image_repo_uri": "<ecr-registry>/cinchy.notification"

Under each cinchy_instance_configs.<instance_name>, add:

"notification_image_tag": "v5.20.0"

Under cinchy_instance_configs.<instance_name>.scaling_parameters, add:

"notification": {
  "notification_cpu_requests": "250m",
  "notification_cpu_limits": "1",
  "notification_memory_requests": "512Mi",
  "notification_memory_limits": "1Gi",
  "notification_hpa_minimum_replicas": "1",
  "notification_hpa_maximum_replicas": "1"
}

Step 2: Apply Terraform and wire IRSA (AWS)​

  1. Run the DevOps Automations tool to generate the Terraform configuration:

    dotnet Cinchy.DevOps.Automations.dll "deployment.json"

  2. From the cluster's Terraform directory (cinchy.terraform/aws/eks_cluster/<cluster-name>/), run:

    terraform init
    terraform apply

  3. Retrieve the SES send policy ARN:

    terraform output -raw CLUSTER_NAME_cinchy_ses_send_policy

  4. Create the IRSA binding for the notification-sa ServiceAccount:

    eksctl create iamserviceaccount \
      --name notification-sa \
      --namespace <cinchy-namespace> \
      --cluster <cluster-name> \
      --role-name "notification-sa-<cluster-name>" \
      --attach-policy-arn <ses-send-policy-arn> \
      --approve \
      --override-existing-serviceaccounts \
      --region <aws-region>

  5. Paste the ARN back into deployment.json:

    • terraform_config.notification.cinchy_ses_send_policy — paste the SES send policy ARN

  6. Re-run the DevOps Automations tool:

    dotnet Cinchy.DevOps.Automations.dll "deployment.json"

  7. Commit and push all changes across cinchy.devops.automations, cinchy.kubernetes, cinchy.argocd, and cinchy.terraform.

Step 3: Publish DNS records (AWS)​

  1. Retrieve the SES DNS verification records:

    terraform output ses_dkim_records

  2. Publish all returned records in your DNS provider:

    • Three DKIM CNAME records for email signing
    • A TXT record for domain verification (_amazonses.<your-domain>)
    • An MX record and SPF TXT record for the MAIL FROM domain

  3. Add a DMARC record (start permissive):

    TypeNameValue
    TXT_dmarc.<your-domain>v=DMARC1; p=none;

  4. Wait for SES to show DKIM status: Verified. DNS propagation can take up to 72 hours.

note

If your domain is already verified in SES, set notification_email_domain to the already-verified domain and skip this step.

Step 4: Verify (AWS)​

Confirm the notification pod is running:

kubectl get pods -n <cinchy-namespace> -l app=notification-app

Expected output:

NAME                            READY   STATUS    RESTARTS   AGE
notification-app-<hash>         1/1     Running   0          2m

Cinchy MCP — Secret Management​

The mcp-app component is updated in v5.20 to store its configuration (BaseUrl, IdentityServerUrl, logging settings) in a Kubernetes Secret mounted as appsettings.json, matching the standard pattern used by other Cinchy components. The DevOps Automations tool generates this secret automatically from your deployment.json values.

No additional deployment.json fields are required beyond what was already present. Ensure the following fields exist in your configuration:

Under image_repo_uris:

"mcp_image_repo_uri": "<ecr-or-acr-registry>/cinchy.mcp"

Under each cinchy_instance_configs.<instance_name>:

"mcp_image_tag": "v5.20.0"

Under cinchy_instance_configs.<instance_name>.scaling_parameters:

"mcp": {
  "mcp_cpu_requests": "200m",
  "mcp_cpu_limits": "1",
  "mcp_memory_requests": "512Mi",
  "mcp_memory_limits": "1Gi",
  "mcp_hpa_minimum_replicas": "1",
  "mcp_hpa_maximum_replicas": "3"
}

AWS Secrets Manager — MCP​

If your environment uses AWS Secrets Manager, add the mcp-secret-appsettings-<cinchy_instance_name> ARN to aws_secrets_manager_arn_list in your deployment.json (see Configure Secrets for the creation command). The Terraform IRSA policy references this ARN to grant the mcp-sa ServiceAccount read access.

Azure Key Vault — MCP​

If your environment uses Azure Key Vault, the mcp-secret-appsettings-<cinchy_instance_name> secret is created and updated automatically by the DevOps pipeline on each run. No manual creation is required. Ensure the AKS managed identity has get and set permissions on the Key Vault before running the pipeline.

WAF — MCP path exclusion (AWS)​

If your environment uses AWS WAF with the Anonymous IP List managed rule, the /mcp path must be excluded from that rule. Without this exclusion, MCP SSE (Server-Sent Events) and JSON-RPC traffic is blocked for clients that route through anonymizing proxies or VPNs.

The Cinchy WAF Terraform module (stacks/catalog/waf/defaults.yaml) includes this exclusion by default from v5.20 onwards. If you manage your WAF configuration separately, add a scope-down NOT statement on STARTS_WITH /mcp to the Anonymous IP List rule.

Verify the MCP pod​

Confirm the MCP pod is running:

kubectl get pods -n <cinchy-namespace> -l app=mcp-app

Expected output:

NAME                      READY   STATUS    RESTARTS   AGE
mcp-app-<hash>            1/1     Running   0          2m

Confirm the health endpoint responds:

curl https://<your-host>/mcp/health
# Expected: Healthy

Limitations and Known Issues​

AWS SES: Sandbox mode​

New AWS accounts and newly created SES identities start in SES sandbox mode. In sandbox mode:

  • Emails can only be sent to and from verified email addresses or domains. Any attempt to send to an unverified recipient will fail with an error such as:

    Email address is not verified. The following identities failed the check in region US-EAST-1: user@example.com

  • The daily sending quota and per-second sending rate are also restricted.

To send email to any recipient, you must request SES production access.

Requesting production access​

  1. In the AWS Console, navigate to Amazon SES > Account dashboard.
  2. Under Sending limits, click Request production access.
  3. Complete the request form. Provide:
    • Your use case (transactional notifications for a SaaS platform)
    • An estimate of daily send volume
    • How you handle bounces and complaints (e.g., via SES feedback forwarding or SNS topics)
  4. AWS typically reviews requests within 24 hours.
note

While waiting for production access, you can verify individual recipient addresses under Amazon SES > Configuration > Identities > Create identity > Email address to test end-to-end delivery without leaving sandbox mode.

AWS SES: SMTP configuration is not required​

The Notification Service configuration includes an Smtp block (host, port, credentials). This block is only used when Smtp.Host is populated. If left empty, the service ignores SMTP entirely.

Provider selection is automatic:

notification_provider_type valueProvider used
"AwsSes" (default on AWS)Amazon SES via IRSA
"AzureCommunication" (default on Azure)Azure Communication Services
"Smtp"SMTP relay
"MicrosoftGraph"Microsoft Graph API

If your deployment uses IRSA for SES (the default AWS path), leave all Smtp.* fields empty.

Azure ACS: Custom domain DNS verification required before sending​

When notification_email_domain is set in your deployment.json, Terraform registers a custom domain in Azure Communication Services. ACS will not send email from that domain until all DNS records are verified. This is a hard requirement enforced by Microsoft — email attempts before verification will fail silently or return a 400 Bad Request from the ACS API.

The records Terraform outputs via terraform output acs_dkim_records must be published at your DNS provider:

Record typePurposeWhere to verify
CNAME (×2)DKIM signing keysACS will show "Verified" when both are live
TXTSPF authorisationTypically v=spf1 include:spf.protection.outlook.com -all or ACS-specific value

DNS propagation can take up to 72 hours. Until both DKIM records show Verified in the Azure Portal (Azure Communication Services > Email > Domains), outbound email will be rejected.

caution

If you use a domain that is also used by another mail sender (Office 365, Google Workspace, etc.), merging SPF records is required. Adding a second v=spf1 TXT record on the same hostname will break both. Append ACS's include: to your existing SPF record instead.

If notification_email_domain is left empty, ACS uses the shared azurecomm.net sender domain — no DNS configuration is needed, but the from address will be noreply@<resource>.azurecomm.net.

From Address Override scope​

The From Address Override column in [Cinchy].[Notification Config] only takes effect when an outbox entry is linked to a specific Data Sync Config, Listener Config, or Automation. The notification service resolves the from address by looking up the matching Notification Config row for the linked config.

If an outbox entry has no linked config (for example, a manually inserted test row), the service falls back to the DefaultFromAddress value in the notification service appsettings.json. In that case, updating the Notification Config table has no effect — the DefaultFromAddress in the Kubernetes Secret must be updated instead.

Practical impact: During initial testing, if you insert a row directly into [Cinchy].[Notification Outbox] without linking a Data Sync Config, ensure the DefaultFromAddress in the notification configuration matches a verified sender in your email provider (SES identity or ACS sender username).