Overview

In Cinchy v5.6, you are now able to run the Connections pod under a service account that uses an AWS IAM (Identity and Access Management) role, which is an IAM identity that you can create to have specific permissions and access to your AWS resources. To set up AWS IAM role authentication, please review the procedure below.

AWS IAM role authentication

1. To check that you have an OpenID Connect set up with the cluster (the default for deployments made using the Cinchy automation process), run the below command within a terminal:

aws eks describe-cluster --name <CLUSTER_NAME> --query "cluster.identity.oidc.issuer"

1. The output should appear like the below. Make sure to note this down for later use.

https://oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID>

1. Log in to your AWS account and create an IAM Role policy through the AWS UI. Ensure that it has S3 access.

2. Run the below command in a terminal to create a service account with the role created in step 3. If your cluster has a special character like an underscore, skip to the next section.

eksctl create iamserviceaccount --name my-service-account --namespace default --cluster my-cluster --role-name "my-role" --attach-policy-arn arn:aws:iam::111122223333:policy/my-policy --approve

Cluster names with special characters

If your cluster name has a special character, like an underscore, you will need to create and apply the YAML. Follow section 1 up until step 4, and then follow the below procedure.

1. In an IDE (Visual Studio, VsCode), create a new file titled my-service-account.yaml in your working directory. It should contain the below content.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: default

1. In a terminal, run the below command:

kubectl apply -f my-service-account.yaml -n <namespace>

1. In an IDE (Visual Studio, VsCode), create a new file titled trust-relationship.json in your working directory. It should contain the below content.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$account_id:oidc-provider/$oidc_provider"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$oidc_provider:aud": "sts.amazonaws.com",
          "$oidc_provider:sub": "system:serviceaccount:$namespace:$service_account"
        }
      }
    }
  ]
}

For example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::204393242335:oidc-provider/oidc.eks.ca-central-1.amazonaws.com/id/8A024CF24ADD3925BEFA224C4BDD005B"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.ca-central-1.amazonaws.com/id/8A024CF24ADD3925BEFA224C4BDD005B:aud": "sts.amazonaws.com",
          "oidc.eks.ca-central-1.amazonaws.com/id/8A024CF24ADD3925BEFA224C4BDD005B:sub": "system:serviceaccount:devops-aurora-1:connections-serviceaccount-s3"
        }
      }
    }
  ]
}

1. Execute the following command to create the role, referencing the above .json file:

aws iam create-role --role-name my-role --assume-role-policy-document file://trust-relationship.json --description "my-role-description"

For example:

aws iam create-role --role-name connections-role-test --assume-role-policy-document file://trust-relationship.json --description "testing sa role for pod"

1. Execute the following command to attach the IAM policy to your role:

aws iam attach-role-policy --role-name my-role --policy-arn=arn:aws:iam::$account_id:policy/my-policy

For example:

aws iam attach-role-policy --role-name connections-role-test --policy-arn=arn:aws:iam::aws:policy/AmazonS3FullAccess

1. Execute the following command to annotate your service account with the Amazon Resource Name (ARN) of the IAM role that you want the service account to assume:

kubectl annotate serviceaccount -n $namespace $service_account eks.amazonaws.com/role-arn=arn:aws:iam::$account_id:role/my-role

For example:

kubectl annotate serviceaccount -n devops-aurora-1 connections-serviceaccount-s3 eks.amazonaws.com/role-arn=arn:aws:iam::204393242335:role/connections-role-test

1. Confirm that the role and service account are correctly configured by verifying the output of the following commands:

aws iam get-role --role-name my-role --query Role.AssumeRolePolicyDocument
aws iam get-role --role-name connections-role-test --query Role.AssumeRolePolicyDocument

Authorize for Data Syncs

To ensure that the Connections pod's role has the correct permissions, the role specified by the user in AWS must have its Trusted Relationships configured as such:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<role-arn-created-above>",
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Confirmation

To confirm that the Connections app is using the service account:

1. Navigate to the cinchy.kubernetes repository > connections/kustomization.yaml file

2. Execute the following:

- op: replace
      path: /spec/template/spec/serviceAccountName
      value: connections-serviceaccount-s3

1. From a terminal, run the below command:

kubectl describe deployment connections-app -n <namespace>

1. The output should look like the following:

Overview

In v5.2, Cinchy implemented the ability to free up database space by using S3 compatible or Azure Blob Storage for file storage. You can set this configuration in the deployment.json of a Kubernetes installation, or the appsettings.json of an IIS installation.

Kubernetes file storage

1. If you are using a Kubernetes deployment, you will change your file storage config in the deployment.json.

2. Navigate to the object storage section, where you will see either S3 or Azure Blob storage, depending on your deployment structure.

Azure Example

        "object_storage": {
          // Cinchy requires a new Azure Blob Storage container for it's file storage. Select a unique name, the template convention follows cinchy
          // Storage Account Names can only consist of lowercase letters and numbers, and must be between 3 and 24 characters long
          "storage_account_name": "cinchynonprod",
          // Two storage containers are created, one for the Connections component and one for the Platform. The default naming convention is -
          "connections_storage_container_name": "nonprod-connections",
          "platform_storage_container_name": "nonprod-platform",
          "connections_storage_type": "AzureBlobStorage",
          // The connection string to the Azure Blob Storage account, it can be retrieved by executing the following command after the terraform apply has completed
          // In the below command the  and  must be replaced with the values for those properties within this file
          // az storage account show-connection-string --name  --resource-group 
          "azure_blob_storage_conn_str": ""
        },

AWS Example

        "object_storage": {
          // Cinchy requires a new S3 bucket for it's file storage. Select a unique name, the template convention follows -
          "cinchy_s3_bucket": "-cinchy-nonprod",
          // During the S3 bucket creation, a tag named "Environment" is added to the resource and populated with the following value
          "cinchy_s3_environment_tag": "cinchy-nonprod",
          "connections_storage_type": "S3",
          // IAM user credentials (access key and secret) for access to this bucket. Ensure that the usre has the necessary privileges defined in IAM
          "connections_s3_access_key": "",
          "connections_s3_secret_access_key": "",
          // Optional - only set this value if you are using a third party S3 compatible service
          "connections_s3_service_url": ""
        },

1. To use Blob Storage or S3, update each line with your own parameters.

2. To use Local storage, leave each line blank except for the Connections_Storage_Type, which you should set to Local:

          "connections_storage_type": "Local",

5. Run the deployment script by using the following command in the root directory of your devops.automations repository:

dotnet Cinchy.DevOps.Automations.dll "deployment.json"

1. Commit and push your changes.

IIS file storage

1. If you are using an IIS deployment, you will change your file storage config in the Cinchy Web AppSettings file.

2. Locate the StorageType section of the file and set it to either Local, AzureBlobStorage, or S3.

  "AppSettings": {
   ...
    "StorageType": ""      // Set this to "Local" to store files within Cinchy's database. Set this to "AzureBlobStorage" to store files within Azure Blob Storage. Set this to "S3" to store files within S3.
  },

1. If you selected AzureBlobStorage, fill out the following lines in the same file:

  "AzureBlobStorageSettings": {
    "ConnectionString": "",    // ConnectionString used to connected to the Azure Blob Storage
    "Container": "",       // The container for Cinchy's file storage
    "BasePath": ""         // The base directory path of where to store files within the container (eg. cinchy/files)
  },

1. If you selected S3, fill out the following lines in the same file:

  "S3Settings": {
    "AccessKey": "",       // Access Key for the IAM user. Ensure that the user has the necessary privileges defined in IAM
    "SecretAccessKey": "", // Secret for the IAM user. Ensure that the user has the necessary privileges defined in IAM
    "Region": "",          // Region of where the S3 bucket is located in
    "Bucket": "",          // S3 bucket for Cinchy's file storage
    "BasePath": "",            // The base directory path of where to store files within the bucket (eg. cinchy/files)
    "ServiceURL": ""       // (Optional) - only set this value if you are using a third party S3 compatible service
  }

1. Execute the following commands in any folder to generate the self-signed certificate:

openssl genrsa -des3 -out rootCA.key 4096

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt

openssl genrsa -out mydomain.com.key 2048

openssl req -new -sha256 -key mydomain.com.key -subj "/C=US/ST=CA/O=MyOrg, Inc./CN=mydomain.com " -out mydomain.com.csr

1. Create a YAML file located at cinchy.kubernetes/platform_components/base/self-signed-ssl-root-ca.yaml.

2. Add the following to the YAML file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: self-signed-ca-pemstore
data:
  rootCA.crt: |
    <rootCA.crt>

1. Add the self signed root CA cert file to the cinchy.kubernetes/environment_kustomizations/cinchy_nonprod/base folder.

2. Add the yaml code snippet to the cinchy.kubernetes/environment_kustomizations/cinchy_nonprod/base/kustomization.yaml file, changing the below files key value as per your root ca cert file name:

configMapGenerator:
- name: self-signed-ca-pemstore
  behavior: replace
  files:
  - rootCA.crt

1. Add the following line to the cinchy.kubernetes/platform_components/base/kustomization.yaml file

- self-signed-ssl-root-ca.yaml

1. Add the below Deployment patchesJson6902 to each of your cinchy.kubernetes/environment_kustomizations/cinchy_nonprod/ENV_NAME/PLATFORM_COMPONENT_NAME/kustomization.yaml files, except base.

• Ensure that the rootCA.crt file name is matched with ConfigMap data, configMapGenerator files, and the patch subpath.

    - op: add
      path: /spec/template/spec/volumes/-
      value: 
        configMap:
          name: self-signed-ca-pemstore
        name: self-signed-ca-pemstore  
    - op: add
      path: /spec/template/spec/containers/0/volumeMounts/-
      value: 
        mountPath: /etc/ssl/certs/rootCA.crt
        name: self-signed-ca-pemstore
        subPath: rootCA.crt

1. Once the changes are deployed, verify the root CA cert is available on the pod under /etc/ssl/certs with below command. Make sure to input your own POD_NAME and NAMESPACE:

 kubectl exec -it POD_NAME -n NAMESPACE -- openssl x509 -in /etc/ssl/certs/rootCA.crt -text

Introduction

This page details the instructions for deployment of Cinchy v5 on Kubernetes. We recommend, and have documented below, that this is done via Terraform and ArgoCD. This setup involves a utility to centralize and streamline your configurations.

The Terraform scripts and instructions provided enable deployment on Azure and AWS cloud environments.

Deployment prerequisites

To install Cinchy v5 on Kubernetes, you need to follow the requirements below. Some requirements depend on whether you deploy on Azure or on AWS.

All platforms

These prerequisites apply whether you are installing on Azure or on AWS.

• You must create the following four Git repositories. You can use any source control platform that supports Git, such as GitLab, Azure DevOps, and GitHub.

  • cinchy.terraform:: Contains all Terraform configurations.

  • cinchy.argocd:: Contains all ArgoCD configurations.

  • cinchy.kubernetes:: Contains cluster and application component deployment manifests.

  • cinchy.devops.automations:: Contains the single configuration file and binary utility that maintains the contents of the above three repositories.

• Download the artifacts for the four Git repositories. See here for information on accessing these. Check the contents of each of the directories into the respective repository.

• You must have a service account with read/write permissions to the git repositories created above.

• Install the following tools on the deployment machine:

  • Terraform

    • For an introduction to Terraform + AWS, see this Get started Guide.

    • For an introduction to Terraform + Azure, see this Get started Guide

  • kubectl (v1.23.0+)

  • .NET Core 6 required for Cinchy v5.8 and higher.

  • Bash (Git Bash may be used on Windows)

• If you are using Cinchy docker images, pull them.

• You will need a single domain for accessing ArgoCD, Grafana, OpenSearch Dashboard, and any deployed Cinchy instances. You have two routing options for accessing these applications - path based or subdomains. See below for an example with multiple Cinchy instances:

• You will need an SSL certificate for the cluster. This should be a wildcard certificate if you will use subdomain based routing. You can also use Self-Signed SSL.

Azure requirements

If you are deploying Cinchy v5 on Azure, you require the following:

Terraform requirements

• A resource group that will contain the Azure Blob Storage with the terraform state.

• A storage account and container (Azure Blob Storage) for persisting terraform state.

• Install the Azure CLI on the deployment machine. It must be set to the correct profile/login

The deployment template has two options available:

• Use an existing resource group.

• Creating a new one.

Existing resource group

If you prefer an existing resource group, you must provision the following before the deployment:

• The resource group.

• A virtual network (VNet) within the resource group.

• A single subnet. It's important that the range be enough for all executing processes within the cluster, such as a CIDR ending with /22 to provide a range of 1024 addresses.

New resource group

• If you prefer a new resource group, all resources will be automatically provisioned.

• The quota limit of the Total Regional vCPUs and the Standard DSv3 Family vCPUs (or equivalent) must offer enough availability for the required number of vCPUs (minimum of 24).

• An AAD user account to connect to Azure, which has the necessary privileges to create resources in any existing resource groups and the ability to create a resource group (if required).

Kubernetes AWS requirements

If you are deploying Cinchy v5 on AWS, you require the following:

Terraform requirements:

• An S3 bucket that will contain the terraform state.

• Install the AWS CLI on the deployment machine. It must be set to the correct profile/login

The template has two options available:

• Use an existing VPC

• Create a new one.

Existing VPC

• If you prefer an existing VPC, you must provision the following before the deployment:

  • The VPC. It's important that the range be enough for all executing processes within the cluster, such as a CIDR ending with /21 to provide a range of 2048 IP addresses.

  • 3 Subnets (one per AZ). It's important that the range be enough for all executing processes within the cluster, such as a CIDR ending with /23 to provide a range of 512 IP addresses.

  • If the subnets are private, a NAT Gateway is required to enable node group registration with the EKS cluster.

New VPC

• If you prefer a new VPC, all resources will be automatically provisioned.

• The limit of the Running On-Demand All Standard vCPUs must offer enough availability for the required number of vCPUs (minimum of 24).

• An IAM user account to connect to AWS which has the necessary privileges to create resources in any existing VPC and the ability to create a VPC (if required).

• You must import the SSL certificate into AWS Certificate Manager (or a new certificate can be requested via AWS Certificate Manager).

• You must import the SSL certificate into AWS Certificate Manager, or a new certificate can be requested via AWS Certificate Manager.

• If you are importing it, you will need the PEM-encoded certificate body and private key. You can find this, you can get the PEM file from your chosen domain provider (GoDaddy, Google, etc.) Read more on this here.

Initial configuration

The following steps detail the instructions for setting up the initial configurations.

Configure the deployment.json file

1. Navigate to your cinchy.devops.automations repository where you will see an aws.json and azure.json.

2. Depending on platform that you are deploying to, select the appropriate file and copy it into a new file named deployment.json (or <cluster name>.json) within the same directory.

3. This file will contain the configuration for the infrastructure resources and the Cinchy instances to deploy. Each property within the configuration file has comments in-line describing its purpose along with instructions on how to populate it.

4. Follow the guidance within the file to configure the properties.

5. Commit and push your changes.

Execute cinchy.devops.automations

This utility updates the configurations in the cinchy.terraform, cinchy.argocd, and cinchy.kubernetes repositories.

1. From a shell/terminal, navigate to the cinchy.devops.automations directory location and execute the following command:

dotnet Cinchy.DevOps.Automations.dll "deployment.json"

1. If the file created in "Configuring the Deployment.json" step 2 has a name other than deployment.json, the reference in the command will will need to be replaced with the correct name of the file.

2. The console output should have the following message:

Completed successfully

Terraform deployment

The following steps detail how to deploy Terraform.

Cinchy.terraform repository structure - AWS

If deploying on AWS: Within the Terraform > AWS directory, a new folder named eks_cluster is created. Nested within that's a subdirectory with the same name as the newly created cluster.

To perform terraform operations, the cluster directory must be the working directory during execution. This applies to everything within step 4 of this guide.

Cinchy.terraform repository structure - Azure

If deploying on Azure: Within the Terraform > Azure directory, a new folder named aks_cluster is created. Nested within that's a subdirectory with the same name as the newly created cluster.

To perform terraform operations, the cluster directory must be the working directory during execution.

Cloud provider authentication

1. Launch a shell/terminal with the working directory set to the cluster directory within the cinchy.terraform repository.

2. If you are using AWS, run the following commands to authenticate the session:

export AWS_DEFAULT_REGION=REGION
export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=YOUR_ACCESS_KEY

1. For Azure, run the following command and follow the on screen instructions to authenticate the session:

az login

Deploy the infrastructure

1. Execute the following command to create the cluster:

bash create.sh

1. Type yes when prompted to apply the terraform changes.

The resource creation process can take about 15 to 20 minutes. At the end of the execution there will be a section with the following header

Output variables

If deploying on AWS, this section will contain 2 values: Aurora RDS Server Host and Aurora RDS Password

If deploying on Azure, this section will contain a single value: Azure SQL Database Password

These variable values are required to update the connection string within the deployment.json file (or equivalent) in the cinchy.devops.automations repository.

Retrieve the SSH keys

The following section breaks down how to retrieve your SSH keys for both AWS and Azure deployments.

AWS SSH keys

1. The SSH key to connect to the Kubernetes nodes is maintained within the terraform state and can be retrieved by executing the following command:

terraform output -raw private_key

Azure SSH keys

1. The SSH key is output to the directory containing the cluster terraform configurations.

Update the deployment.json

The following section pertains to updating the Deployment.json file.

Update the database connection string

1. Navigate to the deployment.json (created in step 3.1) > cinchy_instance_configs section.

2. Each object within represents an instance that will be deployed on the cluster. Each instance configuration has a database_connection_string property. This has placeholders for the host name and password that must be updated using output variables from the previous section.

Create the IAM user for S3 (AWS)

The terraform script will create an S3 bucket for the cluster that must be accessible to the Cinchy application components.

To access this programmatically, an IAM user that has read/write permissions to the new S3 bucket is required. This can be an existing user.

The Access Key and Secret Access Key for the IAM user must be specified under the object_storage section of the deployment.json

Update blob storage connection details (Azure)

1. Within the deployment.json, the azure_blob_storage_conn_str must be set.

2. The in-line comments outline the commands required to source this value from the Azure CLI.

Enable Azure Key Vault secrets

If you have the key_vault_secrets_provider_enabled=true value in the azure.json then the below secrets files would have been created during the execution of step 3.2:

You will need to add the following secrets to your Azure Key Vault:

• worker-secret-appsettings-<cinchy_instance_name>

• web-secret-appsettings-<cinchy_instance_name>

• maintenance-cli-secret-appsettings-<cinchy_instance_name>

• idp-secret-appsettings-<cinchy_instance_name>

• forms-secret-config-<cinchy_instance_name>

• event-listener-secret-appsettings-<cinchy_instance_name>

• connections-secret-config-<cinchy_instance_name>

• connections-secret-appsettings-<cinchy_instance_name>

To create your new secrets:

1. Navigate to your key vault in the Azure portal.

2. Open your Key Vault Settings and select Secrets.

3. Select Generate/Import.

4. On the Create a Secret screen, choose the following values:

   1. Upload options: Manual.

   2. Name: Choose the secret name from the above list. They will all follow the format of: <app>-secret-appsettings-<cinchy_instance_name> or <app>-secret-config-<cinchy_instance_name>

   3. Value: The value for the secret will be the content of each app JSON located in the cinchy.kubernetes\environment_kustomizations\nonprod<cinchy_instance_name>\secrets folder, and as shown in above image.

   4. Content type: JSON

5. Leave the other values to their defaults.

6. Select Create.

Once you receive the message that the first secret has been successfully created, you may proceed to create the other secrets. You must create a total of 8 secrets, as shown in the above list of secret names.

Execute cinchy.devops.automations

This utility updates the configurations in the cinchy.terraform, cinchy.argocd, and cinchy.kubernetes repositories.

1. From a shell/terminal, navigate to the cinchy.devops.automations directory and execute the following command:

dotnet Cinchy.DevOps.Automations.dll "deployment.json"

1. If the file created in section 3 has a name other than deployment.json, the reference in the command will will need to be replaced with the correct name of the file.

2. The console output should end with the following message:

Completed successfully

1. The updates must be committed to Git before proceeding to the next step.

Connect with kubectl

Update the Kubeconfig

AWS

1. From a shell/terminal run the following command, replacing <region> and <cluster_name> with the accurate values for those placeholders:

aws eks update-kubeconfig --region <region> --name <cluster_name>

Azure

1. From a shell/terminal run the following commands, replacing <subscription_id>, <deployment_resource_group>, and <cluster_name> with the accurate values for those placeholders.

az account set --subscription <subscription_id>
az aks get-credentials --admin --resource-group <deployment_resource_group> --name <cluster_name>

Verify the connection

1. Verify that the connection has been established and the context is the correct cluster by running the following command:

kubectl config get-contexts

Deploy and access ArgoCD

In this step, you will deploy and access ArgoCD.

Deploy ArgoCD

1. Launch a shell/terminal with the working directory set to the root of the cinchy.argocd repository.

2. Execute the following command to deploy ArgoCD:

bash deploy_argocd.sh

1. Monitor the pods within the ArgoCD namespace by running the following command every 30 seconds until they all move into a healthy state:

kubectl get pods -n argocd

Access ArgoCD

1. Launch a new shell/terminal with the working directory set to the root of the cinchy.argocd repository.

2. Execute the following command to access ArgoCD:

bash access_argocd.sh

This script creates a port forward using kubectl to enable ArgoCD to be accessed at http://localhost:9090

The credentials for ArgoCD's portal are output at the start of the access_argocd script execution in Base64. The Base64 value must be decoded to get the login credentials to use for the http://localhost:9090 endpoint.

Deploy cluster components

In this step, you will deploy your cluster components.

Deploy ArgoCD applications

1. Launch a shell/terminal with the working directory set to the root of the cinchy.argocd repository.

2. Execute the following command to deploy the cluster components using ArgoCD:

bash deploy_cluster_components.sh

1. Navigate to ArgoCD at http://localhost:9090 and login. Wait until all components are healthy (this may take a few minutes).

Update the DNS

1. Execute the following command to get the External IP used by the Istio ingress gateway.

kubectl get svc -n istio-system

1. DNS entries must be created using the External IP for any subdomains / primary domains that will be used, including OpenSearch, Grafana, and ArgoCD.

Access OpenSearch

The default path to access OpenSearch, unless you have configured it otherwise in your deployment.json, is <baseurl>/dashboard

Access Grafana

The default path to access Grafana, unless you have configured it otherwise in your deployment.json, is <baseurl>/grafana

Deploy Cinchy components

In this step, you will deploy your Cinchy components.

Deploy ArgoCD application

1. Launch a shell/terminal with the working directory set to the root of the cinchy.argocd repository.

2. Execute the following command to deploy the Cinchy application components using ArgoCD:

bash deploy_cinchy_components.sh

1. Navigate to ArgoCD at http://localhost:9090 and login. Wait until all components are healthy (may take a few minutes)

2. You will be able to access ArgoCD through the URL that you configured in your deployment.json, as long as you created a DNS entry for it in step 8.2.

## Troubleshooting

• If ArgoCD Application Sync is stuck waiting for PreSync jobs to complete, you can run the below command to restart the application controller.

kubectl rollout restart sts argocd-application-controller -n argocd

Disable your applications

There might be times when you want to temporarily disable your Kubernetes pods to perform maintenance or upgrades. You can do so through the following steps:

1. Access your ArgoCD.

2. Navigate to the application directory for the namespace you wish to disable, in this case development-cinchy (Image 1). You should see your cluster component applications.

1. Select the main application (development-cinchy) (Image 2).

1. Navigate to Summary > Sync Policy > Automated, then select Disable Auto-Sync > OK (Image 3).

1. For each of the cluster applications that you wish to disable, select the "..." > Delete (Image 5).

1. Your apps should all appear as "out of sync" (Image 6).

Re-enable your applications

1. To re-enable your applications, return to the application directory for your disabled namespace (Image 7).

1. Select the main application (i.e. development-cinchy) (Image 8).

1. Navigate to Summary > Sync Policy, then select Enable Auto-Sync > OK (Image 9).