Opensearch Dashboards
When deploying Cinchy v5 on Kubernetes, we recommend using Opensearch Dashboards for your logging. Opensearch is a community-driven fork of Elasticsearch created by Amazon, and it captures and indexes all of your logs into a single, easily accessible dashboard location. These logs can be queried, searched, and filtered, and Correlation IDs mean that they can also be traced across various components. These logging components take advantage of persistent storage.
Review some of the Opensearch documentation here:
The below sections will guide you through setting up your first Index, Visualization, Dashboard, and Alert.
Opensearch comes with sample data that you can use to get a feel of the various capabilities. You will find this on the main page upon logging in.
- 1.Navigate to your cinchy.kubernetes/environment_kustomizations/instance_template/worker/kustomization.yaml file.
- 2.In the below code, copy the Base64 encoded string in the value parameter.
patch: |-
- op: replace
path: /data/appsettings.json
value: wcxJItEmCWQJQPZidpLUuV6Ll79ZUr8BimlMJysLwcxJItEmCWQJQPZidpLUuV6Ll79ZUr8BimlMJysL
- 3.
- 4.Navigate to the below Serilog section of the code and update the "Default" parameter as needed to set your log level. The options are:
- 1.
VerboseVerbose is the noisiest level, rarely (if ever) enabled for a production app.DebugDebug is used for internal system events that are not necessarily observable from the outside, but useful when determining how something happened. This is the default setting for Cinchy.InformationInformation events describe things happening in the system that correspond to its responsibilities and functions. Generally these are the observable actions the system can perform.WarningWhen service is degraded, endangered, or may be behaving outside of its expected parameters, Warning level events are used.ErrorWhen functionality is unavailable or expectations broken, an Error event is used.FatalThe most critical level, Fatal events demand immediate attention.
"Serilog": {
"MinimumLevel": {
"Default": "Debug",
- 5.Ensure that you commit your changes.
- 6.Navigate to ArgoCD > Worker Application and refresh.
The following are some common search patterns when looking through your Opensearch Logs.
- If an HTTP request to Cinchy Web/IDP fails, check the page's requests and the relevant response headers to find the "x-correlation-id" header. That header value can be used to search and find all logs associated with the HTTP request.
- When debugging batch syncs, filter the "ExecutionId" field in the logs for your batch sync execution ID to narrow down your search.
- When debugging real time syncs, search for your data sync config name in the Event Listener or Workers logs to find all the associated logging information.
The first step to utilizing the power of Opensearch Dashboards is to set up an index to pull data from your sources. An Index Pattern identifies which indices you want to explore. An index pattern can point to a specific index, for example, your log data from yesterday, or all indices that contain your log data.
- 1.Login to Opensearch. You would have configured the access point during your deployment installation; traditionally it will be found at <baseurl>/dashboard.
If this is your first time logging in, the username and password will be set to admin/admin.
2. Navigate to the Stack Management tab in the left navigation menu (Image 1).
Image 1: Select Stack Management
3. From the left navigation, click on Index Patterns (Image 2).
Image 2: Select Index Patterns
4. Click on the Create Index Pattern button.
5. To set up your index pattern, you must define the source. Opensearch will list the sources available to you on the screen below. Input your desired source(s) in the text box (Image 3).
You can use the asterisk (*) to match multiple sources.
Image 3: Define your sources
6. Configure your index pattern settings (Image 4).
- Time field: Select a primary time field to use with the global time filter
- Custom index pattern ID: By default, Opensearch gives a unique identifier to each index pattern. You can use this field to optional override the default ID with a custom one.
Image 4: Configure your index pattern settings
7. Once created, you can review your Index Patterns from the Index Patterns page (Image 5).
Image 5: Review your Index Patterns
8. Click on your Index Pattern to review your fields (Image 6).
Image 6: Reviewing your Index Pattern fields
You can easily pull out any data from your index sources and view them in a variety of visualizations.
- 1.From the left navigation pane, click Visualize (Image 7).
Image 7: Click Visualize
2. If you have any Visualizations, they will appear on this page. To create a new one, click the Create Visualization button (Image 8).
Image 8: Click Create New
3. Select your visualization type from the populated list (Image 9).
Image 9: Select your Visualization type
4. Choose your source (Image 10). If the source you want to pull data from isn't listed, you will need to set it up as an index first.
Image 10: Select your Source
5. Configure the data parameters that appear in the right hand pane of the Create screen. These options will vary depending on what type of visualization you choose in step 3. The following example uses a pie chart visualization (Image 11):
- Metrics
- Aggregation: Choose how you want your data aggregated. This example uses Count.
- Custom Label: You can use this optional field for custom labelling.
- Buckets
- Aggregation: Choose how you want your data aggregated. This example uses Split Slices > Terms.
- Field: This drop down is populated based on the index source your chose. Select which field you want to use in your visualization. This example uses machine.os.keyword.
- Order By: Define how you want your data to be ordered. This example uses Metric: Count, in descending order of size 10.
- Choose wheteher to group other values in a seperate bucket. If you toggle this on, you will need to label the new bucket.
- Choose whether to show missing values.
- Advanced
- You can optionally choose a JSON input. These will be merged with the opensearch aggregation definition.
- Options
- The variables in the options tab can be used to configure the UI of the visualization itself.
Image 11: Creating your Visualization
6. There are a few ways to further focus your visualization:
- Use DQL to search your index data (Image 12). You can also save any queries you write for easy access by clicking on the save icon.
Image 12: Use a query on your Visualization
- Add a filter on any of your fields (Image 13).
Image 13: Add a filter on any of your fields
- Update your date filter (Image 14).
Image 14: Update your date filter
7. Click save when finished with your visualization.
Once you have created your visualizations, you can combine them together on one Dashboard for easy access.
You can also create new visualizations from the Dashboard screen.
- 1.From the left navigation pane, click on Dashboards (Image 15).
Image 15: Click Dashboards
2. If you have any Dashboards, they will appear on this page. To create a new one, click the Create Dashboard button (Image 16).
Image 16: Click Create Dashboard
3. The "Editing New Dashboard" screen will appear. Click on Add an Existing object (Image 17).
Image 17: Click Add An Existing
4. Select any of the visualizations you created and it will automatically add to your Dashboard (Image 18). Repeat this step for as many visualizations as you'd like to appear.
Image 18: Add as many visualizations as you'd like
5. Click Save to finish (Image 19).
Image 19: Click Save.
This capability was added in Cinchy v5.4.
Your Opensearch password can be updated in your deployment.json file (you may have renamed this during your original deployment).
- 1.Navigate to "cluster_component_config" > "opensearch".
- 2.There are two users that you can configure the passwords for: Admin and Kibana Server. Kibana Server is used for communication between the opensearch dashboard and the opensearch server. The default password for both is set to "password";. To update this, you will need to use a machine with docker available.
- 3.Update your Admin password:
- 1.Your password must be hashed. You can do so by running the following command on a machine with docker available, inputting your new password where noted:
docker run -it opensearchproject/opensearch /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p <<newpassword>>
2. Navigate to "opensearch_admin_user_hashed_password" and input your hashed password.
3. You must also provide your password in a base64 encoded format; input your cleartext password here to receive your new encoded password.
4. Navigate to "opensearch_admin_user_password_base64" and input your encoded password.
4. Update your Kibana Server password:
1. Your password must be hashed. You can do so by running the following command on a machine with docker available, inputting your new password where noted:
docker run -it opensearchproject/opensearch /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p <<newpassword>>
2. Navigate to "opensearch_kibanaserver_user_hashed_password" and input your hashed password.
3. You must also provide your new password in cleartext. Navigate to "opensearch_kibanaserver_user_password" and input your cleartext password.
5. Run the below command in the root directory of your devops.automations repo to update your configurations. If you have changed the name of your deployment.json file, make sure to update the command accordingly.
dotnet Cinchy.DevOps.Automations.dll "deployment.json"
4. Commit and push your changes.
5. If your environment is not set-up to automatically apply upon configuration,navigate to the ArgoCD portal and refresh your component(s). If that does not work, re-sync.
Last modified 1mo ago