OpenSearch Debugging Commands
This topic lists commonly used commands to debug and view OpenSearch status.
Health Check
Use the paragon-utils curl
http://opensearch-cluster-master.common:9200/_cat/health?v command
to view the overall status of OpenSearch.
root@pa1:~# paragon-utils curl http://opensearch-cluster-master.common:9200/_cat/health?v epoch timestamp cluster status node.total node.data discovered_cluster_manager shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1738252967 16:02:47 opensearch-cluster green 3 3 true 54 23 0 0 0 0 - 100.0%
The database is considered healthy if the status is green. A green status indicates that all shards are assigned to nodes. A yellow status indicates that primary shards are assigned to nodes, but some replicas are not. A red status indicates that at the least one primary shard is not assigned to a node.
List Indices
Use the paragon-utils curl -s
http://opensearch-cluster-master.common:9200/_cat/indices?v command
to list all OpenSearch indices and their sizes.
root@pa1:~# paragon-utils curl -s http://opensearch-cluster-master.common:9200/_cat/indices?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open ask-paragon-chat-sessions-000001 yvDkfiCDSWS4HlvuSPN-Pg 3 2 0 0 1.8kb 624b green open .plugins-ml-config WNRjb532QNqnTx3VMXrFFQ 1 1 1 0 7.8kb 3.9kb green open .opensearch-observability g5NuXSgcR5WXMXkrh_Sazg 1 2 0 0 624b 208b green open audits_00001 VQKBbdmKQTuQ0jcmXQ3DvQ 1 1 4 0 88.9kb 44.4kb green open ask-paragon-chat-history-000001 J5kScLMgRfS9ZmGV3uOxPw 3 2 0 0 1.8kb 624b green open .opendistro-job-scheduler-lock 7wu2FIqaSlqLEk6BT0SaJg 1 1 3 159 149.9kb 71.7kb green open audits-202501 7ZAFD0QhTwyzKxWztEB_hQ 1 1 4 0 88.9kb 44.4kb green open jcloud_alerts_active OfXRI1wtSQWJ8XRiXzU6-w 1 1 0 0 416b 208b green open .ds-jcloud_alerts_cleared-000001 JMqwDN9bRe-uVap4Cml8uQ 6 1 0 0 2.4kb 1.2kb
Check Shards Status
Use the paragon-utils curl
http://opensearch-cluster-master:9200/_cat/shards?v command to
check OpenSearch primary and replica shards status, nodes to which they are
assigned and their IP addresses.
root@pa1:~# paragon-utils curl http://opensearch-cluster-master:9200/_cat/shards?v index shard prirep state docs store ip node .opendistro-ism-managed-index-history-2025.01.30-000003 0 p STARTED 10.1.2.35 opensearch-cluster-master-1 .opendistro-ism-managed-index-history-2025.01.30-000003 0 r STARTED 10.1.2.49 opensearch-cluster-master-0 .opendistro-ism-config 0 p STARTED 10.1.2.35 opensearch-cluster-master-1 .opendistro-ism-config 0 r STARTED 10.1.2.49 opensearch-cluster-master-0 audits_00001 0 r STARTED 4 44.4kb 10.1.2.53 opensearch-cluster-master-2 audits_00001 0 p STARTED 4 44.4kb 10.1.2.49 opensearch-cluster-master-0 ask-paragon-chat-history-000001 0 p STARTED 0 208b 10.1.2.53 opensearch-cluster-master-2 <output snipped> .ds-jcloud_alerts_cleared-000001 4 p STARTED 0 208b 10.1.2.35 opensearch-cluster-master-1 .ds-jcloud_alerts_cleared-000001 4 r STARTED 0 208b 10.1.2.49 opensearch-cluster-master-0 .ds-jcloud_alerts_cleared-000001 5 r STARTED 0 208b 10.1.2.35 opensearch-cluster-master-1 .ds-jcloud_alerts_cleared-000001 5 p STARTED 0 208b 10.1.2.49 opensearch-cluster-master-0
Consequently, if a shard status is not STARTED, use the explain
API command to debug the reason.
root@pa1:~# paragon-utils curl -s http://opensearch-cluster-master.common:9200/_cluster/allocation/explain?pretty
{
"index" : "ask-paragon-chat-history-000001",
"shard" : 0,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "NODE_LEFT",
"at" : "2025-02-21T19:44:35.163Z",
"details" : "node_left [T92oYspJRVSCyzycX3Am6A]",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "6hSdCXELRDWmSQIy1pHBWg",
"node_name" : "opensearch-cluster-master-0",
"transport_address" : "10.1.2.3:9300",
"node_attributes" : {
"shard_indexing_pressure_enabled" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "a copy of this shard is already allocated to this node [[ask-paragon-chat-history-000001][0], node[6hSdCXELRDWmSQIy1pHBWg], [R], s[STARTED], a[id=G0v4Mf9WSNGJynneOBLQPQ]]"
}
]
},
{
"node_id" : "U2cN4Nv9TbCWL8J95crcdg",
"node_name" : "opensearch-cluster-master-1",
"transport_address" : "10.3.4.5:9300",
"node_attributes" : {
"shard_indexing_pressure_enabled" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "a copy of this shard is already allocated to this node [[ask-paragon-chat-history-000001][0], node[U2cN4Nv9TbCWL8J95crcdg], [P], s[STARTED], a[id=UlUPeUOIQHeDP9croBRngw]]"
}
]
}
]
}Increase OpenSearch Memory
The value for Dynamic Memory Configuration for OpenSearch JVM options such as
heap size setting is stored in opensearch_java_opts in the
config.yml file. Bu default, the value is set to
opensearch_java_opts: "-Xmx4096M -Xms4096M".
To
increase OpenSearch memory:
Log in to a cluster node.
Type configure to enter configuration mode in Paragon Shell.
Use the following command to increase memory and enter the required memory value.
[edit] root@pa1# set paragon cluster custom-config keyValue opensearch_java_opts string "Xmx5120M -Xms5120M" [edit]
Commit the configuration and exit configuration mode.
[edit] root@pa1# commit commit complete [edit] root@pa1# exit
Regenerate the configuration files.
root@pa1> request paragon config Paragon inventory file saved at /epic/config/inventory Paragon config file saved at /epic/config/config
Redeploy OpenSearch.
root@pa1> request paragon deploy cluster input "-t opensearch"