With vSAN cluster, vSAN health checks provide us the consolidated dashboard to monitor the status of vSAN cluster components, diagnose issues, and troubleshoot problems. The health checks cover hardware compatibility, network configuration and operation, advanced vSAN configuration options, storage device health, and virtual machine objects, etc. We need to ensure VSAN health checks are green in the state. If any of the health checks failed, it will display the Failed status for that particular health check. One of the important VSAN health check services is VSAN CLOMD Liveness check.
vSAN CLOMD (Cluster Level Object Manager Daemon) plays a key role in the vSAN cluster operation. It runs on every ESXi host on the vSAN cluster and is responsible for new object creation, initiating repair of existing objects after failures, all types of data moves and evacuations such as vSAN Host Maintenance Mode, Evacuate data on disk removal from vSAN, etc. It also maintains vSAN cluster balance and triggers rebalancing, implementing policy changes, etc.
vSAN CLOMD does not actually participate in the data path, but it triggers data path operations and as such is a critical component in a number of management workflows and failure handling scenarios. Virtual machine power on, or Storage vMotion to vSAN are two operations where CLOMD is required (and which are not that obvious), as those operations require the creation of a swap object, and object creation requires CLOMD.
Similarly, starting with vSAN 6.0, memory snapshots are maintained as objects, so taking a snapshot with memory state will also require the CLOMD. In this article, I will explain the detailed step by step procedure to troubleshoot the failure of VSAN health service – vSAN CLOMD liveness check.
How to troubleshoot VSAN CLOMD liveness Failure
Log in to vCenter Server using vSphere web client or HTML5 cluster. Select the vSAN cluster ->Monitor ->vSAN -> Health -> Click on “Retest”. the vSAN health check will report the possible warning and errors of the VSAN cluster.
In the below screenshot, vSAN health check displays the failure for vSAN CLOMD liveness and also it displays the which host experience the failure of the service “vSAN CLOMD liveness”.
It displays abnormal for one of the hosts “md-esxi-i03” and shows error “Cannot connect to clomd process and possibly it’s down.
There are few of the below possible reasons, CLOMD liveness health check may be failed :
1. CLOMD service is in stopped or not responding state on the particular ESXi host
2. If ESXi host is in non-responding state or host disconnected from the vCenter server
3. Health service is not installed or stopped on particular ESXi host, it displays the state “unknown” for CLOMD liveness state
Let’s log in to particular ESXi host via SSH with root credentials. Check the status of the vSAN CLOMD status using the below command
/etc/init.d/clomd status /etc/init.d/clomd start
Even if the service is in running status. You can try stopping and starting the vSAN clomd service.
Once the service is started, perform the retest of the vSAN health check.
Once the vSAN CLOMD service is started, I can see the “vSAN CLOMD liveness” is “passed”. We are good now.
That’s it. We are done with troubleshooting vSAN CLOMD liveness. I hope this article is informative for you. Thanks for reading !!. Be social and share it with social media if you feel worth sharing it.