Before we get started, let’s talk about what is VMware ESXi 7.0 dump collector and why it’s important. There are times when VMware ESXi gets Purple screen of death (PSOD). In this case, it is very important to have a possibility and recover the memory to somewhere else.
Dump collector is a support tool for VCSA and allows ESXi to save the VMkernel memory to a network server instead to a disk. The vSphere ESXi Dump Collector collects memory dumps over the network instead of trying to save it locally.
Why would you like to change the default location of coredump?
- You might run out of space on a local datastore
- You want to keep coredumps on a network location
In fact, not all ESXi installations are stateful, so for stateless installations, such as those where you’ll use VMware auto deploy, you’ll need an external space to store logs.
You’ll need to configure ESXi Dump Collector by using ESXCLI commands and keep core dumps on a network server for use during debugging. In our case we’ll use VMware vCSA as a storage location.
What is ESXi Dump Collector?
VMware ESXi is basically sending the state of the VMkernel Memory (also called core dumps) to a network server (if you have configured one) each time there is a PSOD.
It is quite rare, but it can happen from time to time, after an upgrade, with obsolete firmware/drivers combination etc.
It is important to be able to troubleshoot the issue via those core dumps.
You can take screenshot of PSOD as the screenshot itself tells you similar story that you’ll find within your core dumps.
If your environment is not new, you might first check whether the dump collector works correctly and that it has been sized appropriately.
Please Note that ESXi Dump Collector is not supported to be configured on a VMkernel interface that is running on NSX-T N-VDS switch.
ESXi Dump Collector – How to setup?
First thing first login to your vCenter Server appliance using web client and start the dump collector service. By default, this service, which starts manually, is in stopped state.
VMware ESXi Dump Collector – Start the service
When this is done, we need to instruct our ESXi hosts to send those core dumps to our VCSA.
Then we will need to configure each VMware ESXi host via a command line. We’ll use Putty SSH client for the job.
Connect your ESXi host through Putty (make sure to enable SSH on your ESXi) and run this command below:
|
1 |
esxcli system coredump network get |
This gives you an information about the current status of our ESXi host.
As you can see, by default it’s disabled.
By default disabled
Use this command to configure the settings to your convenience.
|
1 |
esxcli system coredump network set -- interface-name vmk0 -- server-ipv4 10.x.x.x. --server-port 6500 |
It looks like this in the lab:
Set Coredump via this command
And then enable via this command:
|
1 |
esxcli system coredump network set --enable true |
Then once again, check with the first command if now it’s enabled, just to make sure.
Enable Coredump via this command
As you can see, we have it enabled, but is it working? We can test it. We can test the coredump configuration via this command:
|
1 |
esxcli system coredump network check |
Test the coredump configuration
Where is the coredump folder location on VCSA?
Yes, it is important to know where to look for coredumps on your VCSA in case you need to. In fact, you’ll have to connect to VCSA through putty and enable shell. Once done, just navigate to to:
|
1 |
/var/log/vmware/netdumper/netdumper.log |
There you’ll see the entries. By default, you’ll see that there is a storage space configured 5Gb per-host 20Gb in total.
If that’s not enough for your usage, you might re-configure it for your convenience, however, you might need to check whether you’ll need to increase storage of your VCSA on one of the 12 or 14 virtual hard disks that VCSA is configured with.
Default netdumper.log content
This is it. Once we have all setup, it’s just a luck (or bad luck) that we’ll need it one day.
Final Words
As I mentioned, PSOD does not happens often, but when doing upgrades/updates and you have dependencies, or VMware Installation Bundle (VIB) which is outdated, it might happen.
Quite often I’ve seen hosts going into PSOD because of unsupported hardware, such as NICs or storage add-on cards. Drivers for for those devices sometimes aren’t supported with latest vSphere releases and as such, either those might prevent to upgrade, or they upgrade and cause system instability, performance problems and in some extreme causes, PSODs.
Keeping a separate location for coredumps allows you to have those logs if the host does not have local storage or if the host is an autodeploy host which does not have any boot storage either.
Feel free to leave comments about your best PSOD you’ve experienced and (or) if you have setup an ESXi coredump to be able to investigate the problem.
