While updating VMware ESXi servers, VMware vSphere users may encounter the “No space left on device” error that pops up while executing the “esxcli software vib update” command. Interestingly, the problem occurs even though disks are doing well and have enough free space and df -h command proves that.
Find below how the output with the described error looks like:
|
1 2 3 4 |
[InstallationError] [Errno 28] No space left on device vibs = VMware_bootbank_esx-base_6.5.0-1.41.7967591 Please refer to the log file for more details. |
df -h output, in its turn, declines this claim, saying that there is enough space in these volumes:
So, what’s going on with your ESXi host? Rarely, the error is caused by a lack of inodes file objects. This issue is discussed in detail in KB 1007638. inodes are the file system structures, a maximum number of which for a single VMFS volume can reach up to 640 000. The number of utilized inodes depends on how many files are kept in the file system at a particular moment.
You can check the number of available inodes with stat -f / command:
The output provided in the screenshot above says that these objects are still redundant.
You also can check the number of available inodes by executing the df -I command:
|
1 2 3 |
$ df –i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 942080 507361 11 100% / |
As far as you can see, it is hardly possible to reach this number, yet if you do manage to reach that limit, just delete some files on ESXi host. In this way, there should be something else to cause the error.
For instance, there can be log files that occupy more than 50 MB and other files on ESXi that are kept out of VMFS volumes. You can find these files with the following command (it avoids VM logs on VMFS volumes):
|
1 |
find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -l '{}' \; |
After running the command, there will be generated a list of local files like ISO images, big log files, etc. All that stuff can be deleted if you do not need them or sent to some storage for archival purposes. Have a look through KB 1008643. This article discusses deleting files to free some inodes objects.
Another (and most probable) thing that may cause the described error can be a lack of RAM on the ESXi host. The host, basically, cannot allocate enough RAM for updating. This issue can be resolved by enabling the ESXi system swap located on another datastore where RAM will be flashed during updates if the problem occurs.
In order to enable the ESXi system swap, navigate to Manage > Settings > System Swap. Next, press Edit…:
Specify the datastore and click Ok. You can also set datastore for memory paging in Edit Host Profiles:
Once you select the datastore for system swap, update the ESXi host and reboot it. Actually, it is the most common solution to the discussed issue.
