Unable To Delete Orphaned/Stale VMDK File

So today I got a case where we were trying to delete an orphaned flat.vmdk file.

A brief background of what was being experienced here:

There were three ESXi hosts and 2 shared datastores among these hosts. Now, there were couple of folders in these 2 shared datastores which contained only flat.vmdk files. These flat files were not associated with any virtual machines and also the last modified date of these files were somewhere about a year ago.

However, every time we tried to delete the file from the datastore browser GUI, we got the error:

Cannot Delete File [Datastore Name] File_Name.vmdk

So, when we try to delete this file from the command line using the " rm -f <file_name> " we got the error:

rm: cannot remove 'File.vmdk': No such file or directory

Also:
We were able to move the file to another datastore and remove it successfully. But, the stale file copy was still left behind in the original datastore.

So, how do we remove this stale file?

Step 1:

• Take a SSH session to all the hosts that have access to this datastore where the stale file resides. 
• In my case all the three hosts in the cluster.

Step 2:

• Run the below command. This command has to be executed from the SSH(Putty) of all the hosts having connectivity to that datastore.

# vmkfstools -L release /vmfs/volumes/<datastore_name>/<folder_name>/<file_name>-flat.vmdk

This can result in two error outputs:

First error:

Could not open /vmfs/volumes/xxxxxxxx/xxxxxxx/xxxxxx-flat.vmdk 

Command release failed Error: Device or resource busy

Second error:

Command release failed 

Error: Inappropriate ioctl for device

In my case it was the second error.

The host that gives you the second error has the stale lock on the file. All the three hosts returned the second error, and I had to reboot all the three hosts. 

Once the hosts are rebooted, you can successfully remove the stale flat.vmdk files.