vSphere 5 Upgrade: VMFS Datastores

Per Its Time vSphere 5 Upgrade, time to upgrade VMFS datastores.  Like the last couple of steps this one completed w/o issue.  Note that it is better to create VMFS datastores because you'll get a 1MB block size regardless of what size datastore you're creating, optimizing disk space.  Compare this to the 2-8MB block sizes required based on datastore size in ESXi 4.1 and earlier.

All of my datastores now report to be VMFS version 5.54.

Removing a VMFS Datastore

To remove VMFS datastores in the past, I always made sure there were no VMs left on the datastore, right-clicked and chose "Delete".  Well, apparently thats the wrong way to do it!  I must have been lucky, as VMware claims this method could result in an APD (All Paths Down) state.  If you don't know what that is, let me tell you it's bad (I have experienced this but for a different reason).  Your host(s) will lose access to storage.

I stumbled upon this vSphere blog post that has the procedure to remove a datastore the right way: Best Practice: How to correctly remove a LUN from an ESX host

UPDATE: For ESXi 5.0, I found it better to follow the KB mentioned in that blog post:
Unpresenting a LUN in ESXi 5.x

For ESXi 5.0, here's how I do a slightly modified vesion of the procedure listed in the post above with more information on items such as HA:

1. Make sure all VMs are evacuated from the datastore/LUN.
2. Using Datastore Browser, make sure there aren't any left-over directories or files.  If there are, delete them (make sure they can safely deleted first, of course).  Exception: the HA directory - you'll remove that a different way later.  Also note that you won't be able to delete a file if it's in use.
3. Make sure all vSphere features are disabled for the datastore (i.e. SIOC, Storage DRS)
1. HA Datastore Heartbeating:  There may be some cases where HA has chosen the datastore you're trying to remove.  In this case, edit cluster settings and change Datastore Heartbeating to "select only from my preferred datastores", then select at least three datastores other than the one you're trying to remove.  I highly recommend changing this setting back to "Select any of the cluster datastores" after you're finished removing this one.
4. Next, for each host, go to Configuration\Storage\Datastores View, right-lick on the datastore and choose "unmount":
1. The "Unmount Datastore Wizard" appears. Make sure all hosts are selected.
2. Click next.  Hopefully the next step will look like this:
   
3. Click "Next" then "Finish".  After a minute the datastore will be grayed-out and italicized.
5. Then go to Configuration\Storage\Devices View, right-lick on the datastore and choose "detach"
1. You get a pop-up dialog window similar to the one show above.  Choose "OK.  The device will be grayed-out and italicized after a few seconds.
6. Go in to your SAN and remove LUN masking/unpresent the LUNs to the ESXi hosts.
7. Finally, right-click on your cluster and choose "Rescan for Datastores".  You may get a StorageConnectivityAlarm alert for every host in your cluster unless you disable this alert first.

I like to go into every host and check the storage adapter for the LUN just to make sure but I haven't found a problem with this procedure yet.

Snapshotting Large Files on Large Datastores

The error:

File <unspecified filename> is larger than the maximum size supported by datastore '<unspecified datastore>

The problem:
The VM has 2 virtual disks, one for the primary system OS on datastore1 formatted with a 2MB block size, the other for data on datastore2 formatted with an 8MB block size.

In vSphere 4.1, when a snap is created on a VM with the above configuration it will put the change files in the same datastore/directory as the VMX file (datastore1 in this case).  So it will try to create a VMDK with the same provisioned size as the original VMDK, although the actual size will typically be much smaller since this file holds only the changes since the snap was created.

Consider a VM with VMDK1 on Datastore1 with a size of 30GB and VMDK2 on Datastore2 with a size of 512GB.  Datastore1 was formatted with 2MB block size and Datastore2 was formatted with 4MB size.  As long as VMDK2 is less than 512GB, the snap will complete successfully.  However, if VMDK2 is over 512GB, the snap will fail with the error listed above even though Datastore2 is capable of hosting virtual disks/VMDKs up to 1TB in size.

I haven't tried this with vSphere 5 yet but will update this post as soon as I have completed testing.  I would expect the same results, especially for upgraded datastores.

The solution:
The best practice is to format all datastores with the same block size (a non-issue for vSphere 5). This will allow you to avoid the problem all-together.

You can also exclude the VMDK on the datastore formatted with the larger block size by changing the virtual disk mode to Independent/Persistent.  The main downside is you won't be able to capture image/VMDK-based backups.

Also consider this chart from VMware KB1003565:

Block Size	Largest virtual disk on VMFS-2	Largest virtual disk on VMFS-3	Largest virtual disk on VMFS-5
1MB	456GB	256GB	2TB minus 512B
2MB	912GB	512GB	Valid if upgraded from VMFS-3
4MB	1.78TB	1TB	Valid if upgraded from VMFS-3
8MB	2TB	2TB minus 512B	Valid if upgraded from VMFS-3
16MB	2TB	Invalid block size	Invalid block size
32MB	2TB	Invalid block size	Invalid block size
64MB	2TB	Invalid block size	Invalid block size