Monday, January 9, 2012

Snapshotting Large Files on Large Datastores

The error:

File <unspecified filename> is larger than the maximum size supported by datastore '<unspecified datastore>

The problem:
The VM has 2 virtual disks, one for the primary system OS on datastore1 formatted with a 2MB block size, the other for data on datastore2 formatted with an 8MB block size.

In vSphere 4.1, when a snap is created on a VM with the above configuration it will put the change files in the same datastore/directory as the VMX file (datastore1 in this case).  So it will try to create a VMDK with the same provisioned size as the original VMDK, although the actual size will typically be much smaller since this file holds only the changes since the snap was created.

Consider a VM with VMDK1 on Datastore1 with a size of 30GB and VMDK2 on Datastore2 with a size of 512GB.  Datastore1 was formatted with 2MB block size and Datastore2 was formatted with 4MB size.  As long as VMDK2 is less than 512GB, the snap will complete successfully.  However, if VMDK2 is over 512GB, the snap will fail with the error listed above even though Datastore2 is capable of hosting virtual disks/VMDKs up to 1TB in size.

I haven't tried this with vSphere 5 yet but will update this post as soon as I have completed testing.  I would expect the same results, especially for upgraded datastores.

The solution:
The best practice is to format all datastores with the same block size (a non-issue for vSphere 5). This will allow you to avoid the problem all-together.

You can also exclude the VMDK on the datastore formatted with the larger block size by changing the virtual disk mode to Independent/Persistent.  The main downside is you won't be able to capture image/VMDK-based backups.

Also consider this chart from VMware KB1003565:

Block Size
Largest virtual disk on VMFS-2
Largest virtual disk on VMFS-3
Largest virtual disk on VMFS-5
1MB
456GB
256GB
2TB minus 512B
2MB
912GB
512GB
Valid if upgraded from VMFS-3
4MB
1.78TB
1TB
Valid if upgraded from VMFS-3
8MB
2TB
2TB minus 512B
Valid if upgraded from VMFS-3
16MB
2TB
Invalid block size
Invalid block size
32MB
2TB
Invalid block size
Invalid block size
64MB
2TB
Invalid block size
Invalid block size