Extending A vSphere Replicated Virtual Disk

I recently had a VM that needed one of its virtual disks extended.  vSphere Replication needed to be disabled for this disk before vCenter would allow this operation.
When reconfiguring replication for this VM/disk, it detected the original/smaller disk:

Duplicate File Found. Do you want to use this file as an initial copy?

If you choose yes, it will try to use this file instead of re-sending the entire virtual disk.  This option didn't work. I think the files are just too different (size) and it doesn't know how to handle it.

If you choose no, you can configure a different datastore, but it won't let you use the same one.  This would leave the original replicated virtual disk out there unnecessarily taking up space.

I ended up manually deleting the original replicated VMDK and had VR resend the virtual disk again.  No big deal this time as it was just Disk 0/C: drive but this could be a real PitA if you need to extend a larger disk.

Lessons learned:

1. Size your drives properly the first time.
2. Consider creating a new disk instead of extending an already large virtual disk (30, 40, 50GB+).

ERROR: vSphere Replication shows Not Active

Another strange one.  Existing VM replications appear to be working based on "last sync completed" time stamps.  However, setting up a new replications result in a status of "Not Active".  Right-clicking on a VM and choosing "sychronize now" results in this error:
Call "HmsGroup.OnlineSync" for object "[some long GID]" on Server "[server name/IP]" failed.  An unknown error has occurred.
I Googled the error and found this VMware communities forum thread:
SRM5 using vSphere replication, status shows 'not active'

Read through it but note that you shouldn't have to reboot everything like one poster did.  I rebooted the VRMS server in the recovery site and replications started working for all VMs again.  YMMV.  This happened to me after having rebooted the vCenter server also at the recovery site.

ERROR: A general system error occurred

Recently, when trying to logon to SRM 5, I got the following error:
A general system error occurred: Internal error

Wow, that's real telling!

I tried restarting the SRM servers but no dice.  I then opened a ticket with VMware support.  I started a WebEx with the tech and after reviewing several SRM and vCenter logs, he really couldn't find the root cause of the problem.  However, he did say that they've only seen this generic error with vCenter, not SRM.

We rebooted the recovery side vCenter and viola, I was able to login again successfully.  It's the old saying - if all else fails, reboot!

vSphere 5 Upgrade: SRM - PART 1

Per Its Time vSphere 5 Upgrade, time to upgrade SRM.  Well, I got to step 6.2 and things went downhill from there.  The following is the description I used to open an SR with VMware support:

Upgrading SRM from 4.1.2 to 5.0.  I get the error: "failed to create database tables".  It doesn't appear to make any changes to the database.
I've double-checked settings per KB1015436 and several communities postings.
I've tried re-installing SRM 4.1.2 (successfully), performing a repair, then another upgrade but it still fails with the same error.

The first support tech went down the 'invalid permissions' path but this was not the problem.  Turns out, the SRM 5.0 upgrade does not support upgrading from SRM 4.1.2!  Interesting because the only documentation I can find on the subject clearly states that you can upgrade from SRM 4.1(!).  Looks like VMware needs to do a better job documenting these requirements.

I was then informed that I could wait until the next minor/point release of SRM 5 which would support upgrading from 4.1.2, but I didn't have that kind of time (and who knows when they'll actually release it).  So no upgrade for me, full install from scratch instead!  Besides have to reconfigure mappings, protection groups (which I was going to have to do anyway), etc, the biggest downside is losing the previous DR test results.  Yes, I saved those off as separate Excel files, but it would have been nice to have had all of the results right there in SRM from the beginning.


But wait, there's more!  Now that I have a brand new freshly installed SRM up and running, it's time to setup vSphere Replication.  Did that go problem free you ask?  Ummm, no.  The following is the description I used to open yet another SR with VMware support:

The VRMS servers at both sites fail to connect.  I have unregistered the server, powered down/deleted the appliance VM, re-initialized the VRMS database, repaired SRM, redeployed the VRMS servers and configured them with the same vCenter FQDN per KB2007463 but still have the same problem.

Between the support tech and I it took several hours to figure this one out.  The short of it is that it's a vCenter certificate problem.  What clued me into this was the error I got when registering the VRMS instance:

That "unacceptable signature algorithm" message is not your typical self-signed cert warning!  Turns out, my vCenter self-signed certs had expired.  This hadn't caused a problem until installing vSphere Replication - it wants at least a current/non-expired cert.  I checked the vCenter cert and sure enough, it had expired in 2010.  It was created in 2008 and was valid for only 2 years!

Now I bet you're wondering, how does one fix this cert problem?  Well that's easy, reinstall vCenter!  And repair won't work either so you have to uninstall the current vCenter instance and re-install a new one.  Luckily most settings are maintained in the vCenter database so this could have been much more painful.

While I was at it I checked the new vCenter cert and VMware apparently decided to make this one valid for 10 years.  Now that's more like it!

But wait, there's more!  Look for part PART 2 of this adventure in a near future post.  A little hint - the fun ain't over yet.

It's Time: vSphere 5 Upgrade

The technological benefits of upgrading from VMware vSphere 4.x to vSphere 5.x are not as great as upgrading VMware Virtual Infrastructure 3.x to vSphere 4.x in my opinion.  I don't know what the numbers are but I'm guessing that adoption hasn't been as fast as a consequence.

The latest XtraVirt poll posed this question with the following results:
What is your company's timescale for vSphere 5 deployment?
35%   6-12 Months
27%   1-3 Months
15%   3-6 Months
15%   1-2 Years
8%     No Current Plan

Well over half of admins responding will be upgrading within a year.  That's not bad.  I am in the "1-3 months" category and could argue that I've already started (planning and testing are complete).  The driver in my case is not technology/features-based, but is cost avoidance.

HP EVA SAN replication is licensed by capacity.  I don't want to rant about how utterly wrong this is but I'm not about to pay HP additional thousands of dollars for the privilege of replicating my data, thank you.  And now that SRM 5.0 provides replication for free, I don't have to (see previous post).

I've been planning our vSphere upgrade over the last month.  I've successfully installed and upgraded to vSphere 5 in the lab, now it's time to upgrade production.  I think VMware has finally got the upgrade process right.  It took them awhile although the vSphere 4.x upgrade was fairly smooth.  If you upgraded Virtual Infrastructure 2 to 3 you know what I mean!

I've pasted my upgrade plan below (taking out company-specifics).  I also included some notes from the lab tests. On to the show!

Phase 1: Planning

VMware vSphere Upgrade Center

http://www.vmware.com/products/vsphere/upgrade-center/

VMware vSphere Upgrade and Install Community Forum

http://communities.vmware.com/community/vmtn/server/vsphere/upgradecenter?view=discussions

(Note: I watched the forums and blogosphere in general to determine the stability of the release.  This is always a good practice but especially with a ".0" release - let others test it out.)

As of 9/15/2011, no real “deal-killer”/systemic errors found during or after upgrade.  Here’s the list:

1.       Port group names must be less than 19 characters (no impact to us)

2.       vCenter license must be provided during installation or it will start in expired mode (no impact to us)

3.       One admin gets an error when trying to upgrade ESXi via CLI – “downloading metadata failed” (no impact to us – we’re not using CLI to upgrade ESXi)

4.       One admin had a corrupted esx.conf file that prevented him from upgrading ESXi (no impact to us)

Check Pre-requisites:

1.       Check VMware Infrastructure Licenses

a.       vCenter 5.0  (available)

b.      ESXi 5.0  (available)

c.   Site Recovery Manager 5.0 (available November 19, 2011)

2.       Check Hardware Compatibility

a.       Server Platform

b.   SAN Platform

c.   HBA cards

d.   NIC cards

3.       Check Software Compatibility

a.       Review Guest OS Support

b.      Review 3^rd-party Product Support

                                                   i.      Backup Software

                                                 ii.      Management Software

                                                iii.      Custom Scripts

c.       Review Interoperability matrix

                                                   i.      http://www.vmware.com/resources/compatibility/sim/interop_matrix.php

PROJECT MILESTONE: Planning Complete

Phase 2: Testing

Setup Test Environment

Once the above steps are complete, it’s time to start testing.

(Note: The general idea here is to configure an environment that matches the current production environment as closely as possible.  I've done this in the past using cloned VMs with "mostly" successful results.  This is the way to go if you can keep the test and production networks isolated.  For this upgrade, I decided to configure new VMs - this should still give me valid tests and is faster to get up and running than using cloned VMs based on past experience.  If you have the time, you may want to go the cloned VMs route.)

Step 1: Setup ESXi Hosts

1.       Setup 3 servers as ESXi hosts in Datacenter1

2.       Setup 3 servers as ESXi hosts in Datacenter2

3.       Install vSphere 4.1 (or whatever your current version is) each host per the COMPANY Installation Guide.  (You do have this documented, right?)

Step 2: Setup SAN LUNs and Replication

1.       Create one LUN in Datacenter1 for vCenter and SQL VMs

2.       Create one LUN in Datacenter2 for vCenter and SQL VMs

3.       Create one read-only LUN in Datacenter2 to host the replicated LUN

4.       Configure the Datacenter1 LUN for replication to Datacenter2.

Step 3: Setup Test Virtual Machines

1.       Install one base Windows Server 2008 R2 image on one host as each site to be used as the VCenter/SRM server

2.       Clone images for use as SQL server to host the vCenter databases

3.       Configure images as stand-alone VMs.

Step 4: Configure SQL Server

1.       Install the same SQL Server version used in production both SQL VMs

a.       Use the production configuration as reference.

Step 5: Configure vCenter

1.       Install the SQL Native Client on both vCenter VMs

2.       Install vCenter on both vCenter VMs

a.       Use the production configuration as reference

b.      Configure vCenter with production licenses during installation or it will start in expired mode

3.       Add hosts to vCenter as appropriate

4.       Use vUM to apply the same updates to ESXi hosts as are present in production.

a.       TEST: This will test vUM to upgrade hosts

b.      TEST: This will also test vMotion capability

c.       TEST: This will also test the ESXi host configuration (networks, storage, etc.).

Step 6: Configure SRM

1.       Install SRM on each vCenter VM

2.       Configure SRM for the replicated LUN

3.       TEST: Perform a test SRM recovery to ensure system is functional.

PROJECT MILESTONE: Fully Functional Test Environment Complete

Upgrade to vCenter 5.0/SRM 5.0

Now with a fully functional test environment in place, it’s time to start upgrading.

The upgrade to 5.0 is supposed to be the least intense VI/vSphere upgrade yet.  The steps are fairly straight-forward:

1.     Upgrade vCenter/Upgrade Manager (vUM)
2.     Upgrade the ESXi hosts
3.     Upgrade VMware Tools of the VMs and the Virtual Hardware of the VMs to version 8
4.     Upgrade the VMFS datastores to version 5
5.     Upgrade the SRM to version 5
6.     Upgrade and 3^rd-party tools and scripts.

Step 1: Upgrade vCenter

1.       Backup the vCenter database using SQL Server Management Studio

2.       Backup the SSL certificates (%allusersprofile%\Application Data\VMware\VMware VirtualCenter)

3.      Stop all vCenter services

4.       Install JDK 1.6

5.       Using the vCenter ISO, upgrade vCenter to version 5

a.       Upgrade the recovery site first

b.      Run the vCenter host agent pre-upgrade checker

6.       Configure the new vSphere 5 licenses

7.       Upgrade the vSphere Client

8.       From the recovery site, rejoin the site via Linked Mode:
From the Start menu, select All Programs > VMware > vCenter Server Linked Mode Configuration

Step 2: Upgrade Manager (vUM)

1.       Backup the vUM database using SQL Server Management Studio

2.       Stop all vUM services

3.       Install JDK 1.6

4.       Using the vCenter ISO, upgrade vUM to version 5

a.       Upgrade the recovery site first

5.       Upgrade the vSphere Client vUM plug-in

Step 3: Upgrade the ESXi hosts

There are two options to upgrade the hosts: via vUM or doing a clean install with the OEM custom ESXi 5 image/ISO.  For testing purposes, we’ll try both methods.

1.       Using vUM, upgrade the first host of each cluster

a.       Test vMotioning VMs between 4.1 and 5.0 hosts

(Lab note: Need to force remediation.  Enable "remove incompatible packages".)

2.       Using the ISO, perform a clean install of the last host in each cluster.

(Note: I skipped step 2 - vUM worked so well I decided to go with that method.  HP provides an image that can be "imported" directly into vUM (very nice!) which worked great.)

DECISION POINT: Use vUM Upgrade or Clean Install from ISO

Step 4: Upgrade VMware Tools and Virtual Hardware

1.       Using vUM, upgrade the VMware Tools component of all VMs

2.       Using vUM, upgrade the virtual hardware of all VMs.

(Note: vUM scan skips vCenter and SQL VMs.  You'll need to go back and upgrade these manually.)

Step 5: Upgrade the VMFS Datastore

The VMFS datastores can be upgraded in place while the VMs are running.

Step 6: Upgrade SRM

1.       Snapshot VC and SQL VMs

2.       Upgrade SRM to 5.0 on recovery, then protected site vCenter VM

a.       Stop SRM

b.      Uninstall SAN SRA

c.       Uninstall the SRM plug-in

d.      Upgrade SRM

e.      Install the 5.0 SRA (if using array-based replication)

f.        Restart SRM

3.       Configure SRM

a.       Remove Array Manager

b.      Add new Array Manager

c.       Reconfigure protection group

d.      Reconfigure recovery plan.

4.       Install and Configure vSphere Replication (if not using array-based replication)

a.       At the PROTECTED SITE deploy the vSphere Replication Management Server (vRMS)

                                                   i.      Assign a static IP address

                                                 ii.      Register with the protected site’s vCenter instance

b.      At the RECOVERY SITE deploy the vSphere Replication Management Server (vRMS)

                                                   i.      Assign a static IP address

                                                 ii.      Register with the recovery site’s vCenter instance

c.       At the RECOVERY Site deploy the vSphere Replication Server (vRS)

d.      Configure VMs for replication via the vSphere client

5.       TEST: Perform a test SRM recovery to ensure system is functional

Step 7: Configure Network Monitoring Tool

Note: In our case, we configured 2 "applications" in our NetFlow-based network traffic monitoring system:

1. vSphere Initial: port 31031 (port used to do the inital "seed" copy of the VM)
2. vSphere Ongoing: port 44046 (port used to replicate changes after initial replication completes)

Step 8: Upgrade 3^rd-Party Tools

There’s little need to test tools that vendors have certified for vSphere 5.0.  However, custom scripts will need to be tested.

PROJECT MILESTONE: vCenter and Site Recovery Manager Upgrade Complete

Phase 3: Production Upgrade

Follow steps in Phase 2 to upgrade production.  

Conclusion 

That's it!  Depending on the outcome of your tests in Phase 2, this may become an iterative process.  Also note that there's always the chance that you've tested everything 100 times and still encounter an issue while upgrading production.  It's the nature of the beast.  However, having the experience of performing the upgrade in a test environment will give you a leg up in troubleshooting problems.  And, having worked with VMware support numerous times, I can recommend calling them without hesitation.

Finally, always remember to document things along the way.  Follow these steps and you will be in good shape.

vSphere Replication 1.0

With another problem comes another opportunity...

I have been working on upgrading our vSphere host hardware and migrating VMs from our old EMC Celerra NS350 to a newer HP EVA4400 (that in of itself is worthy of its own blog post).  We had purchased the EVA two years ago to host our ERP data and it has run with a single host accessing it ever since.

So we ordered additional disks and shelves for the EVAs at both primary and DR datacenters.  I installed the HBAs in the hosts, added them to the fabric, created zones, created the LUNs, masked theme off, etc, etc.  Everything was going great until...

I went to setup replication.  I setup array-based replication (ABR) for the first LUN - no problem.  Storage vMotioned a VM over to it and it replicated without issue.  Tried to setup replication for the second LUN - major obstacle time.  HP's replication mechanism for the EVA, Continuous Access (CA), is licensed based on capacity.  And of course, we had licensed 1TB but needed more like 12TB.  Great.  Meanwhile, there's grumblings and doubt by others on the IT team that CA would even be the right choice for replicating this data.

Come on HP, really? Does any vendor license replication by capacity anymore?  You don't do this with LeftHand/P4000 or 3PAR arrays.  Fustrating...

Now I'll be the first to tell you that I hate, hate, hate vendor lock-in.  Technology changes so fast that whatever you're using today, probably isn't what you'll be using 3-5-10 years from now.  Again, a good topic that deserves its own post.  This is one reason that, as a vSphere and storage engineer, I've become a fan of host-based replication (HBR).  There are third-party products that provide this capability for virtual machines today: Veeam Backup and Replication and Quest vReplicator just to name a couple.

But here comes vSphere 5 and SRM 5.  We'll be entitled to both when they're released.  As part of the upgrade we'll get the capability to replicate VMs using vSphere Replication 1.0 for free.  I've started setting up a testing environment and will post my experiences with this new feature.  One thing I'm really curious about is how the bits actually get replicated.  Different arrays handle this differently.  I will have my investigative hat on at VMworld and will ask the storage vendors all the gory details.  I'll follow-up with another article detailing how different vendors implement their replication (geesh, I've got a lot of writing to do!).

In the mean-time, I've gathered some information on vSphere Replication 1.0, all of which is publicly available.  Exciting stuff!  Here are the details:

• This feature is included with all editions of SRM 5
• VMs can be replicated from any storage to any storage, including local disk
• Replicated disks can be place on any ESXi-compatible disks/filesystem
• Breaks storage vendor lock-in
• Replication is an attribute of the VM (not the LUN or some other element)
• You can choose which VMDKs to replicate within the VM
• In some cases you may not want to replicate the system drive/VMDK, only the data drive/VMDK
• Disks are replicated in a "group consistent" manner
• Does not use CBT to track and replicate deltas.  Instead, VMware developed another technology that tracks I/O changes to VMDKs and captures them in a "PSF" or persistent state file.  It does not use VM snapshots
• I'm not sure why they didn't leverage existing CBT technology - more details to follow
• Initial "seed" copy can be made in advanced by FTP, external disk/sneaker net, etc.
• Saves bandwidth - great if you have a slower WAN connection and/or a large number of VMs to replicate
• RPO can be set on a per-VM basis
• 5 - minutes to ?
• If you need an RPO smaller than 5 minutes, you got other challenges to face!

Some limitations:

• VM must be powered-on
• My guess is that the thinking here is that if it's powered-off it must not be critical enough to recovery in a DR scenario.  I hope VMware reconsiders on this one.  I don't have any of these today, but it I can see the possibility of it in the future.
• Will not replicate swap, logs, dumps
• Will replicate VMs with snapshots.  However, snapshots will not be replicated.  Instead, the I/O from the source snapshot is written to the destination VM, effectively making the destination VM look like the source VM after collapsing the snapshot.
• No FT VMs, linked clones, templates, physical RDMs, ISOs or floppies
• Requires VM hardware version 7 or later

That wasn't too painful.  Here's what the (high-level) architecture looks like:

• vRMS - vSphere Replication Management Server
• Required at both sites
• This is a virtual appliance (VA) imported into vCenter
• vRA - vShpere Replciation agent
• Required at the protected site
• Runs on the ESXi 5 hosts
• vRS - vSphere Replication Server
• Runs on the recovery site
• This too is a VA imported into the vCenter at the recovery site

Scalability info:

• VM totals = 500 replicated (1000 total for SRM)
• If you need to protect more that 500 VMs, not only do you have a large environment, you'll need to use ABR or find an alternative HBR solution that can scale higher (if it exists).  With that size of an environment I'd recommend working with your VMware account representative and/or storage vendor.

For a storage geek like me this is pretty exciting stuff.  I think a lot of VMware customers, from the small SMB to the mid-sized and even some larger companies, are going to benefit from this new feature.

Time to kick the tires, stay tuned!