Using the vSphere 5 CLI for Storage Configuration

When Adding a new host, SAN or LUN to your vSphere environment there are some CLI commands and configuration settings you should consider.  Since Windows 7 is my primary desktop OS, I use the VMware vSphere CLI for Windows.  You can download the latest version from VMware's downloads site.

Before using any of the commands below, please review your SAN manufacturers best practices documentation for vSphere environments.  Can't find any?  Then you bought the wrong SAN!  Seriously, most manufacturers provide something around documentation - if Google doesn't help try using Bing... better yet, try contacting your VAR.

View Devices and Their Settings
To view devices and some of their related settings, use the following command:
esxcli -s [HOSTNAME] -u root -p [PASSWORD] storage nmp device list

Set Storage Array Type and Path Selection Policy
One of the first things you'll want to do is change the DEFAULT Path Selection Policy (PSP) for whatever storage array types (SATP) you're using.  This way, when adding a new LUN/device, the type will be set to the proper type automatically.  Most modern arrays support ROUND ROBIN.  To change the array type, use the following command:
esxcli -s [ESXHOST] -u root -p [PASSWORD] storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_ALUA (or VMW_SATP_CX, etc.)

Note that this will not change/update existing LUNs/devices.  I recommend using the vSphere client for this unless you have many that need to be updated.  In this case, use the following command:
FOR /F %G IN ('esxcli --server [HOSTNAME] --username root --password [PASSWORD] storage nmp device list ^| findstr naa.600') DO esxcli --server [HOSTNAME] --username root --password [PASSWORD] storage nmp device set --device %G --psp VMW_PSP_RR

Note 1: In the above command, I loop through and set the RR policy for all devices that begin with "naa.600" - set the NAA ID for your environment as needed.

Note 2: the Linux command uses grep but Windows has FINDSTR.  Took a bit for me to figure out but if nothing else, this is the big find in this article - your welcome!

Set the IOPs Value
Finally, most array manufacturers recommend setting IOPS to "1".  To change the IOPs parameter, use the following command (again, thank you FINDSTR!):
FOR /F %G IN ('esxcli --server [HOSTNAME] --username root --password [PASSWORD] storage nmp device list ^| findstr naa.6005') DO esxcli --server [HOSTNAME] --username root --password [PASSWORD] storage nmp psp roundrobin deviceconfig set -d %G --iops 1 --type iops

Note:  As in the previous command, I loop through and set the IOPs parameter for devices that start with "naa.600" - set this for your environment as needed.

Now the "devices" hosted on your array should be optimally configured for use by vSphere 5.  Don't forget to go through and check each host.

Other Helpful Commands

esxcli storage vmfs extent list
esxcli storage core device detached list
esxcli storage core device detached remove -d [NAA ID]
esxcli storage core adapter rescan [ -A vmhba# | --all ]
esxcli storage filesystem list
esxcli storage filesystem unmount [-u <UUID> | -l <label> | -p <path> ]

ERROR: VMware ESXi with 3PAR SAN and Dead LUN 254

Problem

Last week I discovered  a couple of error messages in the vmkernel.log file that caused me some concern:

2012-06-22T18:18:45.806Z cpu17:939673)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:972:Could not select path for device "Unregistered Device".
2012-06-22T18:18:45.806Z cpu17:939673)WARNING: NMP: nmpPathClaimEnd:1195:Device, seen through path vmhba2:C0:T2:L254 is not registered (no active paths)

I did a "esxcfg-mpath -l" and found 4 dead paths - 2 to each fibre HBA.  I never provisioned a LUN with an ID of 254.  Maybe this was a "special use" LUN?  I doubted it because my HP EVA has one of these and the device is listed in vCenter.  However, there were no devices with LUN ID of 254 listed anywhere in vCenter, only 4 dead paths.

Solution

After a focused Google search, I found the answer.  The HP 3PAR guy that came out and did the installation had us use a host persona of "1 - Generic" when we should have used "6 - Generic-legacy".  Luckily these can be changed on the fly via the InForm Management Console.

After making the change in the IMC, I rescaned each of the hosts and the messages stop appearing in the vmkernel.log and the dead paths were no longer listed in the vSphere Client.

Here are the relevant sites I found per the Google search:

Fix - 3PAR with VMware- LUN 254 Dead

3PAR/VMware LUN 254 Dead???

And, of course, I always recommend following the manufacturer's best practices:

HP 3PAR VMware ESX Implementation Guide

Looks like the guide was recently updated and it does recommend using the persona of "6 - Generic legacy".

Another mystery solved.

ERROR: The query service is not available or was restarted

I wasn't getting any results when going to the Hardware Status tab for all hosts in my recovery site.  When clicking "update", I'd get the error:
The query service is not available or was restarted. Please retry.

Of course retrying doesn't work.  Hardware status worked fine for all hosts in my protected site.  I thought maybe it was a problem with linked mode in vCenter so I logged on to the vCenter server in the recovery site, fired up the vSphere client, opened the Hardware Status tab and got the same results.

I then started another instance of the vSphere client and logged directly on to the ESXi host.  The hardware sensor data worked fine here (it's not in a separate hardware tab, but looks nearly the same).  Hmmm.... must be something with vCenter?

I Googled the error and found this VMware KB:

Host sensor data is not updated in vCenter Server 4.1.x and vCenter Server 5.0.x

Well it's the exact same error message so this must be the fix, right?  Wrong!

First of all, step 14 is incomplete.  Please follow these steps to reset the vCenter Inventory database:

Reset the vCenter Inventory Service Database

Secondly, this was not the only problem and probably didn't ultimately fix the issue.  I found this link in the same Google search:

VERY annoying problem in vSphere 5 - hardware status tab

The above forum posting had a link to the following web site with instructions on updating the ADAM instance vSphere uses for linked mode:

Hardware Status not displaying on vSphere Client – Fix

While this was for 4.1, the same settings apply to 5.0.  The only thing I would recommend is checking all of the common name (CN) properties to make sure the FQDNs are correct.  You do not need to change these to IP addresses!

I did have to reboot the vCenter server in the recovery site after making the changes.  Even then, it didn't seem to start working until the following morning so it may take some time for the changes to propagate.  Not certain about that but now the hardware status tab works for all servers in the recovery site.

VMware ESXi and HP 3PAR Storage

We recently added a new HP 3PAR F400 SAN to our existing VMware cluster.  If you haven't read about this SAN and are interested in SAN storage, I highly recommend you take a look at HP's site for more information: 3PAR Storage.

One thing I've found this SAN lacking in is documentation.  It's easy to find, but bits of information are scattered across several documents.  The HP implementation consultant handed me a thumb drive with 3GBs of data before he left, half of which are PDFs (to give you an idea).

The point of this post is to bring all of these bits together in one place to include settings and best practices for VMware ESXi 5.0 along with the relevant references.  Here's what I've found so far - I will continue to add and update settings as I find them where relevant:

ESXi Advanced Settings:

DiskMaxIOSize   = 128 (Source)
QFullSampleSize = 32  (Source)
QFullThreshold  = 4

Default Path Selection Policy (PSP) = VM_PSP_RR (Source and Powershell Tip)

IOPS = 1

LUN Size = As large as ESXi can handle, currently 2TB (Source)

VMDK format = Eager Zeroed Thick

VMs - zero-out free space (Source):

   For Windows = SDELETE (Download)

   For Linux = DD  (section 3.2.1 in source doc linked above)

vSphere 5 Upgrade: SRM - PART 1

Per Its Time vSphere 5 Upgrade, time to upgrade SRM.  Well, I got to step 6.2 and things went downhill from there.  The following is the description I used to open an SR with VMware support:

Upgrading SRM from 4.1.2 to 5.0.  I get the error: "failed to create database tables".  It doesn't appear to make any changes to the database.
I've double-checked settings per KB1015436 and several communities postings.
I've tried re-installing SRM 4.1.2 (successfully), performing a repair, then another upgrade but it still fails with the same error.

The first support tech went down the 'invalid permissions' path but this was not the problem.  Turns out, the SRM 5.0 upgrade does not support upgrading from SRM 4.1.2!  Interesting because the only documentation I can find on the subject clearly states that you can upgrade from SRM 4.1(!).  Looks like VMware needs to do a better job documenting these requirements.

I was then informed that I could wait until the next minor/point release of SRM 5 which would support upgrading from 4.1.2, but I didn't have that kind of time (and who knows when they'll actually release it).  So no upgrade for me, full install from scratch instead!  Besides have to reconfigure mappings, protection groups (which I was going to have to do anyway), etc, the biggest downside is losing the previous DR test results.  Yes, I saved those off as separate Excel files, but it would have been nice to have had all of the results right there in SRM from the beginning.


But wait, there's more!  Now that I have a brand new freshly installed SRM up and running, it's time to setup vSphere Replication.  Did that go problem free you ask?  Ummm, no.  The following is the description I used to open yet another SR with VMware support:

The VRMS servers at both sites fail to connect.  I have unregistered the server, powered down/deleted the appliance VM, re-initialized the VRMS database, repaired SRM, redeployed the VRMS servers and configured them with the same vCenter FQDN per KB2007463 but still have the same problem.

Between the support tech and I it took several hours to figure this one out.  The short of it is that it's a vCenter certificate problem.  What clued me into this was the error I got when registering the VRMS instance:

That "unacceptable signature algorithm" message is not your typical self-signed cert warning!  Turns out, my vCenter self-signed certs had expired.  This hadn't caused a problem until installing vSphere Replication - it wants at least a current/non-expired cert.  I checked the vCenter cert and sure enough, it had expired in 2010.  It was created in 2008 and was valid for only 2 years!

Now I bet you're wondering, how does one fix this cert problem?  Well that's easy, reinstall vCenter!  And repair won't work either so you have to uninstall the current vCenter instance and re-install a new one.  Luckily most settings are maintained in the vCenter database so this could have been much more painful.

While I was at it I checked the new vCenter cert and VMware apparently decided to make this one valid for 10 years.  Now that's more like it!

But wait, there's more!  Look for part PART 2 of this adventure in a near future post.  A little hint - the fun ain't over yet.

vSphere 5 Upgrade: VMFS Datastores

Per Its Time vSphere 5 Upgrade, time to upgrade VMFS datastores.  Like the last couple of steps this one completed w/o issue.  Note that it is better to create VMFS datastores because you'll get a 1MB block size regardless of what size datastore you're creating, optimizing disk space.  Compare this to the 2-8MB block sizes required based on datastore size in ESXi 4.1 and earlier.

All of my datastores now report to be VMFS version 5.54.

vSphere 5 Upgrade: VM Tools and Virtual Hardware

Per Its Time vSphere 5 Upgrade, time to upgrade VMware Tools and Virtual Hardware.  I'm happy to report that both of these steps completed w/o issue.  After the previous 4.1 upgrade, the VMware Tools upgrade corrupted one of my Windows 2000 VMs.  VMware claimed it was a Microsoft problem, Microsoft claimed it was a VMware problem.  Love the finger-pointing guys!  Anyway, I made sure this VM was upgraded (actually replaced) with a Windows 2008 R2 server and we updated the application while we were at it - a win/win in my book.

No problems this time (phew!).

The only other thing I did differently compared to my previous upgrade was upgrade the SQL and vCenter servers first.  vUpdate Manager will skip these VMs anyway.

vSphere 5 Upgrade: ESXi Hosts

Per Its Time vSphere 5 Upgrade, the ESXi host upgrades.  Not much to report here.  The upgrades completed w/o issue per vUpdate Manager.

Note that I had to force remediation.  Then I enabled "remove incompatible packages".  I didn't seem to loose anything after the upgrade.  Maybe it removed the HP bundles that were installed for ESXi 4.1?

Removing a VMFS Datastore

To remove VMFS datastores in the past, I always made sure there were no VMs left on the datastore, right-clicked and chose "Delete".  Well, apparently thats the wrong way to do it!  I must have been lucky, as VMware claims this method could result in an APD (All Paths Down) state.  If you don't know what that is, let me tell you it's bad (I have experienced this but for a different reason).  Your host(s) will lose access to storage.

I stumbled upon this vSphere blog post that has the procedure to remove a datastore the right way: Best Practice: How to correctly remove a LUN from an ESX host

UPDATE: For ESXi 5.0, I found it better to follow the KB mentioned in that blog post:
Unpresenting a LUN in ESXi 5.x

For ESXi 5.0, here's how I do a slightly modified vesion of the procedure listed in the post above with more information on items such as HA:

1. Make sure all VMs are evacuated from the datastore/LUN.
2. Using Datastore Browser, make sure there aren't any left-over directories or files.  If there are, delete them (make sure they can safely deleted first, of course).  Exception: the HA directory - you'll remove that a different way later.  Also note that you won't be able to delete a file if it's in use.
3. Make sure all vSphere features are disabled for the datastore (i.e. SIOC, Storage DRS)
1. HA Datastore Heartbeating:  There may be some cases where HA has chosen the datastore you're trying to remove.  In this case, edit cluster settings and change Datastore Heartbeating to "select only from my preferred datastores", then select at least three datastores other than the one you're trying to remove.  I highly recommend changing this setting back to "Select any of the cluster datastores" after you're finished removing this one.
4. Next, for each host, go to Configuration\Storage\Datastores View, right-lick on the datastore and choose "unmount":
1. The "Unmount Datastore Wizard" appears. Make sure all hosts are selected.
2. Click next.  Hopefully the next step will look like this:
   
3. Click "Next" then "Finish".  After a minute the datastore will be grayed-out and italicized.
5. Then go to Configuration\Storage\Devices View, right-lick on the datastore and choose "detach"
1. You get a pop-up dialog window similar to the one show above.  Choose "OK.  The device will be grayed-out and italicized after a few seconds.
6. Go in to your SAN and remove LUN masking/unpresent the LUNs to the ESXi hosts.
7. Finally, right-click on your cluster and choose "Rescan for Datastores".  You may get a StorageConnectivityAlarm alert for every host in your cluster unless you disable this alert first.

I like to go into every host and check the storage adapter for the LUN just to make sure but I haven't found a problem with this procedure yet.

vSphere 5 Upgrade: vCenter and vUpdate Manager

Let me say upfront that had I tested the upgrade against clones of the productions VMs, I would have found these problems ahead of time (note to self for future upgrades!).

I followed the steps published in Its Time: vSphere 5 Upgrade so the first component to get upgraded was the vCenter instance at the recovery site.  This completed successfully - no errors - so far so good.

Next up, vCenter upgrade at the protected site.  The database upgrade step failed(!).  The installer gave a fairly generic error: "database upgrade failed. See the log at...".  The log only mentioned: "failed to execute dbuhelper.exe: an error occurred".

A quick Google search finds the log error posted in a VMware Communities post (TGfVC! - thank goodness for VMware Communities!).  The resolution: set the log file growth to unrestricted.  Sure enough, my log file was set to restricted so I changed it, re-ran the install and viola, success!

That evening, I reconfigured the vCenter Linked Mode feature but they never seemed to reconnect.  By time I came in to work the next morning, it had started working.  I must not have been patient enough(?), not sure but it has worked fine since.

Finally (for this step), vUpdate Manager upgrade.  This went smoothly just as it did in the lab.  Now that its working, I can start upgrading the ESXi hosts.

It's Time: vSphere 5 Upgrade

The technological benefits of upgrading from VMware vSphere 4.x to vSphere 5.x are not as great as upgrading VMware Virtual Infrastructure 3.x to vSphere 4.x in my opinion.  I don't know what the numbers are but I'm guessing that adoption hasn't been as fast as a consequence.

The latest XtraVirt poll posed this question with the following results:
What is your company's timescale for vSphere 5 deployment?
35%   6-12 Months
27%   1-3 Months
15%   3-6 Months
15%   1-2 Years
8%     No Current Plan

Well over half of admins responding will be upgrading within a year.  That's not bad.  I am in the "1-3 months" category and could argue that I've already started (planning and testing are complete).  The driver in my case is not technology/features-based, but is cost avoidance.

HP EVA SAN replication is licensed by capacity.  I don't want to rant about how utterly wrong this is but I'm not about to pay HP additional thousands of dollars for the privilege of replicating my data, thank you.  And now that SRM 5.0 provides replication for free, I don't have to (see previous post).

I've been planning our vSphere upgrade over the last month.  I've successfully installed and upgraded to vSphere 5 in the lab, now it's time to upgrade production.  I think VMware has finally got the upgrade process right.  It took them awhile although the vSphere 4.x upgrade was fairly smooth.  If you upgraded Virtual Infrastructure 2 to 3 you know what I mean!

I've pasted my upgrade plan below (taking out company-specifics).  I also included some notes from the lab tests. On to the show!

Phase 1: Planning

VMware vSphere Upgrade Center

http://www.vmware.com/products/vsphere/upgrade-center/

VMware vSphere Upgrade and Install Community Forum

http://communities.vmware.com/community/vmtn/server/vsphere/upgradecenter?view=discussions

(Note: I watched the forums and blogosphere in general to determine the stability of the release.  This is always a good practice but especially with a ".0" release - let others test it out.)

As of 9/15/2011, no real “deal-killer”/systemic errors found during or after upgrade.  Here’s the list:

1.       Port group names must be less than 19 characters (no impact to us)

2.       vCenter license must be provided during installation or it will start in expired mode (no impact to us)

3.       One admin gets an error when trying to upgrade ESXi via CLI – “downloading metadata failed” (no impact to us – we’re not using CLI to upgrade ESXi)

4.       One admin had a corrupted esx.conf file that prevented him from upgrading ESXi (no impact to us)

Check Pre-requisites:

1.       Check VMware Infrastructure Licenses

a.       vCenter 5.0  (available)

b.      ESXi 5.0  (available)

c.   Site Recovery Manager 5.0 (available November 19, 2011)

2.       Check Hardware Compatibility

a.       Server Platform

b.   SAN Platform

c.   HBA cards

d.   NIC cards

3.       Check Software Compatibility

a.       Review Guest OS Support

b.      Review 3^rd-party Product Support

                                                   i.      Backup Software

                                                 ii.      Management Software

                                                iii.      Custom Scripts

c.       Review Interoperability matrix

                                                   i.      http://www.vmware.com/resources/compatibility/sim/interop_matrix.php

PROJECT MILESTONE: Planning Complete

Phase 2: Testing

Setup Test Environment

Once the above steps are complete, it’s time to start testing.

(Note: The general idea here is to configure an environment that matches the current production environment as closely as possible.  I've done this in the past using cloned VMs with "mostly" successful results.  This is the way to go if you can keep the test and production networks isolated.  For this upgrade, I decided to configure new VMs - this should still give me valid tests and is faster to get up and running than using cloned VMs based on past experience.  If you have the time, you may want to go the cloned VMs route.)

Step 1: Setup ESXi Hosts

1.       Setup 3 servers as ESXi hosts in Datacenter1

2.       Setup 3 servers as ESXi hosts in Datacenter2

3.       Install vSphere 4.1 (or whatever your current version is) each host per the COMPANY Installation Guide.  (You do have this documented, right?)

Step 2: Setup SAN LUNs and Replication

1.       Create one LUN in Datacenter1 for vCenter and SQL VMs

2.       Create one LUN in Datacenter2 for vCenter and SQL VMs

3.       Create one read-only LUN in Datacenter2 to host the replicated LUN

4.       Configure the Datacenter1 LUN for replication to Datacenter2.

Step 3: Setup Test Virtual Machines

1.       Install one base Windows Server 2008 R2 image on one host as each site to be used as the VCenter/SRM server

2.       Clone images for use as SQL server to host the vCenter databases

3.       Configure images as stand-alone VMs.

Step 4: Configure SQL Server

1.       Install the same SQL Server version used in production both SQL VMs

a.       Use the production configuration as reference.

Step 5: Configure vCenter

1.       Install the SQL Native Client on both vCenter VMs

2.       Install vCenter on both vCenter VMs

a.       Use the production configuration as reference

b.      Configure vCenter with production licenses during installation or it will start in expired mode

3.       Add hosts to vCenter as appropriate

4.       Use vUM to apply the same updates to ESXi hosts as are present in production.

a.       TEST: This will test vUM to upgrade hosts

b.      TEST: This will also test vMotion capability

c.       TEST: This will also test the ESXi host configuration (networks, storage, etc.).

Step 6: Configure SRM

1.       Install SRM on each vCenter VM

2.       Configure SRM for the replicated LUN

3.       TEST: Perform a test SRM recovery to ensure system is functional.

PROJECT MILESTONE: Fully Functional Test Environment Complete

Upgrade to vCenter 5.0/SRM 5.0

Now with a fully functional test environment in place, it’s time to start upgrading.

The upgrade to 5.0 is supposed to be the least intense VI/vSphere upgrade yet.  The steps are fairly straight-forward:

1.     Upgrade vCenter/Upgrade Manager (vUM)
2.     Upgrade the ESXi hosts
3.     Upgrade VMware Tools of the VMs and the Virtual Hardware of the VMs to version 8
4.     Upgrade the VMFS datastores to version 5
5.     Upgrade the SRM to version 5
6.     Upgrade and 3^rd-party tools and scripts.

Step 1: Upgrade vCenter

1.       Backup the vCenter database using SQL Server Management Studio

2.       Backup the SSL certificates (%allusersprofile%\Application Data\VMware\VMware VirtualCenter)

3.      Stop all vCenter services

4.       Install JDK 1.6

5.       Using the vCenter ISO, upgrade vCenter to version 5

a.       Upgrade the recovery site first

b.      Run the vCenter host agent pre-upgrade checker

6.       Configure the new vSphere 5 licenses

7.       Upgrade the vSphere Client

8.       From the recovery site, rejoin the site via Linked Mode:
From the Start menu, select All Programs > VMware > vCenter Server Linked Mode Configuration

Step 2: Upgrade Manager (vUM)

1.       Backup the vUM database using SQL Server Management Studio

2.       Stop all vUM services

3.       Install JDK 1.6

4.       Using the vCenter ISO, upgrade vUM to version 5

a.       Upgrade the recovery site first

5.       Upgrade the vSphere Client vUM plug-in

Step 3: Upgrade the ESXi hosts

There are two options to upgrade the hosts: via vUM or doing a clean install with the OEM custom ESXi 5 image/ISO.  For testing purposes, we’ll try both methods.

1.       Using vUM, upgrade the first host of each cluster

a.       Test vMotioning VMs between 4.1 and 5.0 hosts

(Lab note: Need to force remediation.  Enable "remove incompatible packages".)

2.       Using the ISO, perform a clean install of the last host in each cluster.

(Note: I skipped step 2 - vUM worked so well I decided to go with that method.  HP provides an image that can be "imported" directly into vUM (very nice!) which worked great.)

DECISION POINT: Use vUM Upgrade or Clean Install from ISO

Step 4: Upgrade VMware Tools and Virtual Hardware

1.       Using vUM, upgrade the VMware Tools component of all VMs

2.       Using vUM, upgrade the virtual hardware of all VMs.

(Note: vUM scan skips vCenter and SQL VMs.  You'll need to go back and upgrade these manually.)

Step 5: Upgrade the VMFS Datastore

The VMFS datastores can be upgraded in place while the VMs are running.

Step 6: Upgrade SRM

1.       Snapshot VC and SQL VMs

2.       Upgrade SRM to 5.0 on recovery, then protected site vCenter VM

a.       Stop SRM

b.      Uninstall SAN SRA

c.       Uninstall the SRM plug-in

d.      Upgrade SRM

e.      Install the 5.0 SRA (if using array-based replication)

f.        Restart SRM

3.       Configure SRM

a.       Remove Array Manager

b.      Add new Array Manager

c.       Reconfigure protection group

d.      Reconfigure recovery plan.

4.       Install and Configure vSphere Replication (if not using array-based replication)

a.       At the PROTECTED SITE deploy the vSphere Replication Management Server (vRMS)

                                                   i.      Assign a static IP address

                                                 ii.      Register with the protected site’s vCenter instance

b.      At the RECOVERY SITE deploy the vSphere Replication Management Server (vRMS)

                                                   i.      Assign a static IP address

                                                 ii.      Register with the recovery site’s vCenter instance

c.       At the RECOVERY Site deploy the vSphere Replication Server (vRS)

d.      Configure VMs for replication via the vSphere client

5.       TEST: Perform a test SRM recovery to ensure system is functional

Step 7: Configure Network Monitoring Tool

Note: In our case, we configured 2 "applications" in our NetFlow-based network traffic monitoring system:

1. vSphere Initial: port 31031 (port used to do the inital "seed" copy of the VM)
2. vSphere Ongoing: port 44046 (port used to replicate changes after initial replication completes)

Step 8: Upgrade 3^rd-Party Tools

There’s little need to test tools that vendors have certified for vSphere 5.0.  However, custom scripts will need to be tested.

PROJECT MILESTONE: vCenter and Site Recovery Manager Upgrade Complete

Phase 3: Production Upgrade

Follow steps in Phase 2 to upgrade production.  

Conclusion 

That's it!  Depending on the outcome of your tests in Phase 2, this may become an iterative process.  Also note that there's always the chance that you've tested everything 100 times and still encounter an issue while upgrading production.  It's the nature of the beast.  However, having the experience of performing the upgrade in a test environment will give you a leg up in troubleshooting problems.  And, having worked with VMware support numerous times, I can recommend calling them without hesitation.

Finally, always remember to document things along the way.  Follow these steps and you will be in good shape.

vSphere Memory Utilization

Determining actual host memory utilization can present a challenge.  The issue boils down to when the host breaks large memory pages into small pages which can then be shared (aka Transparent Page Sharing).  TPS only works with small pages.

To determine actual host memory usage, especially after the host is using a significant amount (65-70%+), you have to monitor an esxtop parameter called COWH.  For example, you may get to 70% utilization and find adding another host to the cluster doesn’t decrease utilization, but adding another VM or two does(!).  This is because at some point the host will  break some of the large pages into small pages and then TPS kicks in.

For more in depth reading see:

Large Pages, Transparent Page Sharing and how they influence the consolidation ratio

Large Pages – a problem of perception and measurement

Large Pages, theory and usage with SQL Server 2008

Changes are a Comin'

Changes are always coming!  There are few things as consistent in IT as change.  The first "system" that I fancied myself an "expert" on was Novell NetWare 3.11.  Then LANtastic, Then Windows NT 3.51.  Then... well you get the picture.  IT technologies, strategies, directions all change over time.  I've turned down job offers when I thought it might lock me into any one product or technology for any length of time.

So this year will be no different.  When I read the feature list for Microsoft Hyper-V 3.0 a couple of months ago, I started telling my peers to keep their eyes open.  For 2012, VMware will remain the market and technology leader.  I have no doubt about that.  But competitors are catching up.  It happens all of the time (i.e. IE vs. Netscape).  The march of innovation will continue to force change on all of us.  That's one of the things I enjoy about IT and technology in general.

I was inspired to write a few thoughts here after reading Mr. Ruben's article on his virtualization landscape predictions for 2012.  Check it out here to read more.