ERROR: Repeating ADWS Errors Every 1 Minute

One of the errors my vCheck report enumerated was the following:
Active Directory Web Services encountered an error while reading the settings for the specified Active Directory Lightweight Directory Services instance. Active Directory Web Services will retry this operation periodically. In the mean time, this instance will be ignored. Instance name: ADAM_VMwareVCMSDS


This error was repeating everyone one minute - needless to say there were quite a few entries.  Apparently this bug has been around since vSphere 4 but hasn't been fix.  Luckily the fix is easy.  Go to the following registry key:
HKLM\SYSTEM\CurrentControlSet\services\ADAM_VMwareVCMSDS\Parameters

Find and delete the "Port SSL" value (the value data should be empty).
Create and new DWORD value with the same Port SSL name.
The value data should be 636 decimal.
Restart the ADWS and VMware_VCMSDS services.

I did the procedure and checked the ADWS log again - problem solved.

That's it!  Er... wait,  new problem found in ADWS log.  Here is the error:

Active Directory Web Services could not find a server certificate with the specified certificate name. A certificate is required to use SSL/TLS connections. To use SSL/TLS connections, verify that a valid server authentication certificate from a trusted Certificate Authority (CA) is installed on the machine.

A quick Google and I found this in the VMware Communities:

This message is simply an informational message and should have no major impact on the running of the Virtual Center Server. The only ways to stop this message from appearing would be joining vCenter Server to a AD Domain. Btw, you CANNOT install AD Domain Controller on the same machine with vCenter, it will not work. Because vCenter 4.1 will install an instance of ADAM (Active Directory Application Mode). It uses this when you use vCenter Linked Mode and ADAM will conflict with its’ own AD services if the server is also a Domain Controller.

Okay so basically ignore it.  Hopefully this doesn't fill up the vCheck report.  Just something we'll have to keep an eye on.

2012 VMware vExpert!

I just received confirmation that my vExpert status has been renewed for 2012!  My home lab and I thank you!   ;)
Now it's time to start earning it for 2013.

COOL TOOL: vCheck

vCheck:  http://www.virtu-al.net/featured-scripts/vcheck/

I saw this mentioned on another blog (I don't remember which one) over a year ago and thought it looked good, but didn't provide much info beyond what I was getting with VKernel and RVTools.  Now I'm wrong.  Alan over at http://www.virtu-al.net wrote a PowerShell script that checks the health of your vCenter environment.  It has recently been updated to handle plugins.  Most of the checks have been converted to plugins, and now there's an Exchange plugin written by Phil Randal.  I would not be surprised to see other plugins for other systems like AD and SharePoint in the future.

I'm running most of the VMware and Exchange checks.  This is providing me information beyond the VKernel and RVTools tools I currently use.  I've set it up to run once per week and email the report to our VMware administrators distribution list.

In just the first week is has brought several problems to light which will be subjects of future blog articles(!).
I highly recommend!

vSphere 5 Upgrade: SRM - PART 1

Per Its Time vSphere 5 Upgrade, time to upgrade SRM.  Well, I got to step 6.2 and things went downhill from there.  The following is the description I used to open an SR with VMware support:

Upgrading SRM from 4.1.2 to 5.0.  I get the error: "failed to create database tables".  It doesn't appear to make any changes to the database.
I've double-checked settings per KB1015436 and several communities postings.
I've tried re-installing SRM 4.1.2 (successfully), performing a repair, then another upgrade but it still fails with the same error.

The first support tech went down the 'invalid permissions' path but this was not the problem.  Turns out, the SRM 5.0 upgrade does not support upgrading from SRM 4.1.2!  Interesting because the only documentation I can find on the subject clearly states that you can upgrade from SRM 4.1(!).  Looks like VMware needs to do a better job documenting these requirements.

I was then informed that I could wait until the next minor/point release of SRM 5 which would support upgrading from 4.1.2, but I didn't have that kind of time (and who knows when they'll actually release it).  So no upgrade for me, full install from scratch instead!  Besides have to reconfigure mappings, protection groups (which I was going to have to do anyway), etc, the biggest downside is losing the previous DR test results.  Yes, I saved those off as separate Excel files, but it would have been nice to have had all of the results right there in SRM from the beginning.


But wait, there's more!  Now that I have a brand new freshly installed SRM up and running, it's time to setup vSphere Replication.  Did that go problem free you ask?  Ummm, no.  The following is the description I used to open yet another SR with VMware support:

The VRMS servers at both sites fail to connect.  I have unregistered the server, powered down/deleted the appliance VM, re-initialized the VRMS database, repaired SRM, redeployed the VRMS servers and configured them with the same vCenter FQDN per KB2007463 but still have the same problem.

Between the support tech and I it took several hours to figure this one out.  The short of it is that it's a vCenter certificate problem.  What clued me into this was the error I got when registering the VRMS instance:

That "unacceptable signature algorithm" message is not your typical self-signed cert warning!  Turns out, my vCenter self-signed certs had expired.  This hadn't caused a problem until installing vSphere Replication - it wants at least a current/non-expired cert.  I checked the vCenter cert and sure enough, it had expired in 2010.  It was created in 2008 and was valid for only 2 years!

Now I bet you're wondering, how does one fix this cert problem?  Well that's easy, reinstall vCenter!  And repair won't work either so you have to uninstall the current vCenter instance and re-install a new one.  Luckily most settings are maintained in the vCenter database so this could have been much more painful.

While I was at it I checked the new vCenter cert and VMware apparently decided to make this one valid for 10 years.  Now that's more like it!

But wait, there's more!  Look for part PART 2 of this adventure in a near future post.  A little hint - the fun ain't over yet.

vSphere 5 Upgrade: VMFS Datastores

Per Its Time vSphere 5 Upgrade, time to upgrade VMFS datastores.  Like the last couple of steps this one completed w/o issue.  Note that it is better to create VMFS datastores because you'll get a 1MB block size regardless of what size datastore you're creating, optimizing disk space.  Compare this to the 2-8MB block sizes required based on datastore size in ESXi 4.1 and earlier.

All of my datastores now report to be VMFS version 5.54.

vSphere 5 Upgrade: VM Tools and Virtual Hardware

Per Its Time vSphere 5 Upgrade, time to upgrade VMware Tools and Virtual Hardware.  I'm happy to report that both of these steps completed w/o issue.  After the previous 4.1 upgrade, the VMware Tools upgrade corrupted one of my Windows 2000 VMs.  VMware claimed it was a Microsoft problem, Microsoft claimed it was a VMware problem.  Love the finger-pointing guys!  Anyway, I made sure this VM was upgraded (actually replaced) with a Windows 2008 R2 server and we updated the application while we were at it - a win/win in my book.

No problems this time (phew!).

The only other thing I did differently compared to my previous upgrade was upgrade the SQL and vCenter servers first.  vUpdate Manager will skip these VMs anyway.

vSphere 5 Upgrade: ESXi Hosts

Per Its Time vSphere 5 Upgrade, the ESXi host upgrades.  Not much to report here.  The upgrades completed w/o issue per vUpdate Manager.

Note that I had to force remediation.  Then I enabled "remove incompatible packages".  I didn't seem to loose anything after the upgrade.  Maybe it removed the HP bundles that were installed for ESXi 4.1?