Showing posts with label virtualization. Show all posts
Showing posts with label virtualization. Show all posts

Friday, September 8, 2017

A New Day, A New Blog

Just a quick note to let everyone know that I have decided to post articles related to VMware, virtualization and cloud computing to a new blog:  http://vmclouds.e-wilkin.com

I will leave current posts as-is but new content will be posted to the new blog.  I will continue to post my other ramblings here moving forward.

Posted by at No comments:
Labels: ,

Thursday, July 28, 2016

VMworld 2016: The End of an Era

I sadly report that I will not be attending VMworld 2016 this year.  This will be the first conference I've missed - "13" really is an unlucky number!  I was not selected per the criteria my organization used to determine who goes and who stays.  I respect their decision, but that doesn't change the fact that I will be watching the general sessions remotely on a computer screen.  Yeah, that won't quite be the same - no assisting customers, meeting friends, attending sessions or keeping up with what's going on with partners in the Solutions Exchange this year for me.

My passion for the conference won't change - I still believe it to be the best IT conference and highly recommend attending.

So the VMworld Alumni Elite group will be at least one less this year.  What a prestigious group!   I know from meeting with these guys year after year that they have a real passion for VMworld, VMware and its products.  Guys - I'll catch up with you next time!

Its ironic - during my interview I was told that the surest way to not attend VMworld was to become a VMware employee.  My first year VMware decided to send my entire organization!  My second year they decided to send most of the organization, yours truly included.  This is my third year and it appears as though my luck has run out.

I did try multiple alternative options but none of those panned-out.  So all good things must come to an end?  C'est la vie - its the end of an era... and the start of a new era - carpe diem!

For those that are going - have a great conference!  For the reset of us, make sure to tune in to the general sessions which will be live-streamed again this year.  And don't forget about the breakout sessions that will be made available shortly after the conference.

Posted by at No comments:
Labels: , ,

Thursday, February 11, 2016

Enabling the Digital Enterprise: VMware Announcements

VMware made several major product announcements this week and I'm super-excite about some of the improvements and new features coming our way.

Our EUC team is really firing on all cylinders and continues to update and integrate products.  In this case AirWatch, Horizon and Identity Manager are brought together with Workspace One.  Did you know we coined the term "Workspace" (well okay, a company we acquired years ago did).  I used to be one of those guys that connected to my View desktop and worked from there.  Now, more often than not I browse our internal Workspace One portal and launch whatever app I need to get work done.  How did I work without this before?

Also worth noting are several SDDC product updates.  The big news here is VSAN 6.2 - what an awesome release!  VSAN was ready to host your tier 1 applications with the release of 6.0 - now with 6.2's dedupe, compression and RAID5/6 features, the nay-sayers won't be able say it's not ready for the enterprise (well, they can but they would be wrong).

Here is a "Did you know" PSA: 
Did you know that if you use vSphere Replication to replicate VMs hosted on VSAN storage, you can set the RPO as low as 5 minutes?
Now you know.

Finally. take a look at the vCloud Suite pricing and packaging changes.  We've really change the Suites to refocus on what customer's really want - the tools needed to run a true SDDC datacenter (wait, isn't that redundant?).
Replay’s of this week's events are available here: http://www.vmware.com/digitalenterprise

Posted by at No comments:
Labels: , , , , , , ,

vROps Alert: One or more ports on the Distributed Port Group are experiencing network contention due to dropped packets

Here is a vROps problem experienced by one of my customers:
Alert: One or more ports are experiencing network connection
(One or more ports on the Distributed Port Group are experiencing network contention due to dropped packets)

There was no evidence in vCenter that packets were being dropped.  The customer found KB2105109: VMware vRealize Operations Manager counters report false dropped packets on vSphere Distributed Switches with the alert: One or more ports are experiencing network contention

However, this seems to only apply to vROps 6.0.x and the customer is running the 6.2.x release.  An SR was opened and the support tech recommended the fix per KB2052917: vCenter Server 5.1/5.5/6.0 performance charts report dropped network packets

Okay great, so a patch needs to be applied to the ESXi 5.5 host.  The customer tried to applied the patch via VUM but it was marked obsoleted.  "Obsoleted" is an Update Manager compliance state.  As explained in the Installing and Administering VMware vSphere Update Manager documentation:


This compliance state applies mainly to patches. The target object has a newer version of the patch. For example, if a patch has multiple versions, after you apply the latest version to the host, the earlier versions of the patch are in Obsoleted By Host compliance state.


(https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.update_manager.doc_50%2FGUID-EBAA4F4A-57E0-45ED-8730-4B851FC846A9.html)

Hmmm.... so the host has a newer version of the patch, so now what?

After some discussion we landed on the fact that the vCenter instance managing this host was likely the culprit as it has a patch level several versions behind the host.  I always recommend customers keep their vCenter and ESXi hosts at the same Update patch level.  Installing patches on hosts in-between major Updates is okay.  In this case, they were on different Update versions.

The vCenter instance was patched to the latest update and voila, the error disappeared from vROps.



Posted by at No comments:
Labels: , , , ,

Monday, February 1, 2016

Horizon View Restart/Reboot Sequence

View admins - please make sure you restart View infrastructure servers in the proper order for fastest uptime!  I recently had a customer that rebooted vCenter and View servers in an effort to fix an apparent connection problem.  When they did not appear to be coming online fast enough, the admin rebooted them again.

Now normally rebooting these servers can cause ADLDS synchronization to take up to fifteen minutes or so, but rebooting them again, before the initial sync completes, can cause them to take thirty minutes or more - and then leave inconsistencies within the pools.

For more information on the proper boot sequence:
Restart order of the View environment to clear ADLDS (ADAM) synchronization in View 4.5, 4.6, 5.0, and 5.1

And to clear out any inconsistencies:
Manually deleting linked clones or stale virtual desktop entries from the View Composer database in VMware View Manager and VMware Horizon View


Posted by at No comments:
Labels: , , , ,

Thursday, August 27, 2015

VMworld 2014 Recap

Well, VMworld 2015 is next week so I guess I'd better get this one published!  I actually found this sitting in my drafts, I was waiting for a picture to come through which took so long I forgot to go back and publish it after receiving it!  So here it is in all it's glory...

This was my first VMworld as an employee.  My new role gave me a unique perspective to the conference.  Having attended every previous conference as a VMware customer has given me knowledge and experience I can pass on to my customers so they can get the most out of the conference.

Consequently, I walked through the Solutions Exchange but didn't spend as much time there as I normally do.  I attended a whopping one break-out session, and all of the general sessions.  Why?  Because I spent most of my time with my customers - assisting them in some form or shape.  I also spent a lot of time at TAM Customer Central.  What is TCC you ask?  It's a top secret location we reserve for VMware TAM customers where we provide special break-out sessions not available to VMworld attendees, receptions and meeting rooms among other things.

The theme this year was NO LIMITS.  Here is some cool desktop wallpaper for you:

And here is the Moscone all dressed up:

We were asked to provide pics of us with our customers for use as a collage at VMworld to be displayed at various times such as just before and after the general sessions.  Here's the pic I sent in with "Mr. Corvette" Jarod and his red sports car that made us mini-celebrities for the week:

Another one of those TAM customer benefits is the Hands-on Labs VIP tour.  This gives select customers a behind-the-scenes look at HOL operations and how the whole thing is managed.  Check out these stats:


Its interesting that the NSX lab remained the most popular lab during the entire conference - indicates a lot of interest by our customers.  And, consequently I think, this product continues to gain momentum in the marketplace.

And pics of how we used Log Insight and vCOps to monitor the environment via custom dashboards:




Even though I attended as a VMware employee this year, I was still invited to the CTO party (I had customers in attendance, so I'm sure that helped):

Why yes Johnny, everything is better with bacon:




And a pic from the Alumni Elite pre-party party - the pic I was waiting for as it was taken by someone else's camera - which wouldn't be complete without a shot with the big guy (me=literally, Mr. Gelsinger=figuratively):

And finally, lookie at what I found:

I found a "Meet the Experts" area hidden (not really) on the third floor of Moscone West.  If you have any technical or strategic product questions, these are the guys to ask and anyone can sign-up.  I highly recommend this resource if we offer it again in 2015.

Here's to seeing you there next year (he says with bacon in one hand, bourbon in the other)!

Posted by at No comments:
Labels: , ,

Friday, August 22, 2014

Back in Black!

I hit the sack.  It's been too long I'm glad to be back...

Yes, it has been one year since I've blogged on VMware and virtualization.  Well, in my defense, a lot has happened keeping me from spending quality time with my beloved blog.

My biggest excuse = I joined VMware!  I am a Technical Account Manager for customers in the Dayton/Cincinnati area.  Shameless plug = if you would like to know more about what a TAM can do for you, please reach out to me by leaving a comment and I'll get back to you.

I was warned that the first year is like drinking from the fire hose and I found that to be true, but in a good way.  So yes, I did neglect my blog to the point of having to delete drafts that are no longer relevant.

But the good new is:  I'm back!  I've personally re-committed to posting articles on a more-frequent/regular basis (although not on a specific schedule, no change there).

Note that the opinions in my blog are mine alone and do not necessarily reflect that of my employer.  Also note the subject matter won't change either - I won't post "newsie" articles as there are too many other blogs that do a good job of that.

What I will publish are articles describing technical problems I'm seeing in the field along with their resolution.  I may also post articles of a more strategic nature on occasion.

My experience with customers in the field gives me a unique perspective (and fresh content!) on many IT topics.  Three of my accounts are in healthcare so I may also publish articles discussing challenges specific to that segment.

Finally, next week is VMworld 2014!  I'm doing things a little differently this year - instead of one VMworld recap article (which I may still publish), I'm going to try to post bits through-out the week so stay tuned!

Posted by at No comments:
Labels: , ,

Sunday, February 2, 2014

Post-VMworld 2013 Recap

Okay, yes, it's been too long coming, but it's finally here - my VMworld recap.  Now, I have a good excuse, but you'll have to read the next post for that!  Note that most of this was written shortly after the event, so in that since the content is "timely".  So here it is, enjoy!

Theme: Defy Convention - on the steps, pretty cool:

Ten years baby, yes!

There was the "10th Annual" wall this year where we all got our pics posted (and by all, and mean every attendee that wanted to have their pic posted).  We were one of the first pics taken so they're at the bottom of the "1".  Can you see me?


No?  Okay, then here's a close-up:

VMworld seemed to be about one word for VMware this year: focus.  I could see the focus (or re-focus in some cases) on the entire stack from the virtual infrastructure to hybrid cloud computing.  This focus seems to be coming from a renewed customer focus = they're really listening.  Proof-in-point = The shifting of vCloud Director features to vSphere and vCloud Automation Center, leaving the multi-tenant capabilities for cloud providers.  I really believe we have Pat Gelsinger to thank for this along with his management team.  With this renewed focus on customers, market and product/feature mix, I think VMware has really put itself in the position to continue to be on the forefront of innovation and an agent of change in the IT world.

Here are some other words that really defined VMworld 2013:
SSD
Flash
Software Defined x (datacenter, networking, storage, etc.)
Convergence (this time, as in compute, network and storage)
Imagine Dragons/Train
Alumni Elite (34 left!)
Server-side flash
Server-side cache
VSAN
vSphere 5.5
NSX
Did I mention SSDs?

Location: San Francisco this year and again next year - I really have nothing else to say about it.  I go there for the conference.  I actually prefer Vegas, but to each his own.

Keynotes: As I mentioned, this year was about a renewed focus on product mix and the datacenter stack including the following four areas - Compute, Storage, Networking and Operations.

The big story in Compute is the announcement of vSphere 5.5.  This will be a big release with a many improvements and new features.  I won't detail them in this post but you can read about them here:
https://www.vmware.com/resources/techresources/10385

The big story in Storage was, of course, VMware's new VSAN feature/technology.  And they announced a public beta you can download here:
http://www.vmware.com/vsan-beta-register

VMworld App:  Much better this year.  WiFi still weak during the general sessions, and it didn't help that the app was updated a half-million times during the event but I suppose there were a lot of schedule changes(?).

Alumni Elite Event:  Whoa!  The 10 year Alumni Elite event was, in a word, awesome!  They bused us down to Candlestick, er, AT&T park an hour before the VMworld party.  We got our own locker in the Giants locker room, along with great food and drink.  I'm not a big fan of clam chowder but it was so highly recommended I figured I had nothing to lose.  Holy cow, it was fantastic!

There were many highlights on this night, meeting VMware's CMO, Robin Matlock, was certainly one:

Getting to talk shop with VMware's CEO, Pat Gelsinger again this year was fantastic:


I also got to meet VMware's COO, Carl Eschenbach.  Let me tell you, all three of these executives were very down-to-earth, highly energetic, and very excited about VMware and the future of IT.  I really didn't expect them to be as genuine and accessible as they were but I was pleasantly surprised.  I really shouldn't have been since VMware is a great company that attracts great talent.

Later that evening we get to try the batting cages - 60MPH pitches, not a bad workout.  Out of six pitches, I made solid contact with three of them, whiffed one, got out of the way of one and fouled the last.  Pat is up after me and let me just say he made me look bad, and this was supposedly after only two hours of sleep!

Finally, I was told there are 34 of us left.  I only count 27 in this picture, so several didn't make it to the pre-party event.  We seem to lose about 10 per year.  25 next year?  I wonder who will be the last one standing?

VMworld Party:  Time to head out to the party:

It was a real "festive" atmosphere.  The entertainment for the night was Imagine Dragons and Train.  I thought both groups put on a great show.  Train was a bit of a surprise for me - I didn't realize how entertaining they would be nor did I realize how many of their songs I know until I heard them:
 

Solutions Exhibit:  There was another record number of exhibitors this year.  As always I highly recommend hitting the floor and talking to as many vendors as you can.  It a great way to discover new tools or solutions you may not have known even existed.  No unique schwag to report on this year, however there was still a lot of it if you wanted it.  And again, I didn't win anything - not that I tried very hard though.  There's always a group that hits it really hard.  I guess I don't have the patience, especially when I know the odds are quite high against me.

Attendance:  If there's one thing that's consistent about VMworld, it's that the attendance numbers go up every year and this year was no different:  over 30,000 - another record.

Break-out Sessions: I attended more of these this year than previous years and found them to be very useful.  This encourages me to go back to VMworld.com and view the ones I missed.  If you can't make VMworld, I highly recommended buying a subscription to gain access to the sessions.

Hands-on Labs:  These are all about timing - if you go at the right time, there's little-to-no line.  On the other hand, I saw lines that wrapped around the corner and outside of the building(!).  I'll try to take note of the best times next year, but for now my advice is if the line is long, keep checking back.  On a positive note, I did notice the line seemed to move at a decent pace.

Adventure:  So we asked a cabbie to take us to Gordon Biersch brewery/restaurant.  We had gone there the previous year and they had good food, great beer and scenery over-looking the bay.  So he drops us off at what looks like the right spot and takes off.  Walking up the steps we saw this:

Gordon Biersch = closed.  New Mozilla offices = opened.  At least it was tech-related?  Weird.  As we were walking back toward the pier, we stumbled upon an indoor/outdoor restaurant with a decent beer selection, so not all was lost!

So another great VMworld again this year.  What will next year bring?  All we know for certain is that it will be in San Francisco at the Moscone Center, August 25th-28th.

One last thing... why were there Hyper-V books in the store?

Until next year...
Posted by at No comments:
Labels: , ,

Wednesday, January 16, 2013

vSphere 5.1 Upgrade: Home Lab

Being a vExpert has its benefits, one of which is getting a year-long evaluation license for VMware vCloud Suite 5 Enterprise.  Since this includes vSphere 5.1 (vCenter and ESXi), I was inspired to get my home lab back up and running again.

Heads-up warning:  this stuff is never as easy as it's supposed to be.  We had an old saying at my previous employer: "it takes 100 hours"!  When someone complained that they were spending too much time on a task, the joke was always, "did you put your 100 hours in?"  Anyway, I digress...

Setting Up the Environment
The first step was getting my old/used Dell 2950 server racked and connected.  Racking it up and connecting to a KVM was easy.  Connecting it to my network was not.  I recently moved my old rack out of my office to an unfinished part of my basement - with no network drops, of course.  So I bought a small 4-port GbE switch and 50ft cable with the idea of running the cable to my existing switch still located in my office.  Since my office is in the finished part of my basement, I needed to run the cable through the ceiling and down the far wall.  I had done this several times using a draw-string to pull the cable through.  Well this time the string decided to break - I heard a snap! - the cable didn't even make it half way.

Great, now I have to figure out how to fish the cable through and find it to pull it back down near the area where my switch lives.  Long story short, there went my first 50 hours, but I did finally get the cable and new pull string through.  Did I mention that I hate pulling cable?

Physical Server Installation and Configuration
Next up - updating BIOS and component firmware of the server.  This went w/o a hitch.  The Remote Access Card works great, no need to stand in front of the server in the basement.

I then installed ESXi 5.1 to a 2GB Kingston flash drive plugged into the back of the server.  Again, installation completes successfully.  I rebooted the server, connected via the vSphere client and configured the standard settings - NTP time, local datastore, etc.

Installing the First VM
I created a VM to host the Active Directory domain controller.  I decided to use Windows Server 2012 since this was a brand new install in a new home lab network.  This allows me to explore/learn the new features of AD on Sever 2012 and give me some future-proofing.

Installing vCenter 5.1
With the domain controller up and running it was time to create a VM for vCenter 5.1.  I'm using the simple installation method since this is for a home lab and I'm keeping all of the components on the same VM.  I decided to use Windows Server 2012 again to be consistent, if for no other reason.  I know this is not supported per the compatibility list, but have read other admin using this version with success.  Not my luck though.  I got an error message stating the the installation was interrupted before it could be completed, and that I should check the logs.  Hmmm... I tried again using a different service account during the installation but no luck.  I even tried logging on as the local administrator security context and still no luck - each time I got the same error.  So vCenter 5.1 simple install on Windows Server 2012 = FAIL.  There goes another 25 hours!

At this point, I've decided to try the vCenter Server appliance.  This should actually be the easist and fastest way to deploy a new instance of vCenter, right?  There are other benefits as well: this will be the only option within the next several releases, the appliance is easy to upgrade, it includes a better, built-in database (vPostgreSQL) and most of the limitations of the 5.0 vCenter appliance have been removed.  Seems like a no-brainer.  Too bad it didn't work either(!).  I'm not sure how a brand new instance deployed on a brand new install of ESXi can fail, but it did.  The OVA deployed fine, vCenter appears to start fine, but 2 plugins failed to start: "VMware vCenter Storage Monitoring and Reporting" and " vCenter Service Status".    The error is "the request failed because the remote server took too long to respond".  Manually enabling the storage monitoring plugin works, gut the service status plugin still fails.  I'd had enough of that so I deleted the VM.  There goes another 10 hours!

Since that experiment failed, I wanted to go back to using a Windows OS which is not a bad thing since I suspect that this is what most environments will be using for some time to come.  Getting back to using a compatible OS, I deployed a Windows Server 2008 R2 VM.  Again, I start a simple installation of vCenter.  This time, I get a different (SSO) error:
Error 29148.STS configuration error.

What the heck!?!?  This is a plan, vanilla install on a newly built VM!  Now I'm losing confidence of vCenter 5.1 and SSO.  I decided to try again, this time choosing to use the local network account for the service account and making sure I choose as many defaults as possible (although I was mostly doing this previously).  And finally, SUCCESS!!! (And final 15 hours!)

I logged on to the new vCenter instance via the vSphere client, added and licensed the new ESXi host and everything looked good.

Conclusion
Whew!  At the end of the day, I wished I could have used the vCenter appliance - it would have been more than enough for my little home lab.  But the Windows-based vCenter still works great and now I'm ready to move on.

Next up:  Web Client and Update Manager installations.  I don't expect any problems (famous last words).  Later this week or early next week I'll post how this went along with the upgrade in my company's dev/test environment.  Quick preview:  SSO issues galore.

Posted by at No comments:
Labels: , , , , ,

Monday, January 14, 2013

Extending A vSphere Replicated Virtual Disk

I recently had a VM that needed one of its virtual disks extended.  vSphere Replication needed to be disabled for this disk before vCenter would allow this operation.
When reconfiguring replication for this VM/disk, it detected the original/smaller disk:
Duplicate File Found. Do you want to use this file as an initial copy?
If you choose yes, it will try to use this file instead of re-sending the entire virtual disk.  This option didn't work. I think the files are just too different (size) and it doesn't know how to handle it.

If you choose no, you can configure a different datastore, but it won't let you use the same one.  This would leave the original replicated virtual disk out there unnecessarily taking up space.

I ended up manually deleting the original replicated VMDK and had VR resend the virtual disk again.  No big deal this time as it was just Disk 0/C: drive but this could be a real PitA if you need to extend a larger disk.

Lessons learned:

  1. Size your drives properly the first time.
  2. Consider creating a new disk instead of extending an already large virtual disk (30, 40, 50GB+).





Posted by at No comments:
Labels: , , , , ,

Thursday, August 2, 2012

Cloud What?

Here's how cloud computing is defined by the Wiki:
"Cloud computing is the delivery of computing and storage capacity as a service to a community of end-recipients. The name comes from the use of a cloud-shaped symbol as an abstraction for the complex infrastructure it contains in system diagrams. Cloud computing entrusts services with a user's data, software and computation over a network."
Yeeoouzzza!  That probably means a little to a few.  Let's simplify terms in an effort to reach a better understanding.  We'll start by using an example:
Take that desktop computer sitting under your desk and put it in the datacenter (which I'm going to assume is in the same building as your office).  See that?  Your desktop is now in a private cloud.
Take that same desktop and move it into a datacenter owned by any other company other than your employer.  Where is it now?  It's in the public cloud.

When you read or hear about the "cloud", it's usually in the context of the public cloud.  And this is really another way of talking about something that is accessed over the Internet.  That something can be a software service such as email (Google Gmail), a platform service such as web services, or an infrastructure service such as a virtual server (Amazon EC2).

So why do we have this word "cloud"?  We had hosted email services over the Internet long before anyone coined the term.  Why not just call it Internet-hosted email?  Or Internet-hosted virtual servers? Etc, etc.  I think I know the answer: it's virtualization's fault.  More specifically, it's VMware's fault.

When VMware virtualization started becoming mainstream, there was a desire by many in the IT industry to jump on the next big thing.  And the desire was to jump ahead quickly.  We got server consolidation, okay great!  Now we have hosted dev/test environments, spectacular!  Now have significantly cheaper DR solutions, awesome!  We can now host desktops, especially for apps where Terminal Services doesn't fit the bill (later called VDI).  So what's the next big thing?

My observation is that several IT vendors, led by VMware, have started this cloud trend/marketing machine and most other vendors have jumped aboard the hype train.  Now we have vendors clamoring to be your cloud-based solution and it's almost become a given that everyone is looking at how the cloud can help their organization.

And this is where one of my motivations to write this article comes from.  Is this cloud thing all hype?  No.  There are some scenarios where it does make sense.  A common scenario is the organization with a need that their small (or no) IT staff can fulfill.  There are certainly cases like this where I would have no problem recommending cloud solutions.

One of the common arguments for choosing cloud services/solutions is to free-up IT staff to focus on the job they're supposed to be doing (or something similar to that).  Well, what's really happening is that when companies move an internally hosted solution such as email to the cloud, the CFO want's to know the financial pay-off, the ROI.  In many of these cases, somebody gets fired.  Maybe this wouldn't happen if we weren't in the Great Recession, but that's little comfort to the Exchange admin that just lost his job!

We need to start thinking of cloud solutions as outsourced solutions.  Not as silver-bullets that are going to bring peace and harmony to an otherwise functional IT department.

Here's another point of consideration:  if the vendors pushing cloud solutions are correct, shouldn't all of us IT folks be rushing out to interview with cloud service providers right now?  If the cloud utopia does occur, IT departments will look very different - much smaller with IT admins acting more as liaisons and requiring much less technical knowledge.  It's hard to believe this will really happen and is certainly open for much debate, but food-for-thought nonetheless.

Finally, my advice is this: the next time a vendor is trying to sell you the cloud, ask yourself these questions:

  • What's in it for you?
  • Will it save you money or cost more?  Hard costs?  Soft costs?
  • Is it similar to something you already have in place?  If so how is this "better"?
  • Why is this cloud service/solution better than hosting it in-house, managed by your existing IT team?

Whenever and wherever you encounter the term "cloud" your mental little red flag should pop up and warn you to proceed with caution.  A little up-front technology skepticism will help you make sure that your decision is the right one.  Again, I believe cloud services and solutions can make sense in some cases.  However, I suggest questioning the status quo - don't assume that just because vendor xyz is pushing the technology that its the right for your organization.
Posted by at 2 comments:
Labels: , , , ,

Monday, June 25, 2012

ERROR: VMware ESXi with 3PAR SAN and Dead LUN 254

Problem

Last week I discovered  a couple of error messages in the vmkernel.log file that caused me some concern:

2012-06-22T18:18:45.806Z cpu17:939673)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:972:Could not select path for device "Unregistered Device".
2012-06-22T18:18:45.806Z cpu17:939673)WARNING: NMP: nmpPathClaimEnd:1195:Device, seen through path vmhba2:C0:T2:L254 is not registered (no active paths)

I did a "esxcfg-mpath -l" and found 4 dead paths - 2 to each fibre HBA.  I never provisioned a LUN with an ID of 254.  Maybe this was a "special use" LUN?  I doubted it because my HP EVA has one of these and the device is listed in vCenter.  However, there were no devices with LUN ID of 254 listed anywhere in vCenter, only 4 dead paths.

Solution

After a focused Google search, I found the answer.  The HP 3PAR guy that came out and did the installation had us use a host persona of "1 - Generic" when we should have used "6 - Generic-legacy".  Luckily these can be changed on the fly via the InForm Management Console.

After making the change in the IMC, I rescaned each of the hosts and the messages stop appearing in the vmkernel.log and the dead paths were no longer listed in the vSphere Client.

Here are the relevant sites I found per the Google search:

And, of course, I always recommend following the manufacturer's best practices:

Looks like the guide was recently updated and it does recommend using the persona of "6 - Generic legacy".

Another mystery solved.

Posted by at No comments:
Labels: , , , , , , , , ,

Thursday, June 14, 2012

ERROR: The query service is not available or was restarted

I wasn't getting any results when going to the Hardware Status tab for all hosts in my recovery site.  When clicking "update", I'd get the error:
The query service is not available or was restarted. Please retry.

Of course retrying doesn't work.  Hardware status worked fine for all hosts in my protected site.  I thought maybe it was a problem with linked mode in vCenter so I logged on to the vCenter server in the recovery site, fired up the vSphere client, opened the Hardware Status tab and got the same results.

I then started another instance of the vSphere client and logged directly on to the ESXi host.  The hardware sensor data worked fine here (it's not in a separate hardware tab, but looks nearly the same).  Hmmm.... must be something with vCenter?

I Googled the error and found this VMware KB:

Well it's the exact same error message so this must be the fix, right?  Wrong!
First of all, step 14 is incomplete.  Please follow these steps to reset the vCenter Inventory database:

Secondly, this was not the only problem and probably didn't ultimately fix the issue.  I found this link in the same Google search:

The above forum posting had a link to the following web site with instructions on updating the ADAM instance vSphere uses for linked mode:

While this was for 4.1, the same settings apply to 5.0.  The only thing I would recommend is checking all of the common name (CN) properties to make sure the FQDNs are correct.  You do not need to change these to IP addresses!

I did have to reboot the vCenter server in the recovery site after making the changes.  Even then, it didn't seem to start working until the following morning so it may take some time for the changes to propagate.  Not certain about that but now the hardware status tab works for all servers in the recovery site.

Posted by at No comments:
Labels: , , ,

ERROR: vSphere Replication shows Not Active

Another strange one.  Existing VM replications appear to be working based on "last sync completed" time stamps.  However, setting up a new replications result in a status of "Not Active".  Right-clicking on a VM and choosing "sychronize now" results in this error:
Call "HmsGroup.OnlineSync" for object "[some long GID]" on Server "[server name/IP]" failed.  An unknown error has occurred.
I Googled the error and found this VMware communities forum thread:
SRM5 using vSphere replication, status shows 'not active'

Read through it but note that you shouldn't have to reboot everything like one poster did.  I rebooted the VRMS server in the recovery site and replications started working for all VMs again.  YMMV.  This happened to me after having rebooted the vCenter server also at the recovery site.

Posted by at No comments:
Labels: , , , ,

ERROR: A general system error occurred

Recently, when trying to logon to SRM 5, I got the following error:
A general system error occurred: Internal error

Wow, that's real telling!
I tried restarting the SRM servers but no dice.  I then opened a ticket with VMware support.  I started a WebEx with the tech and after reviewing several SRM and vCenter logs, he really couldn't find the root cause of the problem.  However, he did say that they've only seen this generic error with vCenter, not SRM.

We rebooted the recovery side vCenter and viola, I was able to login again successfully.  It's the old saying - if all else fails, reboot!
Posted by at No comments:
Labels: , , , ,

Thursday, June 7, 2012

ERROR: Call "VirtualMachine.Relocate" for object...

The full error is:
Call "VirtualMachine.Relocate" for object "VirtualMachineName" on vCenter Server "VirtualCenterServer" failed.

Looks like it thinks there was a snapshot left-over from a Backup Exec backup.  Yet, no snapshot exists, no file on the datastore.  Turns out, it's in the vCenter database...

First hit on Google: VMware KB http://kb.vmware.com/kb/2008957

I unregistered and re-registered the VM in inventory and it fixed the problem.  One thing to note:  make sure you answer the power-on question as "Copy" and not "Move".  Selecting "copy" will cause vCenter to assign the VM a new ID, cleaning up the database.
Posted by at No comments:
Labels: ,

Wednesday, April 25, 2012

ERROR: Repeating ADWS Errors Every 1 Minute

One of the errors my vCheck report enumerated was the following:
Active Directory Web Services encountered an error while reading the settings for the specified Active Directory Lightweight Directory Services instance. Active Directory Web Services will retry this operation periodically. In the mean time, this instance will be ignored. Instance name: ADAM_VMwareVCMSDS


This error was repeating everyone one minute - needless to say there were quite a few entries.  Apparently this bug has been around since vSphere 4 but hasn't been fix.  Luckily the fix is easy.  Go to the following registry key:
HKLM\SYSTEM\CurrentControlSet\services\ADAM_VMwareVCMSDS\Parameters

Find and delete the "Port SSL" value (the value data should be empty).
Create and new DWORD value with the same Port SSL name.
The value data should be 636 decimal.
Restart the ADWS and VMware_VCMSDS services.

I did the procedure and checked the ADWS log again - problem solved.

That's it!  Er... wait,  new problem found in ADWS log.  Here is the error:

Active Directory Web Services could not find a server certificate with the specified certificate name. A certificate is required to use SSL/TLS connections. To use SSL/TLS connections, verify that a valid server authentication certificate from a trusted Certificate Authority (CA) is installed on the machine.

A quick Google and I found this in the VMware Communities:
This message is simply an informational message and should have no major impact on the running of the Virtual Center Server. The only ways to stop this message from appearing would be joining vCenter Server to a AD Domain. Btw, you CANNOT install AD Domain Controller on the same machine with vCenter, it will not work. Because vCenter 4.1 will install an instance of ADAM (Active Directory Application Mode). It uses this when you use vCenter Linked Mode and ADAM will conflict with its’ own AD services if the server is also a Domain Controller.

Okay so basically ignore it.  Hopefully this doesn't fill up the vCheck report.  Just something we'll have to keep an eye on.



Posted by at No comments:
Labels: , ,

COOL TOOL: vCheck

vCheck:  http://www.virtu-al.net/featured-scripts/vcheck/

I saw this mentioned on another blog (I don't remember which one) over a year ago and thought it looked good, but didn't provide much info beyond what I was getting with VKernel and RVTools.  Now I'm wrong.  Alan over at http://www.virtu-al.net wrote a PowerShell script that checks the health of your vCenter environment.  It has recently been updated to handle plugins.  Most of the checks have been converted to plugins, and now there's an Exchange plugin written by Phil Randal.  I would not be surprised to see other plugins for other systems like AD and SharePoint in the future.

I'm running most of the VMware and Exchange checks.  This is providing me information beyond the VKernel and RVTools tools I currently use.  I've set it up to run once per week and email the report to our VMware administrators distribution list.

In just the first week is has brought several problems to light which will be subjects of future blog articles(!).
I highly recommend!

Posted by at 1 comment:
Labels: , , ,

Thursday, March 15, 2012

vSphere 5 Upgrade: SRM - PART 1

Per Its Time vSphere 5 Upgrade, time to upgrade SRM.  Well, I got to step 6.2 and things went downhill from there.  The following is the description I used to open an SR with VMware support:
Upgrading SRM from 4.1.2 to 5.0.  I get the error: "failed to create database tables".  It doesn't appear to make any changes to the database.
I've double-checked settings per KB1015436 and several communities postings.
I've tried re-installing SRM 4.1.2 (successfully), performing a repair, then another upgrade but it still fails with the same error.
The first support tech went down the 'invalid permissions' path but this was not the problem.  Turns out, the SRM 5.0 upgrade does not support upgrading from SRM 4.1.2!  Interesting because the only documentation I can find on the subject clearly states that you can upgrade from SRM 4.1(!).  Looks like VMware needs to do a better job documenting these requirements.

I was then informed that I could wait until the next minor/point release of SRM 5 which would support upgrading from 4.1.2, but I didn't have that kind of time (and who knows when they'll actually release it).  So no upgrade for me, full install from scratch instead!  Besides have to reconfigure mappings, protection groups (which I was going to have to do anyway), etc, the biggest downside is losing the previous DR test results.  Yes, I saved those off as separate Excel files, but it would have been nice to have had all of the results right there in SRM from the beginning.


But wait, there's more!  Now that I have a brand new freshly installed SRM up and running, it's time to setup vSphere Replication.  Did that go problem free you ask?  Ummm, no.  The following is the description I used to open yet another SR with VMware support:
The VRMS servers at both sites fail to connect.  I have unregistered the server, powered down/deleted the appliance VM, re-initialized the VRMS database, repaired SRM, redeployed the VRMS servers and configured them with the same vCenter FQDN per KB2007463 but still have the same problem.
Between the support tech and I it took several hours to figure this one out.  The short of it is that it's a vCenter certificate problem.  What clued me into this was the error I got when registering the VRMS instance:

That "unacceptable signature algorithm" message is not your typical self-signed cert warning!  Turns out, my vCenter self-signed certs had expired.  This hadn't caused a problem until installing vSphere Replication - it wants at least a current/non-expired cert.  I checked the vCenter cert and sure enough, it had expired in 2010.  It was created in 2008 and was valid for only 2 years!

Now I bet you're wondering, how does one fix this cert problem?  Well that's easy, reinstall vCenter!  And repair won't work either so you have to uninstall the current vCenter instance and re-install a new one.  Luckily most settings are maintained in the vCenter database so this could have been much more painful.

While I was at it I checked the new vCenter cert and VMware apparently decided to make this one valid for 10 years.  Now that's more like it!

But wait, there's more!  Look for part PART 2 of this adventure in a near future post.  A little hint - the fun ain't over yet.


Posted by at No comments:
Labels: , , , , ,

vSphere 5 Upgrade: VMFS Datastores

Per Its Time vSphere 5 Upgrade, time to upgrade VMFS datastores.  Like the last couple of steps this one completed w/o issue.  Note that it is better to create VMFS datastores because you'll get a 1MB block size regardless of what size datastore you're creating, optimizing disk space.  Compare this to the 2-8MB block sizes required based on datastore size in ESXi 4.1 and earlier.

All of my datastores now report to be VMFS version 5.54.


Posted by at No comments:
Labels: , , , , ,
Subscribe to: Posts (Atom)