Sunday, March 12, 2017

New NAS Build Update 1

After working with Rockstor and weighing my options with HDDs, I've decided to go a slightly different direction...

NAS OS
I'm a big CentOS fan.  I also have been wanting to learn more about Docker and BTRFS.  This makes Rockstor with its "Rock-ons" and BTRFS filesystem the perfect choice.  Or does it?

BTRFS
BTRFS seems to be Linux's answer to ZFS.  However, it is not nearly as mature, feature complete or ready for primetime in my opinion.  I wanted to work with 4+ drives which implies a RAID5/6 type of setup.  I quickly found that BTRFS has some serious problems with its RAID5/6 implementation to the point where it corrupts data and therefore is not recommended for production.  I could buy a true hardware-based RAID card for around $100, but I'm trying to avoid spending additional coin on this build (see Hardware section).

XPEnology
I'm "upgrading" from a Synology NAS.  I love the simplicity and the 'it just works' factor it provides.  Finding a NAS OS that provides a similar experience has proven to be difficult.  Rockstor will not let you use any non-BTRFS filesystems on data drives.  Per the RAID issue above and the fact that I did not get very good support from their somewhat inactive forums rules out Rockstor for this build.  About the same time I started running into obstacles with Rockstor, I stumbled upon XPEnology.

While not officially supported by Synology, it is based on their OSS code and the forum appears fairly active.  I've been testing it in a VM while waiting for some additional hardware parts to arrive (see hardware section).  This is basically DSM on your hardware of choice.  The real test will be the installation on the TS140-based system.  This is in no way my final decision on the NAS OS as further testing is required.

Hardware
The more I worked with the TS140's case, the less I liked it.  Both internal drive trays require an L-shaped SATA connector since they're so close to the side of the case.  Lenovo only provides one.  That was the final straw.  Luckily, I had a storage server case, an ARK 4U-500-CA that I was about to list on Craigslist but decided to use for this build.  It fits nicely with my 'low cost, greater flexibility' theme.  The rackmount hardware is removable.  Just add some rubber feet on the bottom and presto-chango - it's a server tower case!  It can hold at least 10 drives, has great air-flow and best of all - its free (to me).

In the next update, I'll detail the fun of moving the TS140 guts into a new case and some additional decisions required based on the number of drives needed.


Wednesday, March 1, 2017

Time for a (NAS) Change

I have been running the DVR feature of Plex for several months now.  During the same time the message "this server is not powerful enough to convert video" kept popping up intermittently preventing some videos from playing.  Oddly enough, it doesn't happen all of the time on all shows, but enough to be more than annoying.  Another thing I couldn't understand is why it was trying to transcode video at all when I have this option turned off for local streaming.

I was finally able to determine that the tuners record TV shows using the older MPEG-2 video standard which is still used to broadcast HD television today.  On the other hand, Plex wants to stream video using the newer H.264 standard.  Therefore, Plex will convert recorded TV shows on-the-fly (transcode) when streaming to Plex clients.

My existing NAS is a Synology DS713+ running Plex Media Server along with several other applications.  Upon researching this problem, I found a list of compatible NAS devices along with performance notes here: Plex NAS Compatibility List.

What this list tells me is that I have to spend around $1500 to get a Synology box that does not have any problems streaming 1080P HD video (mostly due to the requirement a modern Intel Core I3 CPU or higher).  I love my Synology but not that much - its hard to justify that kind of coin when I can build a system twice as powerful for half the price (or better).

So now I've decided to build my own.  I used to build and repair systems over twenty years ago and its interesting how little has really changes with system components.  I have spare disks, a RAID controller (if needed) and so my requirements are relatively light:
Form-factor: Tower
CPU: Xeon proc (no more problems with transcoding)
Memory: At least 4GB ECC RAM but expandable
Disk: capable of at least 4 disks, hot-swappable nice-to-have]
Quiet (will live in my office)
Cheap (I'm poor... and cheap)

I narrowed the system choice down to the Dell PowerEdge T20 or the Lenovo ThinkServer TS140.  These units are nearly identical and I wanted to go with the Dell but the ThinkServer prides itself in being quiet (26 decibels) and has an extra PCI slot that pushed it over the 'edge (hidden pun).  I ordered the ThinkServer from eBay/Newegg here: Lenovo ThinkServer TS140 70A4003AUX Tower Server

A note on memory:  I'm going to try the 4GB of memory to start.  It comes as a single DIMM (as opposed to two or more DIMMs) which leaves three open slots.  So if more memory is needed I can add another 4GB DIMM later.  I had tried running a custom built storage server three or four years ago using a desktop board with non-ECC memory running FreeNAS.  I had read on the forums multiple warnings that only ECC memory should be used due to the risk of bit rot and eventual data corruption.  Well, it only took a couple weeks before VMs stored on that systems starting becoming corrupted.  Having been bit by this bug, I won't run a storage server without ECC memory and highly, highly, highly recommend you do the same.

A note on storage:  I'm going to reuse the two 4TB drives in my Synology for this system in addition to two more 4TB drives I'll purchase for this build.  The TS140 has an integrated hardware RAID controller.  So for the data volume, I'll install the two new drives and pull one of the Synology drives for a total of three.  Then transfer the data off the Synology to the new system.  Once I'm satisfied everything is in good working order, I'll pull the fourth drive and install into the new system, expand the volume, etc.  Then I'll have a very nice DS713+ for sale!

A note on the OS: I will miss the easy UI experience Synology DSM provides.  However, I have found a decent alternative: Rockstor.  I looked at several options: OpenFiler (dead) NexentaStor, FreeNAS, etc.  But Rockstor offers features that I really want: web-based interface, based on CentOS 7, Docker-based plugin system called Rock-ons, optimized to run from an USB drive and Apple Time Machine Support.  I also use the Synology for security cameras but found Rockstor also supports ZoneMinder video surveillance system.
It's not all rainbows and unicorns though.  I'm not very confident about BTRFS (pronounced 'butter FS').  Rockstor recommends against using RAID5 or 6 for production systems.  No worries for me since I'm using hardware-based RAID.  I've also read that newer versions are much more stable than early releases.  So while they tout BTRFS as a big feature, it may actually be their biggest drawback.  Time and testing will tell.  There must be a reason that many if not most NAS companies are now offering this filesystem as an option on their devices.

Also note that the TS140 has an internal USB port - perfect for a USB flash drive to host the OS.

Here's the parts list:
System: Lenovo ThinkServer TS140 70A4003AUX Tower Server = $320
4TB Enterprise SATA x 2 = $177
16GB USB Flash Drive: Transcend 16GB JetFlash 820 = $10
Operating System: Rockstor = Free
RAID Adapter (if needed): PERC H310 = Free
NIC Adapter = Intel Quad port = Free

Application Mapping:
Plex Media Server (Synology package) = PMS Rock-on (Rock-on)
Surveillance Station (Synology package) = ZoneMinder (Rock-on)
CrashPlan (Community Package) = CrashPlan (Rock-on or native)
Photo Station (Synology Package) = TBD
Download Station (Synology Station) = CouchPotato?  TBD

So for $100 less I'll have a systems that's much more powerful and expandable  than a Synology 916+ with twice the capacity.  I'll follow-up with some additional thoughts once I have run this build through its paces.

Thursday, July 28, 2016

VMworld 2016: The End of an Era

I sadly report that I will not be attending VMworld 2016 this year.  This will be the first conference I've missed - "13" really is an unlucky number!  I was not selected per the criteria my organization used to determine who goes and who stays.  I respect their decision, but that doesn't change the fact that I will be watching the general sessions remotely on a computer screen.  Yeah, that won't quite be the same - no assisting customers, meeting friends, attending sessions or keeping up with what's going on with partners in the Solutions Exchange this year for me.

My passion for the conference won't change - I still believe it to be the best IT conference and highly recommend attending.

So the VMworld Alumni Elite group will be at least one less this year.  What a prestigious group!   I know from meeting with these guys year after year that they have a real passion for VMworld, VMware and its products.  Guys - I'll catch up with you next time!

Its ironic - during my interview I was told that the surest way to not attend VMworld was to become a VMware employee.  My first year VMware decided to send my entire organization!  My second year they decided to send most of the organization, yours truly included.  This is my third year and it appears as though my luck has run out.

I did try multiple alternative options but none of those panned-out.  So all good things must come to an end?  C'est la vie - its the end of an era... and the start of a new era - carpe diem!

For those that are going - have a great conference!  For the reset of us, make sure to tune in to the general sessions which will be live-streamed again this year.  And don't forget about the breakout sessions that will be made available shortly after the conference.

Thursday, February 11, 2016

Enabling the Digital Enterprise: VMware Announcements

VMware made several major product announcements this week and I'm super-excite about some of the improvements and new features coming our way.

Our EUC team is really firing on all cylinders and continues to update and integrate products.  In this case AirWatch, Horizon and Identity Manager are brought together with Workspace One.  Did you know we coined the term "Workspace" (well okay, a company we acquired years ago did).  I used to be one of those guys that connected to my View desktop and worked from there.  Now, more often than not I browse our internal Workspace One portal and launch whatever app I need to get work done.  How did I work without this before?

Also worth noting are several SDDC product updates.  The big news here is VSAN 6.2 - what an awesome release!  VSAN was ready to host your tier 1 applications with the release of 6.0 - now with 6.2's dedupe, compression and RAID5/6 features, the nay-sayers won't be able say it's not ready for the enterprise (well, they can but they would be wrong).

Here is a "Did you know" PSA: 
Did you know that if you use vSphere Replication to replicate VMs hosted on VSAN storage, you can set the RPO as low as 5 minutes?
Now you know.

Finally. take a look at the vCloud Suite pricing and packaging changes.  We've really change the Suites to refocus on what customer's really want - the tools needed to run a true SDDC datacenter (wait, isn't that redundant?).
Replay’s of this week's events are available here: http://www.vmware.com/digitalenterprise

vROps Alert: One or more ports on the Distributed Port Group are experiencing network contention due to dropped packets

Here is a vROps problem experienced by one of my customers:
Alert: One or more ports are experiencing network connection
(One or more ports on the Distributed Port Group are experiencing network contention due to dropped packets)


However, this seems to only apply to vROps 6.0.x and the customer is running the 6.2.x release.  An SR was opened and the support tech recommended the fix per KB2052917: vCenter Server 5.1/5.5/6.0 performance charts report dropped network packets

Okay great, so a patch needs to be applied to the ESXi 5.5 host.  The customer tried to applied the patch via VUM but it was marked obsoleted.  "Obsoleted" is an Update Manager compliance state.  As explained in the Installing and Administering VMware vSphere Update Manager documentation:


This compliance state applies mainly to patches. The target object has a newer version of the patch. For example, if a patch has multiple versions, after you apply the latest version to the host, the earlier versions of the patch are in Obsoleted By Host compliance state.


(https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.update_manager.doc_50%2FGUID-EBAA4F4A-57E0-45ED-8730-4B851FC846A9.html)

Hmmm.... so the host has a newer version of the patch, so now what?

After some discussion we landed on the fact that the vCenter instance managing this host was likely the culprit as it has a patch level several versions behind the host.  I always recommend customers keep their vCenter and ESXi hosts at the same Update patch level.  Installing patches on hosts in-between major Updates is okay.  In this case, they were on different Update versions.

The vCenter instance was patched to the latest update and voila, the error disappeared from vROps.



Monday, February 1, 2016

Horizon View Restart/Reboot Sequence

View admins - please make sure you restart View infrastructure servers in the proper order for fastest uptime!  I recently had a customer that rebooted vCenter and View servers in an effort to fix an apparent connection problem.  When they did not appear to be coming online fast enough, the admin rebooted them again.

Now normally rebooting these servers can cause ADLDS synchronization to take up to fifteen minutes or so, but rebooting them again, before the initial sync completes, can cause them to take thirty minutes or more - and then leave inconsistencies within the pools.

For more information on the proper boot sequence:
Restart order of the View environment to clear ADLDS (ADAM) synchronization in View 4.5, 4.6, 5.0, and 5.1

And to clear out any inconsistencies:
Manually deleting linked clones or stale virtual desktop entries from the View Composer database in VMware View Manager and VMware Horizon View


Thursday, August 27, 2015

VMworld 2014 Recap

Well, VMworld 2015 is next week so I guess I'd better get this one published!  I actually found this sitting in my drafts, I was waiting for a picture to come through which took so long I forgot to go back and publish it after receiving it!  So here it is in all it's glory...

This was my first VMworld as an employee.  My new role gave me a unique perspective to the conference.  Having attended every previous conference as a VMware customer has given me knowledge and experience I can pass on to my customers so they can get the most out of the conference.

Consequently, I walked through the Solutions Exchange but didn't spend as much time there as I normally do.  I attended a whopping one break-out session, and all of the general sessions.  Why?  Because I spent most of my time with my customers - assisting them in some form or shape.  I also spent a lot of time at TAM Customer Central.  What is TCC you ask?  It's a top secret location we reserve for VMware TAM customers where we provide special break-out sessions not available to VMworld attendees, receptions and meeting rooms among other things.

The theme this year was NO LIMITS.  Here is some cool desktop wallpaper for you:

And here is the Moscone all dressed up:

We were asked to provide pics of us with our customers for use as a collage at VMworld to be displayed at various times such as just before and after the general sessions.  Here's the pic I sent in with "Mr. Corvette" Jarod and his red sports car that made us mini-celebrities for the week:

Another one of those TAM customer benefits is the Hands-on Labs VIP tour.  This gives select customers a behind-the-scenes look at HOL operations and how the whole thing is managed.  Check out these stats:


Its interesting that the NSX lab remained the most popular lab during the entire conference - indicates a lot of interest by our customers.  And, consequently I think, this product continues to gain momentum in the marketplace.

And pics of how we used Log Insight and vCOps to monitor the environment via custom dashboards:




Even though I attended as a VMware employee this year, I was still invited to the CTO party (I had customers in attendance, so I'm sure that helped):

Why yes Johnny, everything is better with bacon:




And a pic from the Alumni Elite pre-party party - the pic I was waiting for as it was taken by someone else's camera - which wouldn't be complete without a shot with the big guy (me=literally, Mr. Gelsinger=figuratively):

And finally, lookie at what I found:

I found a "Meet the Experts" area hidden (not really) on the third floor of Moscone West.  If you have any technical or strategic product questions, these are the guys to ask and anyone can sign-up.  I highly recommend this resource if we offer it again in 2015.

Here's to seeing you there next year (he says with bacon in one hand, bourbon in the other)!