Just a quick note to let everyone know that I have decided to post articles related to VMware, virtualization and cloud computing to a new blog: http://vmclouds.e-wilkin.com
I will leave current posts as-is but new content will be posted to the new blog. I will continue to post my other ramblings here moving forward.
Friday, September 8, 2017
Sunday, March 12, 2017
New NAS Build Update 1
After working with Rockstor and weighing my options with HDDs, I've decided to go a slightly different direction...
NAS OS
I'm a big CentOS fan. I also have been wanting to learn more about Docker and BTRFS. This makes Rockstor with its "Rock-ons" and BTRFS filesystem the perfect choice. Or does it?
BTRFS
BTRFS seems to be Linux's answer to ZFS. However, it is not nearly as mature, feature complete or ready for primetime in my opinion. I wanted to work with 4+ drives which implies a RAID5/6 type of setup. I quickly found that BTRFS has some serious problems with its RAID5/6 implementation to the point where it corrupts data and therefore is not recommended for production. I could buy a true hardware-based RAID card for around $100, but I'm trying to avoid spending additional coin on this build (see Hardware section).
XPEnology
I'm "upgrading" from a Synology NAS. I love the simplicity and the 'it just works' factor it provides. Finding a NAS OS that provides a similar experience has proven to be difficult. Rockstor will not let you use any non-BTRFS filesystems on data drives. Per the RAID issue above and the fact that I did not get very good support from their somewhat inactive forums rules out Rockstor for this build. About the same time I started running into obstacles with Rockstor, I stumbled upon XPEnology.
While not officially supported by Synology, it is based on their OSS code and the forum appears fairly active. I've been testing it in a VM while waiting for some additional hardware parts to arrive (see hardware section). This is basically DSM on your hardware of choice. The real test will be the installation on the TS140-based system. This is in no way my final decision on the NAS OS as further testing is required.
Hardware
The more I worked with the TS140's case, the less I liked it. Both internal drive trays require an L-shaped SATA connector since they're so close to the side of the case. Lenovo only provides one. That was the final straw. Luckily, I had a storage server case, an ARK 4U-500-CA that I was about to list on Craigslist but decided to use for this build. It fits nicely with my 'low cost, greater flexibility' theme. The rackmount hardware is removable. Just add some rubber feet on the bottom and presto-chango - it's a server tower case! It can hold at least 10 drives, has great air-flow and best of all - its free (to me).
In the next update, I'll detail the fun of moving the TS140 guts into a new case and some additional decisions required based on the number of drives needed.
NAS OS
I'm a big CentOS fan. I also have been wanting to learn more about Docker and BTRFS. This makes Rockstor with its "Rock-ons" and BTRFS filesystem the perfect choice. Or does it?
BTRFS
BTRFS seems to be Linux's answer to ZFS. However, it is not nearly as mature, feature complete or ready for primetime in my opinion. I wanted to work with 4+ drives which implies a RAID5/6 type of setup. I quickly found that BTRFS has some serious problems with its RAID5/6 implementation to the point where it corrupts data and therefore is not recommended for production. I could buy a true hardware-based RAID card for around $100, but I'm trying to avoid spending additional coin on this build (see Hardware section).
XPEnology
I'm "upgrading" from a Synology NAS. I love the simplicity and the 'it just works' factor it provides. Finding a NAS OS that provides a similar experience has proven to be difficult. Rockstor will not let you use any non-BTRFS filesystems on data drives. Per the RAID issue above and the fact that I did not get very good support from their somewhat inactive forums rules out Rockstor for this build. About the same time I started running into obstacles with Rockstor, I stumbled upon XPEnology.
While not officially supported by Synology, it is based on their OSS code and the forum appears fairly active. I've been testing it in a VM while waiting for some additional hardware parts to arrive (see hardware section). This is basically DSM on your hardware of choice. The real test will be the installation on the TS140-based system. This is in no way my final decision on the NAS OS as further testing is required.
Hardware
The more I worked with the TS140's case, the less I liked it. Both internal drive trays require an L-shaped SATA connector since they're so close to the side of the case. Lenovo only provides one. That was the final straw. Luckily, I had a storage server case, an ARK 4U-500-CA that I was about to list on Craigslist but decided to use for this build. It fits nicely with my 'low cost, greater flexibility' theme. The rackmount hardware is removable. Just add some rubber feet on the bottom and presto-chango - it's a server tower case! It can hold at least 10 drives, has great air-flow and best of all - its free (to me).
In the next update, I'll detail the fun of moving the TS140 guts into a new case and some additional decisions required based on the number of drives needed.
Wednesday, March 1, 2017
Time for a (NAS) Change
I have been running the DVR feature of Plex for several months now. During the same time the message "this server is not powerful enough to convert video" kept popping up intermittently preventing some videos from playing. Oddly enough, it doesn't happen all of the time on all shows, but enough to be more than annoying. Another thing I couldn't understand is why it was trying to transcode video at all when I have this option turned off for local streaming.
I was finally able to determine that the tuners record TV shows using the older MPEG-2 video standard which is still used to broadcast HD television today. On the other hand, Plex wants to stream video using the newer H.264 standard. Therefore, Plex will convert recorded TV shows on-the-fly (transcode) when streaming to Plex clients.
My existing NAS is a Synology DS713+ running Plex Media Server along with several other applications. Upon researching this problem, I found a list of compatible NAS devices along with performance notes here: Plex NAS Compatibility List.
What this list tells me is that I have to spend around $1500 to get a Synology box that does not have any problems streaming 1080P HD video (mostly due to the requirement a modern Intel Core I3 CPU or higher). I love my Synology but not that much - its hard to justify that kind of coin when I can build a system twice as powerful for half the price (or better).
So now I've decided to build my own. I used to build and repair systems over twenty years ago and its interesting how little has really changes with system components. I have spare disks, a RAID controller (if needed) and so my requirements are relatively light:
Form-factor: Tower
CPU: Xeon proc (no more problems with transcoding)
Memory: At least 4GB ECC RAM but expandable
Disk: capable of at least 4 disks, hot-swappable nice-to-have]
Quiet (will live in my office)
Cheap (I'm poor... and cheap)
I narrowed the system choice down to the Dell PowerEdge T20 or the Lenovo ThinkServer TS140. These units are nearly identical and I wanted to go with the Dell but the ThinkServer prides itself in being quiet (26 decibels) and has an extra PCI slot that pushed it over the 'edge (hidden pun). I ordered the ThinkServer from eBay/Newegg here: Lenovo ThinkServer TS140 70A4003AUX Tower Server
A note on memory: I'm going to try the 4GB of memory to start. It comes as a single DIMM (as opposed to two or more DIMMs) which leaves three open slots. So if more memory is needed I can add another 4GB DIMM later. I had tried running a custom built storage server three or four years ago using a desktop board with non-ECC memory running FreeNAS. I had read on the forums multiple warnings that only ECC memory should be used due to the risk of bit rot and eventual data corruption. Well, it only took a couple weeks before VMs stored on that systems starting becoming corrupted. Having been bit by this bug, I won't run a storage server without ECC memory and highly, highly, highly recommend you do the same.
A note on storage: I'm going to reuse the two 4TB drives in my Synology for this system in addition to two more 4TB drives I'll purchase for this build. The TS140 has an integrated hardware RAID controller. So for the data volume, I'll install the two new drives and pull one of the Synology drives for a total of three. Then transfer the data off the Synology to the new system. Once I'm satisfied everything is in good working order, I'll pull the fourth drive and install into the new system, expand the volume, etc. Then I'll have a very nice DS713+ for sale!
A note on the OS: I will miss the easy UI experience Synology DSM provides. However, I have found a decent alternative: Rockstor. I looked at several options: OpenFiler (dead) NexentaStor, FreeNAS, etc. But Rockstor offers features that I really want: web-based interface, based on CentOS 7, Docker-based plugin system called Rock-ons, optimized to run from an USB drive and Apple Time Machine Support. I also use the Synology for security cameras but found Rockstor also supports ZoneMinder video surveillance system.
It's not all rainbows and unicorns though. I'm not very confident about BTRFS (pronounced 'butter FS'). Rockstor recommends against using RAID5 or 6 for production systems. No worries for me since I'm using hardware-based RAID. I've also read that newer versions are much more stable than early releases. So while they tout BTRFS as a big feature, it may actually be their biggest drawback. Time and testing will tell. There must be a reason that many if not most NAS companies are now offering this filesystem as an option on their devices.
Also note that the TS140 has an internal USB port - perfect for a USB flash drive to host the OS.
Here's the parts list:
System: Lenovo ThinkServer TS140 70A4003AUX Tower Server = $320
4TB Enterprise SATA x 2 = $177
16GB USB Flash Drive: Transcend 16GB JetFlash 820 = $10
Operating System: Rockstor = Free
RAID Adapter (if needed): PERC H310 = Free
NIC Adapter = Intel Quad port = Free
Application Mapping:
Plex Media Server (Synology package) = PMS Rock-on (Rock-on)
Surveillance Station (Synology package) = ZoneMinder (Rock-on)
CrashPlan (Community Package) = CrashPlan (Rock-on or native)
Photo Station (Synology Package) = TBD
Download Station (Synology Station) = CouchPotato? TBD
So for $100 less I'll have a systems that's much more powerful and expandable than a Synology 916+ with twice the capacity. I'll follow-up with some additional thoughts once I have run this build through its paces.
I was finally able to determine that the tuners record TV shows using the older MPEG-2 video standard which is still used to broadcast HD television today. On the other hand, Plex wants to stream video using the newer H.264 standard. Therefore, Plex will convert recorded TV shows on-the-fly (transcode) when streaming to Plex clients.
My existing NAS is a Synology DS713+ running Plex Media Server along with several other applications. Upon researching this problem, I found a list of compatible NAS devices along with performance notes here: Plex NAS Compatibility List.
What this list tells me is that I have to spend around $1500 to get a Synology box that does not have any problems streaming 1080P HD video (mostly due to the requirement a modern Intel Core I3 CPU or higher). I love my Synology but not that much - its hard to justify that kind of coin when I can build a system twice as powerful for half the price (or better).
So now I've decided to build my own. I used to build and repair systems over twenty years ago and its interesting how little has really changes with system components. I have spare disks, a RAID controller (if needed) and so my requirements are relatively light:
Form-factor: Tower
CPU: Xeon proc (no more problems with transcoding)
Memory: At least 4GB ECC RAM but expandable
Disk: capable of at least 4 disks, hot-swappable nice-to-have]
Quiet (will live in my office)
Cheap (I'm poor... and cheap)
I narrowed the system choice down to the Dell PowerEdge T20 or the Lenovo ThinkServer TS140. These units are nearly identical and I wanted to go with the Dell but the ThinkServer prides itself in being quiet (26 decibels) and has an extra PCI slot that pushed it over the 'edge (hidden pun). I ordered the ThinkServer from eBay/Newegg here: Lenovo ThinkServer TS140 70A4003AUX Tower Server
A note on memory: I'm going to try the 4GB of memory to start. It comes as a single DIMM (as opposed to two or more DIMMs) which leaves three open slots. So if more memory is needed I can add another 4GB DIMM later. I had tried running a custom built storage server three or four years ago using a desktop board with non-ECC memory running FreeNAS. I had read on the forums multiple warnings that only ECC memory should be used due to the risk of bit rot and eventual data corruption. Well, it only took a couple weeks before VMs stored on that systems starting becoming corrupted. Having been bit by this bug, I won't run a storage server without ECC memory and highly, highly, highly recommend you do the same.
A note on storage: I'm going to reuse the two 4TB drives in my Synology for this system in addition to two more 4TB drives I'll purchase for this build. The TS140 has an integrated hardware RAID controller. So for the data volume, I'll install the two new drives and pull one of the Synology drives for a total of three. Then transfer the data off the Synology to the new system. Once I'm satisfied everything is in good working order, I'll pull the fourth drive and install into the new system, expand the volume, etc. Then I'll have a very nice DS713+ for sale!
A note on the OS: I will miss the easy UI experience Synology DSM provides. However, I have found a decent alternative: Rockstor. I looked at several options: OpenFiler (dead) NexentaStor, FreeNAS, etc. But Rockstor offers features that I really want: web-based interface, based on CentOS 7, Docker-based plugin system called Rock-ons, optimized to run from an USB drive and Apple Time Machine Support. I also use the Synology for security cameras but found Rockstor also supports ZoneMinder video surveillance system.
It's not all rainbows and unicorns though. I'm not very confident about BTRFS (pronounced 'butter FS'). Rockstor recommends against using RAID5 or 6 for production systems. No worries for me since I'm using hardware-based RAID. I've also read that newer versions are much more stable than early releases. So while they tout BTRFS as a big feature, it may actually be their biggest drawback. Time and testing will tell. There must be a reason that many if not most NAS companies are now offering this filesystem as an option on their devices.
Also note that the TS140 has an internal USB port - perfect for a USB flash drive to host the OS.
Here's the parts list:
System: Lenovo ThinkServer TS140 70A4003AUX Tower Server = $320
4TB Enterprise SATA x 2 = $177
16GB USB Flash Drive: Transcend 16GB JetFlash 820 = $10
Operating System: Rockstor = Free
RAID Adapter (if needed): PERC H310 = Free
NIC Adapter = Intel Quad port = Free
Application Mapping:
Plex Media Server (Synology package) = PMS Rock-on (Rock-on)
Surveillance Station (Synology package) = ZoneMinder (Rock-on)
CrashPlan (Community Package) = CrashPlan (Rock-on or native)
Photo Station (Synology Package) = TBD
Download Station (Synology Station) = CouchPotato? TBD
So for $100 less I'll have a systems that's much more powerful and expandable than a Synology 916+ with twice the capacity. I'll follow-up with some additional thoughts once I have run this build through its paces.
Thursday, July 28, 2016
VMworld 2016: The End of an Era
I sadly report that I will not be attending VMworld 2016 this year. This will be the first conference I've missed - "13" really is an unlucky number! I was not selected per the criteria my organization used to determine who goes and who stays. I respect their decision, but that doesn't change the fact that I will be watching the general sessions remotely on a computer screen. Yeah, that won't quite be the same - no assisting customers, meeting friends, attending sessions or keeping up with what's going on with partners in the Solutions Exchange this year for me.
My passion for the conference won't change - I still believe it to be the best IT conference and highly recommend attending.
So the VMworld Alumni Elite group will be at least one less this year. What a prestigious group! I know from meeting with these guys year after year that they have a real passion for VMworld, VMware and its products. Guys - I'll catch up with you next time!
Its ironic - during my interview I was told that the surest way to not attend VMworld was to become a VMware employee. My first year VMware decided to send my entire organization! My second year they decided to send most of the organization, yours truly included. This is my third year and it appears as though my luck has run out.
I did try multiple alternative options but none of those panned-out. So all good things must come to an end? C'est la vie - its the end of an era... and the start of a new era - carpe diem!
For those that are going - have a great conference! For the reset of us, make sure to tune in to the general sessions which will be live-streamed again this year. And don't forget about the breakout sessions that will be made available shortly after the conference.
My passion for the conference won't change - I still believe it to be the best IT conference and highly recommend attending.
So the VMworld Alumni Elite group will be at least one less this year. What a prestigious group! I know from meeting with these guys year after year that they have a real passion for VMworld, VMware and its products. Guys - I'll catch up with you next time!
Its ironic - during my interview I was told that the surest way to not attend VMworld was to become a VMware employee. My first year VMware decided to send my entire organization! My second year they decided to send most of the organization, yours truly included. This is my third year and it appears as though my luck has run out.
I did try multiple alternative options but none of those panned-out. So all good things must come to an end? C'est la vie - its the end of an era... and the start of a new era - carpe diem!
For those that are going - have a great conference! For the reset of us, make sure to tune in to the general sessions which will be live-streamed again this year. And don't forget about the breakout sessions that will be made available shortly after the conference.
Thursday, February 11, 2016
Enabling the Digital Enterprise: VMware Announcements
VMware made several major product announcements this week and I'm super-excite about some of the improvements and new features coming our way.
Our EUC team is really firing on all cylinders and continues to update and integrate products. In this case AirWatch, Horizon and Identity Manager are brought together with Workspace One. Did you know we coined the term "Workspace" (well okay, a company we acquired years ago did). I used to be one of those guys that connected to my View desktop and worked from there. Now, more often than not I browse our internal Workspace One portal and launch whatever app I need to get work done. How did I work without this before?
Also worth noting are several SDDC product updates. The big news here is VSAN 6.2 - what an awesome release! VSAN was ready to host your tier 1 applications with the release of 6.0 - now with 6.2's dedupe, compression and RAID5/6 features, the nay-sayers won't be able say it's not ready for the enterprise (well, they can but they would be wrong).
Here is a "Did you know" PSA:
Did you know that if you use vSphere Replication to replicate VMs hosted on VSAN storage, you can set the RPO as low as 5 minutes?
Now you know.
Finally. take a look at the vCloud Suite pricing and packaging changes. We've really change the Suites to refocus on what customer's really want - the tools needed to run a true SDDC datacenter (wait, isn't that redundant?).
- Yesterday’s VMware EUC Announcement
- Horizon 7
- Main: http://www.vmware.com/products/horizon-view/
- Press Release: http://www.vmware.com/go/pr-horizon-app
- VMware Blog: http://blogs.vmware.com/euc/2016/02/vmware-horizon-7.html
- Workspace One
- Main: https://www.vmware.com/products/workspace-one/
- VMware Blog: http://blogs.vmware.com/euc/2016/02/new-vmware-workspace-one.html
- ZDNet Article: http://www.zdnet.com/article/vmwares-workspace-one-targets-mobile-enterprise-with-hybridity/
- PCWorld Article: http://www.pcworld.com/article/3031792/vmware-gives-enterprises-a-byod-boost-with-three-new-tools.html
- Horizon Air with Hybrid Mode
- Main: http://www.vmware.com/cloud-services/desktop/horizon-air-hybrid-mode/
- VMware Blog: http://blogs.vmware.com/euc/2016/02/vmware-horizon-air-hybrid-mode.html
- Madden Article: http://www.brianmadden.com/blogs/gabeknuth/archive/2016/02/09/vmware-announces-horizon-air-hybrid-mode_2C00_-plus-horizon-7-and-appvolumes-3-0-here-s-our-full-analysis.aspx
- Today’s SDDC Announcements
- vCloud Suite 7 and vRealize Suite 7
- VMware Blog: http://blogs.vmware.com/management/2016/02/introducing-vmware-vcloud-suite-7-vrealize-suite-7.html
- CTR Article: http://wwpi.com/vmware-advances-cloud-management-platform-debuts-all-flash-hyper-converged-solutions/
- vRealize Operations 6.2
- Main: https://www.vmware.com/products/vrealize-operations/
- VMware : Bloghttps://blogs.vmware.com/management/2016/02/whats-new-in-vrealize-operations-6-2.html
- vRealize Log Insight 3.3
- Main: http://www.vmware.com/products/vrealize-log-insight
- VMware Blog: https://blogs.vmware.com/management/2016/02/whats-new-log-insight-3-3.html
- vRealize Automation 7
- Main: https://www.vmware.com/products/vrealize-automation
- VMware Blog: http://blogs.vmware.com/management/2015/11/whats-new-vrealize-automation-7.html
- vRealize Business for Cloud 7
- Main: https://www.vmware.com/products/vrealize-business-for-cloud/
- VMware Blog: https://blogs.vmware.com/management/2016/02/changes-vrealize-business-pricing-packaging.html
- Virtual SAN (VSAN) 6.2
- Main: http://www.vmware.com/products/whats-new-virtual-san.html
- VMware Blog: http://blogs.vmware.com/virtualblocks/2016/02/10/whats-new-vmware-virtual-san-6-2/
- Duncan’s Blog: http://www.yellow-bricks.com/2016/02/10/whats-new-for-virtual-san-6-2/
- Cormac’s Blog: http://cormachogan.com/2016/02/10/vsan-6-2-an-overview-of-the-new-virtual-san-6-2-features/
- StorageReview Article: http://www.storagereview.com/vmware_introduces_vsan_62_with_an_eye_on_allflash
- New pricing and packaging
Replay’s of this week's events are available here: http://www.vmware.com/digitalenterprise
vROps Alert: One or more ports on the Distributed Port Group are experiencing network contention due to dropped packets
Here is a vROps problem experienced by one of my customers:
Alert: One or more ports are experiencing network connection
This compliance state applies mainly to patches. The target object has a newer version of the patch. For example, if a patch has multiple versions, after you apply the latest version to the host, the earlier versions of the patch are in Obsoleted By Host compliance state.
(https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.update_manager.doc_50%2FGUID-EBAA4F4A-57E0-45ED-8730-4B851FC846A9.html)
Alert: One or more ports are experiencing network connection
(One or more ports on the Distributed Port Group are experiencing network contention due to dropped packets)
There was no evidence in vCenter that packets were being dropped. The customer found KB2105109: VMware vRealize Operations Manager counters report false dropped packets on vSphere Distributed Switches with the alert: One or more ports are experiencing network contention
However, this seems to only apply to vROps 6.0.x and the customer is running the 6.2.x release. An SR was opened and the support tech recommended the fix per KB2052917: vCenter Server 5.1/5.5/6.0 performance charts report dropped network packets
Okay great, so a patch needs to be applied to the ESXi 5.5 host. The customer tried to applied the patch via VUM but it was marked obsoleted. "Obsoleted" is an Update Manager compliance state. As explained in the Installing and Administering VMware vSphere Update Manager documentation:
This compliance state applies mainly to patches. The target object has a newer version of the patch. For example, if a patch has multiple versions, after you apply the latest version to the host, the earlier versions of the patch are in Obsoleted By Host compliance state.
(https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.update_manager.doc_50%2FGUID-EBAA4F4A-57E0-45ED-8730-4B851FC846A9.html)
Hmmm.... so the host has a newer version of the patch, so now what?
After some discussion we landed on the fact that the vCenter instance managing this host was likely the culprit as it has a patch level several versions behind the host. I always recommend customers keep their vCenter and ESXi hosts at the same Update patch level. Installing patches on hosts in-between major Updates is okay. In this case, they were on different Update versions.
The vCenter instance was patched to the latest update and voila, the error disappeared from vROps.
Monday, February 1, 2016
Horizon View Restart/Reboot Sequence
View admins - please make sure you restart View infrastructure servers in the proper order for fastest uptime! I recently had a customer that rebooted vCenter and View servers in an effort to fix an apparent connection problem. When they did not appear to be coming online fast enough, the admin rebooted them again.
Now normally rebooting these servers can cause ADLDS synchronization to take up to fifteen minutes or so, but rebooting them again, before the initial sync completes, can cause them to take thirty minutes or more - and then leave inconsistencies within the pools.
For more information on the proper boot sequence:
Restart order of the View environment to clear ADLDS (ADAM) synchronization in View 4.5, 4.6, 5.0, and 5.1
And to clear out any inconsistencies:
Manually deleting linked clones or stale virtual desktop entries from the View Composer database in VMware View Manager and VMware Horizon View
Subscribe to:
Posts (Atom)