Re-Purposing an Old EMC Celerra

One of the reasons I was hired by my current employer was to implement an EMC Celerra NS352 NAS/SAN that was purchased the year before.  That was six years ago.  Since then I have moved all workloads off of this storage and on to other SANs.  So what to do with this device?  We tried selling it to four different vendors that specialize in used hardware - none of them would take it, not even for free.  Ouch!  Instead of paying a recycling company to come pick it up, I decided we could re-use it for tier II or III data.  Note that the Celerra has a CLARiiON SAN back-end.  I was never impressed with the Celerra's performance (putting it nicely) but connecting the CLARiiON directly to the SAN fabric and shutting-down/disconnecting all of the NAS head components should be worth the effort.  It does have roughly 13TB of 2Gb fibre drives after-all.

A word about maintenance:  This NAS/SAN system EOL'ed in February of this year.  Now the data I'm hosting on here will not be very important - backups, temporary data, test VMs, etc, but if a disk or other component fails (and it will), it would be more than nice to get the item replaced in a timely manner.  Enter third-party maintenance.  The Celerras were already under third-party maintenance, I simply worked with the existing vendor to convert the asset from Celerra to CLARiiON.  This also had the unexpected benefit of further reducing the cost of the maintenance contract.

The following sections document the process and procedures I used to re-purpose the CLARiiON back-end of two Celerras, giving a little more life to what would have otherwise been sent to the scrap heap.

Shutdown and Cabling

1. Power off Celerra NAS head (not CLARiiON disk shelves or Service Processor bay)
2. Disconnect Ethernet and fiber cables from SP bay.

Re-IP The Service Processors

1. Connect a laptop to the Ethernet port on SPA
2. Change laptop's IP to 128.221.252.111/255.255.255.0
3. Browse to 128.221.252.200 (SPA)
4. Logon using nasadmin/nasadmin
5. Drill down to SPA, right-click and choose properties
6. Under the Network tab, change IP and SP Network Name.
7. Unplug the Ethernet cable from SPA an connect to SPB.

Repeat steps above but Browse to 128.221.252.201 (SPB).
Connect both SPs to Ethernet switch.

Now the remaining steps can be performed remotely.

User Management

I recommend setting up an administrative user to coexist with nasadmin. There is also an existing "admin" user that I would just leave there. I create an account named "Administrator" with a unique password.

1. Go to the Tools menu and select Security\User Management\Add.

Clean-up Existing LUN

There will likely be many LUNs that were configure for and consumed by the Celerra NAS part of the system.  I simply deleted all of these, event the DART/OS LUNs and RAID Groups.  I did not have any hosts defined/registered.

Enable Access Logix

1. Right-click on the CLARiiON (serial number) and choose Properties
2. In the Storage Access tab, in the Data Access pane, check "Access Control Enabled"

Disk Layout/Hot Spares

Next I generate a disk layout report to see what disks are assigned as hot spares. Go to the Reporting node and generate a Configuration\Available Storage report.  Check to see where the current hot spare disks are located. I typically like to have these at the end of the disk cabinet/bus enclosure. Move the spare(s) to the appropriate drive(s) as needed.

The last EMC recommendation I'm aware of is 1 hot spare per 30 disks.

SAN Fabric Configuration

Time to connect the CLARiiON to your fibre fabric. 

• Each SP should have 2 ports. Connect one port to each fabric.
• Both SPs should be connected to both fabrics.
• Create an alias for each port/WWN in your SAN switch.
• Create the zones in your SAN switch to allow the hosts to "see" the CLARiiON.
• Save and enable the new configuration.

Host and LUN Creation, Masking

Now that the CLARiiON can see the hosts, its time to register the hosts with the CLARiiON:

1. Right-click on the CLARiiON (the serial number) and choose Connectivity Status
2. Note that you will need to know the WWNs of each FC port of your hosts. All of my hosts are ESXi servers, so the following instructions will be for these types of hosts.
3. Highlight the Initiator Name and click Register
4. The Initiator Type should be "CLARiiON Open"
5. Enter the HBA information
6. Enter the Host information
7. When finished click OK
8. Click the Refresh button and you should see the host name appear in the "Server Name" column
9. Complete the remaining initiators.

After you have all of the hosts registered, its time to create the LUN(s) you want the hosts to consume:

1. Create a RAID Group -
1. Right-click on the RAID Groups node and choose 'Create RAID Group'
2. RAID Group ID: leave the default value ("0" for the first one)
3. Number of disks: this is the stuff that starts religous wars. To keep it simple I use groups of 5 using RAID5.
4. RAID Type: RAID 5
5. Disk Selection: you could leave the default of "Automatic" and Navisphere will choose for you. However, I've never liked it's choices, so set it to manual and choose the disks that make the most sense (for example, the first 5 disks in enclosure x).
2. Create the LUN -
1. Right-click on the RAID Group you just created and choose "Bind LUN"
2. RAID Type - default should match the RAID Group setting
3. RAID Group - should be the same ID as the one you selected
4. Rebuild Priority - leave default
5. Verify Priority - leave default
6. Default Owner - Choose Auto
7. LUN Size - Choose MAX from the drop-down list

Connecting It All Together

1. Create the Storage Group -
1. Right-click on Storage Groups and choose "Create Storage Group"
2. I used the name of my cluster since this storage will be shared among all hosts in that cluster
2. Add hosts to the storage group - 
1. Right-click on the storage group name and choose "Connect Hosts"
2. Select all of the hosts that should be a member of this group and move them to the right-hand pane
3. Click OK
3. Add LUNs to the Storage Group -
1. Right-click on the storage group and choose "Select LUNs"
2. Expand out the appropriate component and select the LUN(s)
3. Click OK

That's it! Your hosts should now be connected to the CLARiiON storage and consuming LUNs per what youy have configured above.

Other Notes

• I recommend documenting the config as your setting this up. This includes network names, IPs and SP WWNs which can be found at:
       Storage Domains\LocalDomain\[SerialNo]\Physical\SPs\SPA(B)\Ports
       (It's the second half of each WWN)
• Don't forget to recan your ESXi hosts and check to make sure the paths are using RoundRobin, VMW_SATP_CX and IOPS=1
• I would also generate a new Available Storage report after each RAID Group/LUN creation event. I save these to an Excel spreadsheet for future reference
• Use a tool such as Solarwinds Storage Manager to monitor SAN usage and availability.

Using the vSphere 5 CLI for Storage Configuration

When Adding a new host, SAN or LUN to your vSphere environment there are some CLI commands and configuration settings you should consider.  Since Windows 7 is my primary desktop OS, I use the VMware vSphere CLI for Windows.  You can download the latest version from VMware's downloads site.

Before using any of the commands below, please review your SAN manufacturers best practices documentation for vSphere environments.  Can't find any?  Then you bought the wrong SAN!  Seriously, most manufacturers provide something around documentation - if Google doesn't help try using Bing... better yet, try contacting your VAR.

View Devices and Their Settings
To view devices and some of their related settings, use the following command:
esxcli -s [HOSTNAME] -u root -p [PASSWORD] storage nmp device list

Set Storage Array Type and Path Selection Policy
One of the first things you'll want to do is change the DEFAULT Path Selection Policy (PSP) for whatever storage array types (SATP) you're using.  This way, when adding a new LUN/device, the type will be set to the proper type automatically.  Most modern arrays support ROUND ROBIN.  To change the array type, use the following command:
esxcli -s [ESXHOST] -u root -p [PASSWORD] storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_ALUA (or VMW_SATP_CX, etc.)

Note that this will not change/update existing LUNs/devices.  I recommend using the vSphere client for this unless you have many that need to be updated.  In this case, use the following command:
FOR /F %G IN ('esxcli --server [HOSTNAME] --username root --password [PASSWORD] storage nmp device list ^| findstr naa.600') DO esxcli --server [HOSTNAME] --username root --password [PASSWORD] storage nmp device set --device %G --psp VMW_PSP_RR

Note 1: In the above command, I loop through and set the RR policy for all devices that begin with "naa.600" - set the NAA ID for your environment as needed.

Note 2: the Linux command uses grep but Windows has FINDSTR.  Took a bit for me to figure out but if nothing else, this is the big find in this article - your welcome!

Set the IOPs Value
Finally, most array manufacturers recommend setting IOPS to "1".  To change the IOPs parameter, use the following command (again, thank you FINDSTR!):
FOR /F %G IN ('esxcli --server [HOSTNAME] --username root --password [PASSWORD] storage nmp device list ^| findstr naa.6005') DO esxcli --server [HOSTNAME] --username root --password [PASSWORD] storage nmp psp roundrobin deviceconfig set -d %G --iops 1 --type iops

Note:  As in the previous command, I loop through and set the IOPs parameter for devices that start with "naa.600" - set this for your environment as needed.

Now the "devices" hosted on your array should be optimally configured for use by vSphere 5.  Don't forget to go through and check each host.

Other Helpful Commands

esxcli storage vmfs extent list
esxcli storage core device detached list
esxcli storage core device detached remove -d [NAA ID]
esxcli storage core adapter rescan [ -A vmhba# | --all ]
esxcli storage filesystem list
esxcli storage filesystem unmount [-u <UUID> | -l <label> | -p <path> ]

ERROR: VMware ESXi with 3PAR SAN and Dead LUN 254

Problem

Last week I discovered  a couple of error messages in the vmkernel.log file that caused me some concern:

2012-06-22T18:18:45.806Z cpu17:939673)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:972:Could not select path for device "Unregistered Device".
2012-06-22T18:18:45.806Z cpu17:939673)WARNING: NMP: nmpPathClaimEnd:1195:Device, seen through path vmhba2:C0:T2:L254 is not registered (no active paths)

I did a "esxcfg-mpath -l" and found 4 dead paths - 2 to each fibre HBA.  I never provisioned a LUN with an ID of 254.  Maybe this was a "special use" LUN?  I doubted it because my HP EVA has one of these and the device is listed in vCenter.  However, there were no devices with LUN ID of 254 listed anywhere in vCenter, only 4 dead paths.

Solution

After a focused Google search, I found the answer.  The HP 3PAR guy that came out and did the installation had us use a host persona of "1 - Generic" when we should have used "6 - Generic-legacy".  Luckily these can be changed on the fly via the InForm Management Console.

After making the change in the IMC, I rescaned each of the hosts and the messages stop appearing in the vmkernel.log and the dead paths were no longer listed in the vSphere Client.

Here are the relevant sites I found per the Google search:

Fix - 3PAR with VMware- LUN 254 Dead

3PAR/VMware LUN 254 Dead???

And, of course, I always recommend following the manufacturer's best practices:

HP 3PAR VMware ESX Implementation Guide

Looks like the guide was recently updated and it does recommend using the persona of "6 - Generic legacy".

Another mystery solved.