Monthly Archives: August 2011

EMC Host Agent Not Registering with Unisphere

There is an issue with a host agent on a Linux machine that was installed correctly, the firewalls were not preventing any traffic between the Service Processors and the host agent was still not registering. I worked intermittently on this particular issue over the course of a few days. After narrowing it down to being an issue with the host. I found my answer in the /etc/hosts file!

Problem File

[root@linuxsys ~]$more /etc/hosts
127.0.0.1       linuxsys localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6

Good File

[root@linuxsys ~]$more /etc/hosts
127.0.0.1       localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6

The host was named linuxsys and it referenced it self in the host file which caused a conflict while trying to communicate between the service processors and the host agent.

Tagged , , , , , , ,

SNMP v3 + Cisco IOS Crash Course

SNMPv3 allows for authentication and encryption (AES, DES, 3-DES) for managing core routers and switches. CiscoWorks LAN Manager Solution (LMS) requires SNMPv3 to be enabled and setup correctly for doing port VLAN tagging and a various other features through the LMS web interface. Here is a simple crash course on setting up SNMPv3 on an Cisco IOS device.

snmp-server view readview internet included
snmp-server view writeview internet included

This step will attach a view to a group. To keep things simple we will follow the same layout as the views, a group for read (readgroup) and a group for read/write (writegroup).

snmp-server group readgroup v3 priv read readview
snmp-server group writegroup v3 priv write writeview

Add a username to the readgroup or writegroup with an authpassword and a passphrase. Repeat this step for additional users.

snmp-server user <username> <group> v3 auth sha <authpass> priv aes 256 <passphrase>

Finally, specify access to the host and user that your monitoring software will use to connect to the Cisco Switch via SNMP.

snmp-server host <ipaddress/fqdn> version 3 priv <user>
Tagged , , , , , , , , ,

EMC PowerPath Multipathing in RedHat Linux Guide

First and foremost, I want to give the credit to Will’s Notes for the original article on Multipathing in RHEL5. I was able to use this guide with a Xiotech SAN to configure Multipathing. EMC makes it a ton easier to configure Multipathing. EMC has a product called PowerPath, PowerPath can be used with or without a license. If you install and use PowerPath in an unlicensed fashion you have an active-passive connection back to the EMC SAN. If PowerPath is licensed, this allows for an active-active connection back to the EMC SAN. Active-active is not only highly available it is load balanced.

Configuring PowerPath was rather easy to my surprise. Download PowerPath from powerlink.emc.com for your correct distribution and install the RPM.

[root@localhost ~]# rpm -iv EMCPower.LINUX-5.6.0.00.00-143.RHEL5.x86_64.rpm

If you are using PowerPath in licensed mode, register license key with the first command and check the registration of PowerPath with the second command listed below.

[root@localhost ~]# emcpreg -install
[root@localhost ~]# powermt check_registration

Once PowerPath is installed you can rescan the bus or if you do not know how simply reboot RHEL.

To view the information about PowerPath issue the following command.

[root@localhost ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=<SERIALNUMBER> [STROAGEGROUP_NAME]
Logical device ID=STORAGEGROUP_WWN [LUN 400]
state=alive; policy=BasicFailover; priority=0; queued-IOs=0;
Owner: default=SP B, current=SP B       Array failover mode: 4
==============================================================================
--------------- Host ---------------   - Stor -   -- I/O Path --  -- Stats ---
###  HW Path               I/O Paths    Interf.   Mode    State   Q-IOs Errors
==============================================================================
3 qla2xxx                  sdb       SP A0     unlic   alive       0      0
3 qla2xxx                  sdd       SP B0     unlic   alive       0      0
3 qla2xxx                  sdf       SP A4     active  alive       0      0
3 qla2xxx                  sdh       SP B4     active  alive       0      0
4 qla2xxx                  sdj       SP A1     unlic   alive       0      0
4 qla2xxx                  sdl       SP B1     unlic   alive       0      0
4 qla2xxx                  sdn       SP A5     unlic   alive       0      0
4 qla2xxx                  sdp       SP B5     unlic   alive       0      0

Pseudo name=emcpowerb
CLARiiON ID=<SERIALNUMBER> [STROAGEGROUP_NAME]
Logical device ID=STORAGEGROUP_WWN [LUN 401]
state=alive; policy=BasicFailover; priority=0; queued-IOs=0;
Owner: default=SP A, current=SP A       Array failover mode: 4
==============================================================================
--------------- Host ---------------   - Stor -   -- I/O Path --  -- Stats ---
###  HW Path               I/O Paths    Interf.   Mode    State   Q-IOs Errors
==============================================================================
3 qla2xxx                  sdc       SP A0     unlic   alive       0      0
3 qla2xxx                  sde       SP B0     unlic   alive       0      0
3 qla2xxx                  sdg       SP A4     active  alive       0      0
3 qla2xxx                  sdi       SP B4     active  alive       0      0
4 qla2xxx                  sdk       SP A1     unlic   alive       0      0
4 qla2xxx                  sdm       SP B1     unlic   alive       0      0
4 qla2xxx                  sdo       SP A5     unlic   alive       0      0
4 qla2xxx                  sdq       SP B5     unlic   alive       0      0

Once PowerPath is installed and PowerPath is able to access the LUNs that are presented to the host, begin to create the filesystem. Create the file system like you would format any ordinary storage device but instead of /dev/sda, /dev/sdb, etc… EMC’s PowerPath devices are /dev/emcpowera, /dev/emcpowerb, /dev/emcpowerc, etc..

Tagged , , , , , , , , , , , ,

EMC Naviseccli View Service Processor Cache Utilization

After hooking up a host to an EMC SAN it is very helpful to have the host that is connected to the SAN to have Naviseccli installed along with the Host Agent. Naviseccli allows you to connect to the Service Processors to gather information among other things.

Dirty Cache Pages is data in the cache that is a solid state physical format waiting to be written to disk. You can monitor how full the cache is by the following commands on each Service Processor.

[root@localhost bin]# ./naviseccli -h <IP/Hostname SPA> getcache -pdp
[root@localhost bin]# ./naviseccli -h <IP/Hostname SPB> getcache -pdp

If you receive the following error, “Security file not found. Already removed or check -secfilepath option.” Issue to following commands to allow security to issue commands remotely.

[root@localhost bin]# ./naviseccli -User sysadmin -Password <password> -Scope 0 -h <IP/Hostname SPA> -AddUserSecurity
[root@localhost bin]# ./naviseccli -User sysadmin -Password <password> -Scope 0 -h <IP/Hostname SPB> -AddUserSecurity
Tagged , , , , , , , , , , ,

Symantec Backup Exec RALUS Linux Agent ///// Error

Dealing with Symantec is a complete nightmare and it’s more of a nightmare when you feel like you’ve have been held hostage by a product that does not have the support or the quality of code that it needs to be considered an ‘enterprise’ software solution. As you’ve already guessed, we are talking about Symantec Backup Exec 2010 R2.

I have had the RALUS/VRTSralus Linux Agent working just fine for months until what I suspect was a LiveUpdate patch which may have broken the Linux remote agent. The root cause of this particular issue is really unknown but has to do with the error listed below.

43bd9940 Sat Jul 30 01:48:19 2011 :
vx_stat_by_name: Getting stat of /backup/dir/home/user/build/emcgrab/tools/lite//////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
//////////////////////////////////ecc.sh
43bd9940 Sat Jul 30 01:48:19 2011 :
vx_get_link_target for /backup/dir/home/user/build/emcgrab/tools/lite//////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////

After speaking with Symantec support and a bunch of go around over the telephone and emails, my team and I are lead to believe that Soft-link directories in Linux is causing the issue above. There is a known limitation with Backup Exec’s Linux agent which prevents paths being longer than 255 characters. I believe when the agent encounters a soft-linked directory, the agent fails and appends trailing slashes in an infinite loop until it hits the 255 character limit which causes the remote agent to crash. The backup job then terminates early and forces the job to Verify if you job is configured to do so.

I have disabled the backup of soft-linked directories at the job level and the global application level to determine if this has resolved the issue. NOTE: This is to a RALUS agent which has an NFS mount pointing to the data.

To get this log out of the Backup Exec RALUS Linux Agent, follow this article on how you can put the agent into debug mode, TECH35477. If you plan on contacting Symantec be sure that your operating system is supported, in our case RedHat Enterprise Linux 5.4 is the lastest supported RedHat Linux, in addition, know how to put the Linux Agent in debug mode and how to perform core dumps, TECH53188. Finally, be ready to work at their snail’s pace.

Tagged , , , , , , ,

Setup Linux RedHat Up.time SSL Agent

The Windows version of the SSL Up.time Agent was cover by the following article, Setup an Windows Up.time SSL Agent (stunnel & Scripted).

This version goes over setting up a Linux agent on a RedHat based distribution. The distribution that I was working with through this guide was RedHat Enterprise Linux 5.6. First start off by going to the Up.time Software website and downloading the latest rpm package of the Linux Up.time Agent.

Here is the step by step break down of how I was able to configure the Up.time Agent to use SSL in RHEL.

Install the up.time Linux Agent by issuing the following command.

sudo rpm -ivh uptimeagent-5.3.0-linux-x86_64.rpm

Copy-paste the contents to ‘/opt/uptime-agent/conf/agent.conf’, overwrite the existing agent.conf file.

port=9997
user=uptimeagent
mode=xinetd
timeout=60
log_file=/opt/uptime-agent/logs/uptimeagent.log
log_level=NONE

Create ‘uptimeagent.conf’ and copy the contents below to the file

cert=/etc/stunnel/uptime_agent.pem
exec=/opt/uptime-agent/bin/uptimeagent

Generate the SSL certificate

openssl req -x509 -nodes -days 3650 -subj '/C=US/ST=Ohio/L=Cleveland/O=My Company/OU=My Department/CN=uptime-agent' -newkey rsa:1024 -keyout uptime_agent.pem -out uptime_agent.pem

Copy ‘uptimeagent.conf’ and ‘uptime_agent.pem’ to ‘/etc/stunnel’.

sudo cp uptimeagent.conf /etc/stunnel
sudo cp uptime_agent.pem /etc/stunnel

Set the correct ownership and permissions by running the following:

sudo chown root:nobody /etc/stunnel/uptime_agent.pem
sudo chmod 640 /etc/stunnel/uptime_agent.pem
sudo chmod 755 /etc/stunnel/uptimeagent.conf

Replace the contents of ‘/etc/xinetd.d/uptimeagent’ with

service uptimeagent
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = nobody
server = /usr/sbin/stunnel
server_args = /etc/stunnel/uptimeagent.conf
}

In ‘/etc/services’ ensure the last lines look like the following

# *** Installed by the uptimeagent installer
uptimeagent 9997/tcp # uptimeagent agent

Restart the ‘xinetd’ service

sudo /etc/init.d/xinetd restart
Tagged , , , , , , ,

Setup Windows Up.time Monitoring SSL Agent

This guide will show you how to install and configure an Up.time Agent for Windows using SSL. Up.time provides a guide which give a good outline of the steps required to get an Up.time agent configured in Windows using SSL but I think many will find this information very useful since it will automate installations using a batch file and will determine the architecture of Windows then places the files/registry keys in the correct location based on the processor architecture.

First, a certificate must be generated for the Up.time agent to use. OpenSSL tools will be required to generate the appropriate certificate. To generate the certificate, issue the following command from the bin directory of the OpenSSL installation.

openssl req -x509 -nodes -days 3650 -subj '/C=US/ST=Ohio/L=Cleveland/O=My Company/OU=My Department/CN=uptime-agent' -newkey rsa:1024 -keyout uptime_agent.pem -out uptime_agent.pem

Stunnel is the piece of software that wraps around the Up.time agent port and encrypts the traffic since the Up.time agent is not natively using SSL. Stunnel is an open-source project and can be downloaded at www.stunnel.org.

Download Stunnel and extract it to a directory. Place the newly generate certificate in the directory. Overwrite the downloaded stunnel.conf with the stunnel.conf that is listed below. Modify the up.time-stunnel-agent.bat script to the correct UNC/SMB/SAMBA paths and save the file.

Finally, place the remaining files into the directory structure. Follow the directory structure below
Uptime Agent SSL Directory View Screenshot

Don’t forget to goto the Up.time Software website to get the .exe version of the Windows Up.time Agent.

NOTE: If you wish to change the port numbers that is in hexadecimal in the UptimeCMDPassword_x86.reg and UptimeCMDPassword_x64.reg files and must be changed in the stunnel.conf file.

The rest should explain it self. If you have any questions, post a comment and I will try to help you the best that I can.

Continue reading

Tagged , , , , , ,

Update Firmware on Emulex QLogic HBA

During a SAN installation, storage vendors indicate a minimum supported firmware version for HBAs during the specification/building process. Upgrading HBA adapters will ensure that your configuration is supported by the storage vendor should any issues arise. In my particular situation, the QLogic branded HBA was an Emulex and the firmware can be found at http://www.emulex.com/downloads/emulex.html and the tool to update the firmware can be found at http://www.emulex.com/downloads/emulex/cnas-and-hbas/utilities/offline-utilities/dos-offline-utility.html

NOTE: That I had to use a previous version of the DOS offline utility to perform the firmware on the particular LPe12000 I was working with. The version that worked successfully was version 1.0a19.

I used FDOEMCD.builder to create a bootable ISO and issue the following command to update the firmware on the QLogic HBA.

C:\dos\lpcfg.exe download n=1 i=C:\<FILE>.PRG
Tagged , , , , ,