Category Archives: EMC

EMC PowerPath Internal Error Migrations May Be Pending Fix

A host side migration between arrays can be a nerve racking task especially when you come across issues. Data loss is a constant fear in the back of your mind and what is your fail-back plan should you need to execute it. During a PowerPath migration, I learned the hard way that a host side copy of the boot-from-san lun is NOT supported. After setting up the migration and upon the sync command the Windows machine froze to a halt until it went offline.

After troubleshooting it was clear that the EMC PowerPath Migration Enabler Service needed to be disabled for the Windows machine to fully boot. After enabling EMC PowerPath Migration Enabler after the host was booted would immediately cause the Windows host to go unresponsive and hard power cycle was the only fix.

I could not start the PowerPath Migration Enabler service to abort the session since it would immediately freeze the server and secondly I was unable to uninstall PowerPath Migration Enabler since there was a session pending. I was in a pickle!

EMC PowerPath PPME Removal Migration Pending

After a support ticket with EMC, the resolution requires you to manually remove the PowerPath Migration Enabler database and keys within the registry. After preforming a few deletions then you will be able to star the service successfully without freezing your server and with no active sessions going.

  1. Delete the UMD by deleting the files from C:\Program Files\EMC\PPME\db*.* 
  2. Delete the all subkeys with Prefix “dm_” EXCEPT for dev_conf under, HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_*.
    The Keys would be dm_ac, dm_control_io_to_clones, dm_funnel_io, dm_wc.EMC PowerPath PPME Removal Migration Pending_Registry
  3. Reboot.

 

Advertisements
Tagged , , , ,

EMC VNX Check Hotspare Rebuild Successfully After Disk Failure

There are two status codes in a SP Collect that a CE should look at before removing a failed drive.

67d is the hexadecimal address which can be found in the SP Collect logs which indicates that the drive successfully failed over to the hot spare. This can be found in the SPA_navi_getlog.txt and the SPB_navi_getlog.txt respectively.

78b is the address which indicates that the drive has been removed from the array.

EMC VNX SPA Log Rebuild Screenshot

EMC VNX SPB Log Rebuild Screenshot

I received the two tips in this article from a local CE that came onsite one day after a disk failure and is definitely worth a mention.

Tagged , , , , , , ,

EMC VNX Check Transitioning Equalizing Faulted LUN Disk

After a disk has faulted, the disk goes into a transitioning or an equalizing state. EMC uses different terminology to describe the action of repairing a faulted disk. Transitioning or equalizing times can vary based on the type and speed of the disk but your SAN utilization will have a direct impact on how fast a drive rebuilds. No where in Unisphere does it indicate a status or an approximate ETA time of when the drive is to complete.

You can check the state of the transitioning or an equalizing state but it is burried in the SP Collect logs. Once you perform an SP Collect, it is one large zip file. Expand the zip file and you will find more zip files. There is a zip file that ends in _sas.zip, within that file there will be SPA_cfg_info.txt or SPB_cfg_info.txt depending on which service processor you performed the SP Collect on. Within that file look for the information that shows you the status of the transitioning/equalizing process!

EMC VNX Equalizing Log

Tagged , , , , , , , , ,

EMC VNX Integration Quest Vintela Authentication Services

Quest (Vintela) Authentication Services (VASD) provides schema extensions to Active Directory to authentication against LDAP by providing SID to UID/GID mappings and vice versa. In multiprotocol environments that provide NFS and SMB protocols to the same underlying data it becomes tricky with permissions and file ownership since it must be maintained in the *nix and Windows environments. There are many ways to provide mappings for this situation but for those that use Quest (Vintela) Authentication Services (VASD) here is an proven guide on how to leverage your existing authentication services against the EMC VNX/Celerra for file. ldap.conf

# --------------------------------------------------------------------
# This template must be copied to /.etc/ldap.conf when the ldap
# server[s] used by the data mover is using the Quest Vintela
# Authentication Services schema installed on Windows Server.
# --------------------------------------------------------------------
nss_schema			rfc2307bis

nss_base_passwd		DC=northwind,DC=lan?sub
nss_base_shadow		DC=northwind,DC=lan?sub
nss_base_group		DC=northwind,DC=lan?sub

nss_map_objectclass posixAccount		User
nss_map_objectclass shadowAccount		User
nss_map_objectclass posixGroup			Group

nss_map_attribute	uid					sAMAccountName
nss_map_attribute	uniqueMember		member
nss_map_attribute	givenname			givenName
nss_map_attribute 	ou					description
nss_map_attribute	shadowLastChange	pwdLastSet
nss_map_attribute 	homeDirectory		unixHomeDirectory
nss_map_attribute	uidNumber			uidNumber
nss_map_attribute	gidNumber			gidNumber
nss_map_attribute	gecos				gecos
nss_map_attribute	loginShell			loginShell

nsswitch.conf

passwd:         files ldap
group:          files ldap
hosts:          files dns ldap
netgroup:       files ldap

Copy the contents of ldap.conf and push to server_2.

[nasadmin@CELERRA ~]$ cp ***ldap.conf*** /nas/site/ldap.conf.server_2
[nasadmin@CELERRA ~]$ server_file server_2 -put /nas/site/ldap.conf.server_2 ldap.conf

Copy the contents of nsswitch.conf and push to server_2.

[nasadmin@CELERRA ~]$ cp ***nsswitch.conf*** /nas/site/nsswitch.conf.server_2
[nasadmin@CELERRA ~]$ server_file server_2 -put /nas/site/nsswitch.conf.server_2 nsswitch.conf

Disables the default usermapper and removes any existing usermapper configurations

[nasadmin@CELERRA ~]$ server_usermapper server_2 -disable
[nasadmin@CELERRA ~]$ server_usermapper server_2 -remove -all

Bind the primary DataMover (server_2) to the domain and associate a binding user distinguished name and password for LDAP authentication lookups.

[nasadmin@CELERRA ~]$ server_ldap server_2 -set -p -basedn "DC=northwind,DC=lan" -binddn "CN=EMCServiceUser,OU=Users,DC=northwind,DC=lan" -servers 192.168.1.100,192.168.1.101

Use the following commands to verify connectivity and lookup capabilities against LDAP.

[nasadmin@CELERRA ~]$ server_ldap server_2 -info
server_2 :
LDAP domain:      northwind.lan
State:            Configured - Connected
Schema:           Active Directory
Base dn:          dc=northwind,dc=lan
Bind dn:
Configuration:    RFC-2307 defaults
LDAP server:      192.168.1.100 - Port: 389 - Active
    SSL:          Not enabled
LDAP server:      192.168.1.101 - Port: 389 - Spare
    SSL:          Not enabled

[nasadmin@CELERRA site]$ server_ldap server_2 -service -status
server_2 :
LDAP domain "northwind.lan" is active - Configured with file "ldap.conf"

[nasadmin@CELERRA ~]$ server_ldap server_2 -lookup -user jsmith
server_2 :
user: jsmith, uid: 500, gid: 301, homeDir: /northwind/home/jsmith

Set parameters to utilize LDAP to look up the SID to UID/GID mappings.

[nasadmin@CELERRA ~]$ server_param server_2 -facility cifs -modify resolver -value 1
[nasadmin@CELERRA ~]$ server_param server_2 -facility cifs -modify useADMap -value 1
server_2 : done
Warning 17716815753: server_2 : You must stop and start the service associated with the cifs facility for changes to useADMap to take effect

**** REBOOT server_2 DATAMOVER **** This will disrupt connectivity to the DataMover and should be done in a maintenance window.

server_cpu server_2 -reboot warm -monitor now

We have to create the mapping which will force the user to be looked up in LDAP. Finally, check that the user was looked up successfully.

[nasadmin@CELERRA ~]$ server_cifssupport server_2 -secmap -create -name jsmith -domain northwind
[nasadmin@CELERRA ~]$ server_cifssupport server_2 -secmap -list
server_2 : done

SECMAP USER MAPPING TABLE

UID         Origin      Date of creation         Name                        SID
500	        ldap        Fri Aug 31 07:40:23 2012 NORTHWINDjsmith           S-1-5-15-4376b78a-a9aad504-d4f8c2d6-460

If you are having troubles use the following command to provide verbose information which will contain any informational, warning or critical error messages.

[nasadmin@CELERRA ~]$ server_ldap server_2 -info -verbose

Please let me know if you have any questions! Thanks!

Tagged , , , , , , , , , , , ,

EMC VNX Celerra NAS Multiprotocol Environment Planning

A mixed protocol environment means that you have two means of accessing the same data generally through SMB/CIFS/Samba and NFS. SMB is natively supported in Windows and NFS is natively in *nix environments.

There are complexities an organization will always run into when running in a mixed protocol environment which is a result of how Windows and *nix systems handle security at the core of their respective operating systems. *nix platforms handles permissions by using permission bits and Windows uses Access Control Lists (ACL). Ownership of a file is either defined by a User ID (UID) in the Unix world or a Security ID (SID) in Windows.

Fundamentally, *nix and Windows platforms handles files, permissions and sharing completely differently and this introduces new complexities because of this business requirement. It is my personal advice, try to avoid running a mixed protocol environment if possible. If a there is a requirement that mandates this need, then design and plan accordingly to prevent future headaches down the road.

This article is not a how-to guide on how to implement a mixed protocol environment since each implementation is unique since there is a complexity reason for driving to use the mixed protocol environment to begin with. Use the following as things to understand and to take in account BEFORE you implement a mixed protocol environment.

This article is a cliff-notes version of the following EMC documents,  Multiprotocol Environment Guide and the User Mapper Guide.

It is important that the sections below are completely understood a head of time for accurate planning of multiprotocol environment. If a change is required after the filesystem is already in production, there is no easy way to change these settings without creating another filesystem and rsync’ing/robocopy’ing the data between filesystems.

Understanding CIFS and Unix Permissions

CIFS

  • Access Control Lists
  • SID Ownership
  • Username Passwords

*nix

  • Permission Bits
  • UID & GID Ownership Bits
  • IP Address Access

Understand the CIFS User ID Resolution Options

• Active Directory
• LDAP Directory
• Local Files
• Network Information Service (NIS)
• Usermapper (Internal or External)

To understand your mappings in technical detail please review the EMC Naming Services Guide.

NOTE: EMC recommends use of Internal Usermapper in Windows-only environments.
NOTE: Avoid static mappings if possible, it becomes a management nightmare. Even if you use custom Active Director Schema Attributes/software to deal with Windows-to-Unix mappings, the Naming Services Guide will provide you with the information to accommodate custom schema attributes and fields.

Understanding Access-checking Policies

Access-checking policy Description
Native (default)
  • Access to a file or directory through NFS or UNIX authenticated FTP is granted only if the UNIX permissions on the file or directory allow it.
  • Access through CIFS or Windows authenticated FTP is granted only if the Windows permissions on the file or directory allow it.
  • ACL and UNIX permissions are maintained for every file and directory.
  • A change in permissions on a file system object in NFS has no impact on permissions in CIFS and a change in permissions on a file system object in CIFS has no impact on permissions in NFS.
NT
  • Access to a file or directory through NFS or UNIX authenticated FTP is granted only if the UNIX and Windows permissions allow it.
  • Access through CIFS or Windows authenticated FTP is granted only if the Windows permissions allow it (the UNIX permissions do not have any effect).
  • ACL and UNIX permissions are maintained for every file and directory.
  • A change in permissions on a file system object in NFS has no impact on permissions in CIFS and a change in permissions on a file system object in CIFS has no impact on permissions in NFS.
UNIX
  • Access to a file or directory through NFS or UNIX authenticated FTP is granted only if the UNIX permissions allow it (the Windows permissions do not have any effect).
  • Access through CIFS or Windows authenticated FTP is granted only if the UNIX and Windows permissions on the file or directory allow it.
  • ACL and UNIX permissions are maintained for every file and directory.
  • A change in permissions on a file system object in NFS has no impact on permissions in CIFS and a change in permissions on a file system object in CIFS has no impact on permissions in NFS.
SECURE
  • Provides the greatest security across CIFS and NFS.
  • Access to a file or directory through either NFS or FTP or CIFS is granted only if the UNIX and Windows permissions on the file or directory allow it.
  • ACL and UNIX permissions are maintained for every file and directory.
  • A change in permissions on a file system object in NFS has no impact on permissions in CIFS and a change in permissions on a file system object in CIFS has no impact on permissions in NFS.
MIXED
  • Access to a file or directory through either NFS or FTP or CIFS is always determined by the ACL.
  • ACL and UNIX permissions are maintained for every file and directory.
  • ACLs for files and directories are created from the protocol that last set or changed the permissions. For example, if an NFS client sets or changes permissions on a file or directory, the ACL is rebuilt based on the UNIX mode bits. If a CIFS client sets or changes permissions on a file or directory, the ACL is built based on the standard Windows permissions.
  • In all cases, the ACL determines file and directory access regardless of whether the client is using the NFS, CIFS or FTP protocol.
  • ACL permissions are more granular than UNIX mode bits, consequently not all permissions set in an ACL can be translated to UNIX mode bits. In some cases, the mode bits might show more permissions than a user actually has
MIXED_COMPAT
  • Access to a file or directory through NFS or FTP or CIFS is determined by which protocol (NFS or CIFS) last set or modified the permissions.
  • ACL and UNIX permissions are maintained for every file and directory.
  • If the permissions of a file or directory are set or changed from a CIFS client, then access is determined by the ACL (EXPLICIT ACL). UNIX mode bits are generated based on the ACL but are not used for access checking.
  • If the permissions of a file or directory are set or changed from a UNIX client, then UNIX mode bits dictate access. An ACL is still created but is not used for access checking.
  • ACL permissions are more granular than UNIX mode bits, consequently not all permissions set in an ACL can be translated to UNIX mode bits. In some cases, the mode bits might show more permissions than a user actually has.


Understand Permission Translations

ACL to Unix Rights
EMC VNX Celerra ACL to Unix Rights Screenshot

Unix to ACL Rights
EMC VNX Celerra Unix to ACL Rights Screenshot

Understand your Inheritance

NATIVE, UNIX, NT, and SECURE
EMC VNX Celerra Native Inheritance Modes

MIXED and MIXED_COMPAT
EMC VNX Celerra Mixed Inheritance Modes

You can find this information and a wealth of other information in EMC’s Multiprotocol Environment Guide and User Mapper Guide.

Other Observed Notes:

  • Taking ownership from Windows Permissions/ACL will change the underlying UID/GID owner even though it is stated otherwise in Native Mode.
Tagged , , , , , , , , , , , ,

EMC DataDomain Default BIOS CMOS Password

If you ever have a need to get into an EMC DataDomain BIOS screen the default password is a simple pattern based off the major series model number. For example:

DD460 = d400d (delta four zero zero delta)

DD670 = d600d (delta six zero zero delta)

DD880 = d800d (delta four zero zero delta)

The pattern is simple, “d + major series model number + d

Tagged , , , , , , ,

EMC Isilon Clear Cached UID GID Mappings

During testing, the EMC Isilon storage was successfully reading the Active Directory Attributes but the data was incorrect. The data that was in the uid & gid fields were id’s that were automatically generated by the Isilon storage itself. The id’s that were generated by the Isilon storage took precedence over any mappings that were stored in the Active Directory Schema.

The fix was to delete all mappings from the Isilon storage by logging into the CLI and issuing a

isi_clean_idmap --clean-idmap

… to clean up all the mappings!

EMC Isilon Test Mappings Screenshot

EMC Isilon Clean Up Mappings Screenshot

Tagged , , , , ,

EMC Isilon Integration Quest Vintela Authentication Services VASD

There are many challenges that are faced when an organization is forced to run a mixed protocol environment to serve up the same data. This introduces additional management tasks and additional complexities. In a mixed protocol environment you must manage Windows Access Control Lists (ACL) to enforce Windows permissions in addition to managing *nix user/group ids with permission bits to control access. The EMC Isilon solution is a great platform to support mixed protocol environments. In my opinion this far, the Isilon platform is the ideal solution to deal with a mixed protocol environment due to it’s integration with authentication services such as Windows Active Directory or any LDAP service. There are a number of products that provide extensions to Windows Active Directory to provide UID/GID authentication and mappings. One of those products is Quest’s (Vintela) Authentication Services.

Quest Authentication Services uses five fields in the Windows Active Directory. These are the five attributes that Quest Authentication Services uses:

  • gecos
  • uidNumber
  • gidNumber
  • loginShell
  • unixHomeDirectory

Using that information, we are now able to integrate the Quest (Vintela) Authentication Services with the EMC Isilon NAS Storage. The screenshot below displays the correct settings to use on the EMC Isilon storage to integrate with Quest Authentication Services.

EMC Isilon Quest Authentication Services Settings Screenshot

Finally, test your mappings to ensure your AD/LDAP authentication and mappings work correctly.

Tagged , , , , , , , , , , , ,

Naviseccli Take Default Ownership Script

In event of a path failure or a service processor failure, the LUNs ownership will transfer to the service processor still serving I/O. It is possible to take back LUN ownership through Unisphere but with a GUI this can be a long task. This script will take back each service processors default ownership.

NOTE: Don’t forget to edit the script for your service processor’s IP address information.

NOTE: It also requires you to have naviseccli command line tools installed on the host which you are running the script from.

NOTE: Rename the txt file to a cmd file.

Download here, VNXTrespassMine.txt

Tagged , , , , ,

Restart EMC VNX Management Services in Unisphere

Restarting the Management Services for EMC Unisphere is simple and it does not require an outage. The process will require you to log into each Service Processor separately through a web interface.

1. Open up a web browser and goto https://<Service Processor IP/FQDN>/setup
2. Login using the ‘sysadmin’ user account and password.
EMC VNX Restart Management Services

3. Scroll down to ‘Restart Management Services’ and press that button.
EMC VNX Restart Management Service
4. Confirm the Management Services Restart and wait until the restart has completed. Note, this can take up to 10 minutes.
5. Once the first Service Processor reboot has occurred and you are able to log back into that Service Processors’ Web Services then proceed to the second Service Processor.

This should clear up any Unisphere GUI issues or incorrect reporting of information on the Host Agent’s page.

Tagged , , , , , , ,