Tag Archives: Storage

vSA vSphere Storage Appliance Performance Benchmark Test

The article below goes into depth with my experience with VMware vSA Performance Benchmark Testing. I’ve tried to be as detailed as possible to give you a complete picture of my findings. I believe that there is a space where storage virtualization may thrive but with recent experience with the VMware vSA product, I am less than satisfied with the results, manageability and most of all performance. I believe storage virtualization has a few more years until maturity until it can be truly considered a serious candidate in the small & remote office scenarios.  This statement holds true for other two/three node storage virtualization technologies including Falconstor’s storage virtualization.

VMware Version Information

VMware vCenter Server 5.1.0, 947673
VMware vStorage Appliance 5.1.3, 1090545
VMware ESXi 5.1 U1, HP OEM Bundle, 1065491 (VMware-ESXi-5.1.0-Update1-1065491-HP-5.50.26.iso)

HP ProLiant DL385 G2 Hardware Configuration
– 4 CPUs x 2.6 GHz
– Dual-Core AMD Opteron Processor 2218
– AMD Opteron Generation EVC Mode
– HP Smart Array P400, 512MB Cache, 25% Read / 75% Write
– RAID-5, 8x 72 GB 10K RPM Hard Drives
– HP Service Pack 02.2013 Firmware

vStorage Appliance Configuration
– 2 Node Cluster
– Eager Zero Full Format
– VMware Best Practices

IOZone Virtual Machine Configuration
– Oracle Linux 6.4 x86_64
– 2 vCPU
– 1 GB Memory
– 20 GB Disk, Thick Eager Zero Provisioned
– VMware Tool (build-1065307)

IOZone Test Paramaters
/usr/bin/iozone -a -s 5G -o

-a   Used to select full automatic mode. Produces output that covers all tested file operations for record sizes of 4k to 16M for file sizes of 64k to 512M.

-s #   Used to specify the size, in Kbytes, of the file to test. One may also specify -s #k (size in Kbytes) or -s #m (size in Mbytes) or -s #g (size in Gbytes).

-o   Writes are synchronously written to disk. (O_SYNC). Iozone will open the files with the O_SYNC flag. This forces all writes to the file to go completely to disk before returning to the benchmark.

VMware ESXi/vSA Network Configuration

VMware vSA Architecture

IOZone Performance Benchmark Results

vSA Read Graph

vSA Stride Read Graph

vSA Random Read Graph

vSA Backward Read Graph

vSA Fread Graph

vSA Write Graph

vSA Random Write Graph

vSA Record Rewrite Graph

vSA Fwrite Graph

Download RAW Excel Data


The vSA performed far less than the native onboard storage controller which was expected due to the additional layer of virtualization. I honestly expected better performance out of the 8-disk RAID-5 even without storage virtualization since they were 10,000 RPM drives. On average, across all the tests there is 76.3% difference between the native storage and the virtualized storage! Wow! That is an expensive down grade! I understand that the test bed was using not the latest and greatest hardware but in general terms of disk performance is generally limited by the spinning platter. I would really be interested in seeing the difference using newer hardware.

I believe this only depicts a fraction of the entire picture, performance. There is other concerns that I have at the moment with storage virtualization such as complexity and manageability. I found the complexity to be very frustrating while setting up the vSA, there are many design considerations and limitations with this particular storage virtualization solution most of which were observed during the test trails. The vSA management is a Flash-based application which had it’s quirks and crashes as well. Crashes at a storage virtualization layer left me thinking that this would be a perfect recipe for data loss and/or corruption. In addition, a single instance could not manage multiple vSA deployments due to IP addressing restrictions which was a must for the particular use-case which I was testing for.

For now, storage virtualization is not there yet in my opinion for any production use. It has alot of room to grow and I will certainly be interested in revisiting this subject down the road since I believe in the concept.

Reference Articles That May Help 

Tagged , , , , , , ,

VMware vSA vSphere Storage Appliance Installation Parameters

During recent troubleshooting of installing VMware’s vSphere Storage Appliance, I’ve under covered two installation parameters which could be needed depending on your environment and installation.

The first one allows you to change the username and password that the vSA will use to connect to vCenter with.

VMware-vsamanager.exe /v"VM_SHOWNOAUTH=1"

VMware vSA Install Screenshot

The other parameter, allows you to specify the vCenter IP addresss or FQDN.

VMware-vsamanager.exe /v"VM_IPADDRESS=<fqdn/ip>"

VMware vSA Install Screenshot

… or the two can be combined like

VMware-vsamanager.exe /v"VM_SHOWNOAUTH=1 VM_IPADDRESS=<fqdn/ip>"
Tagged , , , ,

EMC VNX Check Hotspare Rebuild Successfully After Disk Failure

There are two status codes in a SP Collect that a CE should look at before removing a failed drive.

67d is the hexadecimal address which can be found in the SP Collect logs which indicates that the drive successfully failed over to the hot spare. This can be found in the SPA_navi_getlog.txt and the SPB_navi_getlog.txt respectively.

78b is the address which indicates that the drive has been removed from the array.

EMC VNX SPA Log Rebuild Screenshot

EMC VNX SPB Log Rebuild Screenshot

I received the two tips in this article from a local CE that came onsite one day after a disk failure and is definitely worth a mention.

Tagged , , , , , , ,

EMC VNX Check Transitioning Equalizing Faulted LUN Disk

After a disk has faulted, the disk goes into a transitioning or an equalizing state. EMC uses different terminology to describe the action of repairing a faulted disk. Transitioning or equalizing times can vary based on the type and speed of the disk but your SAN utilization will have a direct impact on how fast a drive rebuilds. No where in Unisphere does it indicate a status or an approximate ETA time of when the drive is to complete.

You can check the state of the transitioning or an equalizing state but it is burried in the SP Collect logs. Once you perform an SP Collect, it is one large zip file. Expand the zip file and you will find more zip files. There is a zip file that ends in _sas.zip, within that file there will be SPA_cfg_info.txt or SPB_cfg_info.txt depending on which service processor you performed the SP Collect on. Within that file look for the information that shows you the status of the transitioning/equalizing process!

EMC VNX Equalizing Log

Tagged , , , , , , , , ,

EMC VNX Integration Quest Vintela Authentication Services

Quest (Vintela) Authentication Services (VASD) provides schema extensions to Active Directory to authentication against LDAP by providing SID to UID/GID mappings and vice versa. In multiprotocol environments that provide NFS and SMB protocols to the same underlying data it becomes tricky with permissions and file ownership since it must be maintained in the *nix and Windows environments. There are many ways to provide mappings for this situation but for those that use Quest (Vintela) Authentication Services (VASD) here is an proven guide on how to leverage your existing authentication services against the EMC VNX/Celerra for file. ldap.conf

# --------------------------------------------------------------------
# This template must be copied to /.etc/ldap.conf when the ldap
# server[s] used by the data mover is using the Quest Vintela
# Authentication Services schema installed on Windows Server.
# --------------------------------------------------------------------
nss_schema			rfc2307bis

nss_base_passwd		DC=northwind,DC=lan?sub
nss_base_shadow		DC=northwind,DC=lan?sub
nss_base_group		DC=northwind,DC=lan?sub

nss_map_objectclass posixAccount		User
nss_map_objectclass shadowAccount		User
nss_map_objectclass posixGroup			Group

nss_map_attribute	uid					sAMAccountName
nss_map_attribute	uniqueMember		member
nss_map_attribute	givenname			givenName
nss_map_attribute 	ou					description
nss_map_attribute	shadowLastChange	pwdLastSet
nss_map_attribute 	homeDirectory		unixHomeDirectory
nss_map_attribute	uidNumber			uidNumber
nss_map_attribute	gidNumber			gidNumber
nss_map_attribute	gecos				gecos
nss_map_attribute	loginShell			loginShell


passwd:         files ldap
group:          files ldap
hosts:          files dns ldap
netgroup:       files ldap

Copy the contents of ldap.conf and push to server_2.

[nasadmin@CELERRA ~]$ cp ***ldap.conf*** /nas/site/ldap.conf.server_2
[nasadmin@CELERRA ~]$ server_file server_2 -put /nas/site/ldap.conf.server_2 ldap.conf

Copy the contents of nsswitch.conf and push to server_2.

[nasadmin@CELERRA ~]$ cp ***nsswitch.conf*** /nas/site/nsswitch.conf.server_2
[nasadmin@CELERRA ~]$ server_file server_2 -put /nas/site/nsswitch.conf.server_2 nsswitch.conf

Disables the default usermapper and removes any existing usermapper configurations

[nasadmin@CELERRA ~]$ server_usermapper server_2 -disable
[nasadmin@CELERRA ~]$ server_usermapper server_2 -remove -all

Bind the primary DataMover (server_2) to the domain and associate a binding user distinguished name and password for LDAP authentication lookups.

[nasadmin@CELERRA ~]$ server_ldap server_2 -set -p -basedn "DC=northwind,DC=lan" -binddn "CN=EMCServiceUser,OU=Users,DC=northwind,DC=lan" -servers,

Use the following commands to verify connectivity and lookup capabilities against LDAP.

[nasadmin@CELERRA ~]$ server_ldap server_2 -info
server_2 :
LDAP domain:      northwind.lan
State:            Configured - Connected
Schema:           Active Directory
Base dn:          dc=northwind,dc=lan
Bind dn:
Configuration:    RFC-2307 defaults
LDAP server: - Port: 389 - Active
    SSL:          Not enabled
LDAP server: - Port: 389 - Spare
    SSL:          Not enabled

[nasadmin@CELERRA site]$ server_ldap server_2 -service -status
server_2 :
LDAP domain "northwind.lan" is active - Configured with file "ldap.conf"

[nasadmin@CELERRA ~]$ server_ldap server_2 -lookup -user jsmith
server_2 :
user: jsmith, uid: 500, gid: 301, homeDir: /northwind/home/jsmith

Set parameters to utilize LDAP to look up the SID to UID/GID mappings.

[nasadmin@CELERRA ~]$ server_param server_2 -facility cifs -modify resolver -value 1
[nasadmin@CELERRA ~]$ server_param server_2 -facility cifs -modify useADMap -value 1
server_2 : done
Warning 17716815753: server_2 : You must stop and start the service associated with the cifs facility for changes to useADMap to take effect

**** REBOOT server_2 DATAMOVER **** This will disrupt connectivity to the DataMover and should be done in a maintenance window.

server_cpu server_2 -reboot warm -monitor now

We have to create the mapping which will force the user to be looked up in LDAP. Finally, check that the user was looked up successfully.

[nasadmin@CELERRA ~]$ server_cifssupport server_2 -secmap -create -name jsmith -domain northwind
[nasadmin@CELERRA ~]$ server_cifssupport server_2 -secmap -list
server_2 : done


UID         Origin      Date of creation         Name                        SID
500	        ldap        Fri Aug 31 07:40:23 2012 NORTHWINDjsmith           S-1-5-15-4376b78a-a9aad504-d4f8c2d6-460

If you are having troubles use the following command to provide verbose information which will contain any informational, warning or critical error messages.

[nasadmin@CELERRA ~]$ server_ldap server_2 -info -verbose

Please let me know if you have any questions! Thanks!

Tagged , , , , , , , , , , , ,

EMC DataDomain Default BIOS CMOS Password

If you ever have a need to get into an EMC DataDomain BIOS screen the default password is a simple pattern based off the major series model number. For example:

DD460 = d400d (delta four zero zero delta)

DD670 = d600d (delta six zero zero delta)

DD880 = d800d (delta four zero zero delta)

The pattern is simple, “d + major series model number + d

Tagged , , , , , , ,

EMC Isilon Clear Cached UID GID Mappings

During testing, the EMC Isilon storage was successfully reading the Active Directory Attributes but the data was incorrect. The data that was in the uid & gid fields were id’s that were automatically generated by the Isilon storage itself. The id’s that were generated by the Isilon storage took precedence over any mappings that were stored in the Active Directory Schema.

The fix was to delete all mappings from the Isilon storage by logging into the CLI and issuing a

isi_clean_idmap --clean-idmap

… to clean up all the mappings!

EMC Isilon Test Mappings Screenshot

EMC Isilon Clean Up Mappings Screenshot

Tagged , , , , ,

EMC Isilon Integration Quest Vintela Authentication Services VASD

There are many challenges that are faced when an organization is forced to run a mixed protocol environment to serve up the same data. This introduces additional management tasks and additional complexities. In a mixed protocol environment you must manage Windows Access Control Lists (ACL) to enforce Windows permissions in addition to managing *nix user/group ids with permission bits to control access. The EMC Isilon solution is a great platform to support mixed protocol environments. In my opinion this far, the Isilon platform is the ideal solution to deal with a mixed protocol environment due to it’s integration with authentication services such as Windows Active Directory or any LDAP service. There are a number of products that provide extensions to Windows Active Directory to provide UID/GID authentication and mappings. One of those products is Quest’s (Vintela) Authentication Services.

Quest Authentication Services uses five fields in the Windows Active Directory. These are the five attributes that Quest Authentication Services uses:

  • gecos
  • uidNumber
  • gidNumber
  • loginShell
  • unixHomeDirectory

Using that information, we are now able to integrate the Quest (Vintela) Authentication Services with the EMC Isilon NAS Storage. The screenshot below displays the correct settings to use on the EMC Isilon storage to integrate with Quest Authentication Services.

EMC Isilon Quest Authentication Services Settings Screenshot

Finally, test your mappings to ensure your AD/LDAP authentication and mappings work correctly.

Tagged , , , , , , , , , , , ,

Naviseccli Take Default Ownership Script

In event of a path failure or a service processor failure, the LUNs ownership will transfer to the service processor still serving I/O. It is possible to take back LUN ownership through Unisphere but with a GUI this can be a long task. This script will take back each service processors default ownership.

NOTE: Don’t forget to edit the script for your service processor’s IP address information.

NOTE: It also requires you to have naviseccli command line tools installed on the host which you are running the script from.

NOTE: Rename the txt file to a cmd file.

Download here, VNXTrespassMine.txt

Tagged , , , , ,

Xiotech Magnitude 3D 3000 4000 Controller Fan Replacement

Just recently, there was a pesky failing fan inside of our controllers on a Xiotech Magnitude 3D 4000. A replacement part dispatched and when the controller was taken off line, it was difficult to determine which fan was “Board Fan 1” since there were no markings. In addition, the fan numbering could have started with zero for all we knew. After trial-and-error, this is the correct fan numbering schema.

Xiotech Magnitude 3D 4000 Fan Number Locations

Xiotech’s Controller Fan Replacement Guide: Mag3D_Mod-03_Controller_Fan_Replacement_040110

In addition, this is the manual for the SuperMicro X6DH8/X6DHE-XG2 motherboard inside a Xiotech Controller: SuperMicro-X6DH8-X6DHE-XG2-MNL-0737

Tagged , , , , , ,