Fiber status view in the web client

Now that I’m getting use to the vCenter web client, one status feature in particular caught my eye. The Fiber channel status. In the web client, you actually get the status of the link whether it is up or down. Wouldn’t it fantastic to see more about the link status like congestion or fabric speed? That info may come from 3rd party tools.

Typically you would check the path policy on a datastore to see if one side of the link is down (or both).  This is something you could create an alarm for in vCenter, vCOPS or even 3rd party monitoring tools.

When checking the C# client you can view the HBA, but it does not give a status on the device.

HBA unknown - client

If you take a look in the web client at the same level view, you get a “status” column for the HBA.

HBA unknown - webv2

In my particular case I found the fiber cards were set to the default speed of “auto”. In this fiber fabric, the blade module was 8GB, the brocade directors were 16GB and the storage controllers were 8GB. It is best to set the HBA to the lowest speed in the fabric.

For a Dell M620 blade, this is how you set the fiber card speed. These options may vary slightly based on your version of firmware.

Enter the BIOS of the server (not the “ctrl+q” during the boot process).

enter bios

Go to the device settings.

configure device settings

You will then get a list of devices. Make sure to update both of the fiber channel cards.

device list

Select the port configuration page.

configure device port settings

The last step is to set the port speed for the HBA.

configure device port speed settings

After you have saved these updates, the vCenter web client should show the status as “online”. Different conditions may exist as to why your HBA is in an unknown state. It could be a bad fiber cable, it could be unplugged or  a link configuration mismatch.

VMworld 2014 wrap up

Wow, another VMworld just went by? This year was great. So much is changing. These are the highlights from VMworld this year:

1. VMware introduced (ROBO) remote branch office server licensing. You can compare the two editions here.

2. VMware workspace suite for Horizon and Airwatch.

3. With 5.5 U2 you can now modify version 10 virtual machines with the vSphere client. The update will also include a few feature updates.

4. Bummer, vSphere 6 is still beta. You can sign up for the beta and test it out! Nice stuff like 4 vCPU FT, vMotion changes, the web client has improved, vMotion across vCenters and virtual datacenters (just to name a few things)

5. Virtual Volumes (vVols).

6. VMware integrated OpenStack. (VIO)

7. New network certifications, VCP-NV, VCDX-NV and VCIX-NV. More certifications to add to the list!

8. vCloud Air. There will be a government services platform coming in September. In 2015 there will be an on demand version, but there will be a beta available soon. EMC Viper has a tech preview of object storage.

9. EVO:Rack and EVO:RAIL. 6 vendors so far will be selling these setups.

10. VMware vSphere data protection version 5.8 and vSphere replication. SRM is also at version 5.8.

11. VMware vCAC 6.1 will arrive in September.

12. VSAN version 2.0 is out, but in beta.

13. vRealize operations suite.

There are also a lot more announcements based on development of Horizon DaaS, vGPU from Nvidia and Project Fargo.

This was a great VMworld this year. I had a chance to meet up with some really smart guys. It was a great pleasure to see what all the storage vendors are doing in the market place. It will take some time to digest all of these great announcements coming from VMware.





VMworld 2014 goals

Whether this is your first time or twelfth time at VMworld, it is important to have a goal when going to VMworld. It is not just about the swag and attending the parties. Remember, this is a huge learning opportunity. At the solutions exchange, ask questions, put them on the spot. The specialists are there to answer questions. See if you can get away with pulling the drive on some SAN arrays.

For me this year it is all about DR and storage. The market for host based storage has exploded over the last few years. From PernixData to VSAN. All flash and hybrid arrays have really taken off as well. There are some really awesome storage arrays like Tintri that makes life really simple. There are new players on the market and old dogs with new tricks.

Take things one day at a time. Take notes on technologies you need to follow up on. You may forget them by the end of the day due to the information overload at the conference.

When good vCenters go bad

The idea of virtualizing the vCenter server is not new. I believe it was version 4.x that really started to push the virtual vCenter hard (eat your own dog food approach). 5.x gave us the Linux vCenter virtual appliance. Even with the virtual appliance, there are special considerations to keep in mind when having a virtual vCenter. Although resource requirements have changed since ver 4.x, best practices around creating and placing the virtual vCenter have not really changed. Typically it comes down to understanding your vSwitch configurations when it comes to getting out of a jam with vCenter. In the past some have relied on vCenter server Heartbeat, but that is EoA as of June 2nd 2014.

Mandvis has a couple of good posts on recovering a vCenter during an outage and also special considerations around using hardware ver 10 on your vCenter server.

I would also like to point out a couple of other scenarios to keep in mind when placing your virtual vCenter server on a host and when it comes to recovery during an outage.

Scenario 1:

You have a blade chassis with different fabrics. Fiber, 10 GB and 1GB management. Virtual machines are connected to the 10GB fabric and host management is connect to 1GB fabric. Fiber channel storage is the primary storage for virtual machines, which traverses the fiber fabric. NFS volumes are mounted to house ISO files and templates, which traverses the 1GB network. I had a situation where the network admin used 4 uplinks from each 1GB fabric and properly split them between upstream switches. This would be a proper design (see diagram below). But, instead of bonding the four 1GB cables from each switch, only 1 cable out of 8 was active to the upstream switch. From a blade perspective, all NICs look active. So when we lost the network on the upstream switch, we lost management to the entire enclosure hosting VMware blades.

blade connections

This also effected the vCenter server that had a CD-ROM attached ISO file. The NFS mount was over the 1GB network. This caused the VM to “pause” with a warning message…

Message on vCenter01: Operation on CD-ROM image file
5.5.0-1991310-20140201-update01.iso has failed. Subsequent
operations on this file are also likely to fail unless the image file
connection is corrected. Try disconnecting the image file, then
reconnecting it to the virtual machine’s CD-ROM drive. Select
Continue to continue forwarding the error to the guest operating
system. Select Disconnect to disconnect the image file.

As you can see, the VM would not resume until action was taken on the CD-ROM from the host console. This required knowing which host the vCenter VM lived on. It is still best practice to create a DRS rule to keep the vCenter VM on a known host (sometimes the first host in the cluster is best). We could not acknowledge this prompt from vCenter, because the VM was in a paused state. Once the message was acknowledged from the VM, vCenter came out of its pause state.

Scenario 2:

The host freeze. Not a PSOD, but the hypervisor going into a hung state. I have only seen this happen once. Even from the DCUI you are unable to restart the management services. But virtual machines continue to run. You are unable to log in to the host console to take action on any virtual machines. It is in a “zombie host” state. I’m not sure if host isolation elections even kicked in.

We accepted the only course of action was to pull the power cord on the host server to force a fail over. With doing this, HA should kick in and fail over the virtual machines. But even after powering off the host, the virtual machines stayed registered to the host. Even a manual “unregister” was not accepted while the host was powered off. The host would not release the locks on the VMDK files. We had to remove the host from vCenter and then re-register the virtual machines to new hosts in the cluster. So it may have been a combination of the vCenter DB and the host isolation response. This was the first time I have seen HA not work properly. Even VMware support could not pinpoint the issue.

So what do you do when you are in a scenario like this and vCenter is on the host that is in a hung state and will not release the locks on the VMDK files even after a host is powered off? I would image you would need to do something nasty to the storage volume to release those connections. Or possibly restore vCenter from backups to another host server.

Of course there are other recovery scenarios you have to keep in mind with vCenter… DB becomes full, OS corruptions, miss-configuration by other admins (like deleting the wrong SQL tables),  no DB backups or issues with any of the other components (like SSO) installed on your vCenter server.


VCAP-DCD 5.1 vs 5.5

 — Edit 8/26/2014 —

So I had a chance to sit the VDCD550 exam on 8/24. Unfortunately the exam crashed on me twice with only 30 minutes to go. I decided to continue on with the exam and receive a modified grade in a week, omitting the question that crashed the exam (I cannot say which one). I have now taken the 5.1 and the 5.5 exam. The biggest question that has come up is “is it easier”. Not really. Just because there are not as many questions as the 5.1 does not make it easier.

I cannot go in to detail on the type of questions I had. But I will say, read the exam blue print. Pay attention to section 1.2 “a mixture of drag-and-drop items and design items using an in-exam design tool”. Do not expect any questions that will give you a single radio button and you just move on. The blueprint tells you the style of questions you will have.

I found the design questions were not as hard on the 5.5 like they were on the 5.1 exam. They did a good job on cutting out the fluff and getting to the point. It was easy to read the scenario while you did the design. The important details will pop out at you.

I don’t think I am allow to say what the master design question is. Again, read the blue print. You have 5 “design” questions and one “master design”.

As far as the content goes, the blueprint is what you need to focus on. Just reading about the topics in each section will not help you. You need to install and configure each component like VSAN, VMware storage appliance, Auto deploy, vCenter virtual appliance, vCenter heartbeat, update manager and any other core product the ties in with vSphere. It helps to read the “what’s new” guide and go from there. Anything that can tie into 5.5 is fair game. Also brush up on the ITIL v3 documents listed in the blueprint. Be aware of storage architectures and how each one is different. Again, the blueprint tells you what you need to focus on. Get the theme? Study the blueprint.

I prepared by reading the vSphere design books, doing the PluralSight videos, reading blogs, watching the vBrownbag sessions and making myself flash cards. I’m telling you though, you need to go through the design process yourself. Use your home lab and come up with a scenario to create concept, logical and physical designs. As crazy it may sound, come up with a project to deploy vSphere and oracle on a laptop and go through the design process. Yes, it will not work, but document the process. If you have a good home lab with multiple whitebox hosts and central storage, that will help.

If for some reason I do receive a failing grade, I will be creating a comprehensive study guide made from the exam blue print for VDCD550. I think the exam was a tough but good one.

Who do I think this exam is good for? Anyone who wants to achieve it. I have been doing architecture for the past 5 years at the same company. I have been in IT for the past 18 years. I have had projects from time to time that require some form of project documentation, but nothing as intense as a vendor coming in to deploy a new technology.

Who do I think would have no problem passing this exam? Consultants who work with multiple different customers to deploy solutions. It is they’re bread and butter to come up with a design to win business. Someone who does consulting for a living would have not problems with this exam.

Who do I think would have a tough time passing the exam? Admins who are not involved in the design process. Usually a process is handed over and tasked to the admins to build. Architects who have been at the same place for a long time and know the infrastructure. Is is not a challenge to think about storage or network architecture you work with every day unless the workloads for a project require something else. It would also be tough for admins or architects who do not have VMware as they’re only focus. Someone who must work on Microsoft or other platforms 50% of the time outside of the VMware infrasture may have a hard time. If you have a project to deploy a large SharePoint  environment leading up to the exam, it may be a little tough passing the exam.

But hey, I could be wrong. There could be some super smart guys out there who are helpdesk pursuing CCIE or VCDX. It is up to you to know what you feel confident with. Rise to the challenge and defeat the exam!




– 225 minutes
– 100 questions
– 6 design questions

– 195 Minutes
– 46 questions
– 5 design questions
– 1 Master design question

One big change I also see – there is no mention in blocking you from going back to review flagged questions. This is a big change. Although time management may not allow a whole lot of time to review flagged questions. I am guessing the design questions will still take 15 to 20 minutes a piece. The Master design question needs 30 minutes.

The “Master design question” still remains a mystery.

If you are sitting the exam, you still have the option to cancel your current exam and reschedule for the 5.5 version. I had to problems at all with scheduling the new exam. Just make sure you do it before your cancellation window.




Exam discounts:

You will still need to request authorization for the exam even if you were approved for 5.1.


VCAP-DCD VMworld 2014 study group

I want to see how many people would be interested in getting together on 8/23 for a study session on VCAP-DCD. I am not a certified instructor, this will be a group discussion. Or we can cut out a bunch of VMware visio shapes and slap stuff together for a design practice session (pin the vSwitch on the donkey). This is very informal, no sponsors and no budget. If there are just a handful of people, I think whom ever has the best hotel lounge, that’s where we can meet. Let’s make it a couple of hours, 6PM to 8PM. If we get hundreds or thousands of people, I’m going to jump in the river. Or we can try to relocate to a park. I will put the word out on twitter.

I am staying at the Triton Hotel. After looking at pictures of the lounge, maybe 5 or 10 people can fit. Email me if you are interest:

VCP IaaS exam experience

It has been a year or two since I’ve had time to sit down and actually take an exam. I knew with the announcement this year that VCP certifications would need to be renewed before March 2015, I had to do something.

My background leading up to the decision to take the VCP IaaS exam: I have been working with vCloud for the past 7 months. I deployed a single cell in our development environment for the company I am with. It has been great for controlling VM sprawl. I built it as a model for test and production. I kept it simple at first with easy catalogs that joined the development domain, no wild double NAT’s or crazy vApps with multi networks. It is best to get your feet planted firmly in the concepts of vCloud before you take off with that stuff.

My study materials:

– My first exposure to vCloud was last year at VMworld when I won a book from the VMUG group titled “VMware vCloud Architecture Toolkit (vCAT)“. It is a beast of a book. It is not something you want to sit down with and just read from end to end. It is a collection of reference documents. I jumped back and forth with the book to overview material that was important to me. I used it often when designing my architecture.

– I spent time watching the VMware blogs on vCAT.

– I spent plenty of time watching the PluralSight videos from Chris Wahl: VMware vCloud Director 5.1 Essentials “Installing and Configuring” and “Managing and Monitoring“. Also Jake Robinson’s “VMware vCloud Director Organizations” and “VMware vCloud Director Essentials” with David Davis. These are the best training videos! They are worth every bit of money you pay for a subscription to PluralSight.

– I did have a chance to attend the VMware vCloud Director: Install, Configure, Manage course a few months ago. It was really cut and dry material. It was not anything really new for me. It was great to get my hands on the class material though. The book really helped with studying for the exam.

– As a part of the VMware vCloud course, the instructor Shawn Bolan gave us access to practice exams for VCP-Cloud. For $100 you can get two months of access to this practice exam. For me, I was taking the IaaS exam, so it was a little different taking this practice exam based on a different exam (VCP-Cloud).

– I did plenty of the practice exams. 1. The VMware VCP IaaS mock exam. If I missed questions, I would research the answers. 2. Practice exams from Paul McSharry here, here and here. 3. Mock exams. For each mock exam I would actually have my vCAT book and my class material to go over questions and answers. It is not just an exercise to pass a mock exam, but an experiance to learn something new.

– Read plenty of PDF’s. 1. The vShield Installation and Upgrade Guide. 2. The vCloud Director User’s Guide. 3. The vCenter Chargeback Manager Users’s Guide.

The Exam itself:

– I really hate sitting exams. For me it is like sitting there waiting for an electric shock at the end. I had 90 minutes for 85 questions. I think I averaged 1 question every 45 seconds. The questions were not super wordy like a VCAP exam. The mix of questions between vCloud, chargeback, vCloud connector and vShield seemed pretty balanced like it was in the mock exam. You will see every topic from the exam blueprint! I had marked about 10 questions for review. At the end I had about 15 minutes left. I spent a few minutes going over anything that I might have missed. I felt fairly confident in my answers, so I ended the exam. No electric shock at the end, I passed!

You really need hands on with the products. I have to admit, I have not work with chargeback or vCloud connector yet. Those are optional products. I do not have a need for them in my environment. I only reviewed the PDF documents for these. These are probably questions that I missed. If you have a home lab, try to build all of these components out!

You really need time to deploy not just simple networking and vApps, but some of the more complicated items as well. Networking is a big focus in the exam. It is also vital to know when using vCloud. Knowing how to navigate the vCloud administrative options is vital!

Why the VCP-IaaS exam and not VCP-vCloud? Gregg Robertson has a pretty good post on the differences between the two.

Good luck to anyone seeking to take the exam!

Working with XenDesktop 7.x printing policies for external sessions

This post is specifically looking at printer policies to block external users from enabling printers. There is a lack of documentation and a bit of confusion when looking at the policies in XenDesktop.

Lets say you have a simple deployment, internal and external users.


Citrix StoreFront Deployment

Let’s focus on printing for now. You want to allow internal users to map printers and deny external users from mapping home printers. This would be a DLP strategy to keep data from leave the session from a remote location. Looking at the policies, you would think that just allowing internal and deny internal would work.

assign ctx policy 2 - incorrect

Wrong. Printing is actually enabled by default without a policy in place. I could not find this documented anywhere! That should be the first disclaimer on

It took working with support for weeks to find the proper configuration to block printing from external clients. I even worked with the NetScaler teams, thinking the policy had to be tied to the SmartHost name.

The correct configuration is almost a double negative. Set “Client printer redirection – Prohibit” and “Auto-create client printers – Do not create client printers”.

define ctx policy

Set the access controller filter to “Allow with Netscaler” using asterisk (if you have just one NetScaler) as the farm name and access condition. This is allowing the “deny printers” policy to apply to anyone who uses the NetScaler. If you want to use a specific NetScaler, use the SmartHost name for the Farm Name.

assign ctx policy 3

Also Apply a second filter to the Delivery Group with “Allow”.

assign ctx policy 4The final configuration should look like this:

assign ctx policy 5This will block external users from using home printers to printer data from a Citrix session. Users will also need to authenticate against the NetScaler when logging on from an external network. This is best practice, but you do have the option to authenticate directly against storefront (not recommended).



VMworld 2014 Alumni

Don’t forget to sign up for the VMworld Alumni program this year as you sign up for VMworld 2014. If you have attended two or more VMworld conferences as a full paid attendee, you qualify!

When I first saw Alumni, I thought “Yes I have attended previous VMworld events”. But it is a new program this year. This alumni protal launched on 8/25/2013. You will get 25 points for signing up and 100 points for every VMworld you attended. I think these points are awarded after you attend the event. No word if you can receive 100 points for previous VMworlds attended. The link to transfer previous enrollments is currently a place holder page.

Learn more here:

The alumni program allows you to earn points for merchandise rewards, various activities and special offers. I have not seen anything specific. Earn CloudCred for registering for the Alumni Portal while on site at VMworld, and earn extra CloudCred points for referring a friend! Alumni members have enjoyed a $200 discount on VMworld registration for the past several years.

Stop by the VMworld Alumni lounge located at Jillians at the Metreon. Pick up your free gift, use the free wifi, enjoy snacks and beverages and relax playing a game of pool.


PernixData and Dell – first test results

Before I go to deep into the layout of this benchmark, let me say that Frank Denneman came out with some great articles on testing SSDs. I highly recommend reading some of his posts to understand how to benchmark hardware and understand the results. To see what PernixData is all about, check out this post from Jason Nash.

I’ve been testing PernixData with Dell hardware this week, trying to find the ceiling on local SSD drives first. My plan is to test what I can place closet to the hypervisor as possible (within a blade). That’s right, I’m testing a Dell M620 blade solution with Dell Compellent storage on FC. It is more common to find add-on PCIe cards for rack mount servers when using SSD solutions, but I am looking to find what kind of performance I can get out of a blade system with SSD drives on a PERC controller. I will be testing SLC SSD drives (Toshiba MK4001GRZB) that are controlled from the local PERC H710 controller.  PernixData has a great set of documents for configuring disk controllers. I am not using the H710P controller (which has a FastPath for IO to bypass the controller cache and get committed directly to the physical disk from host RAM through the second controllers dual-core ROC processor). It sounds a little like EMC ExtremeIO, but on a much smaller scale. CTIO and FastPath provide enhanced performance benefits to SSD volumes. It is important to remember that if you are working with multiple drives on a RAID controller and JBOD is not an option, you need to configure individual disks in RAID 0, not grouped in RAID 0 (although this can be done to take advantage of the performance of both drives at once).

The tests I ran involved running 5 VM’s with IOMeter, 4K and 100% reads on a 30GB file. The queue depth is the default VMware 64. Of course all work loads are different. Not all applications are built the same. If you are looking to test something like SQL, I recommend using BenachMark factory from Quest (Dell). You can record a production workload and play it back on the test platform to see how well something like this would work in your environment. The purpose of the test is to find out how many IOPs I can get out of the solution. I would not recommend relying on something like IOMeter to benchmark something for production.

Make sure your VM guest has a separate paravirtual SCSI controller for the data drive you are testing. Also, make sure everything in the environment along the storage fabric is tuned for best performance. From the server BIOS, PERC controller, HBA cards, fiber switches, fiber interconnects and storage controllers.

Compellent Disk configurations in VMware

 My first test was with Write Back. These test results had better results of course, but only by 10K or so IOPS. I saw as high as 150K IOPS for the FVP cluster, but it usually stayed around 120K IOPS.

PD Cluster level performance 01 post 1-5 upgrade (Write Back)

PD Cluster level performance 02 post 1-5 upgrade (Write Back)

 My second test results was with write through, which is my preferred model since the data is written to the datastore at the same time. You can see that IOPS came in just under 120K IOPS. Still not bad! The dip in this chart is from me starting up another VM with the same test.

PD Cluster level performance 01 post 1-5 upgrade (Write thru)

PD Cluster level performance 02 post 1-5 upgrade (Write thru)


You can see what goes on with my Compellent storage on the back end with the same results:

PD Compellent volume last day perf (Write Back)PD Compellent SSD last day perf (Write Back)PD Compellent 15K last day perf (Write Back)PD Compellent 7K last day perf (Write Back)

All I can say is Holy Cow! SSD’s sure do give great performance when they are closer to the server! I do start to wonder what this does to the life cycle of the drives if they run at a constant rate like this. But like I said, every workload is different. I saw as high as 60K IOPS per SSD in the Dell M620 blades. Would I say this first hardware test is an enterprise solution? Perhaps, it is defiantly cost effective! It depends on your level of comfort with the hardware and your use case.

Working with the Pernix Data software is so easy! It is very simple to install and manage. It is also a breeze to remove when you are done with a POC. If you are working with iSCSI, you will need to adjust your path selection policies after it is removed. You can also use the software without any SSDs, to see what type of performance you are getting from your datastore. PernixData FVP works with block storage protocols today (FC, iSCSI, and FCoE), and will soon support NFS. FVP uses server-side flash (SSDs or PCIe cards) to increase storage performance in vSphere environments.


My next tests? I think this will involve using the Dell M620 Blade with PCIe to see what results I can get from that using PernixData. Dell is really on me to use FluidCache, but that is something down the road I will get to.