Zerto vs array based snapshots with replication

I want to take a moment and discuss a few feature sets from some of the popular storage vendors on the market today and where their replication technology may overlap with Zerto. I have been doing some shopping lately for a storage array. I’m not talking about the big boys at EMC, NetApp, IBM or HP. I’m talking about the guys growing in popularity like Solidfire, Nimble StorageTintri and Tegile. Violin and Pure Storage are all flash arrays, these are a different animal but provide the same type of replication and snaps. When you are looking at these storage array vendors in the enterprise or cloud space (more often than not) you will find overlapping features with some tools you may own. In the Zerto documentation it states that replication and snapshot management requires IT overhead when using the built storage array features. How true is this?  Nimble Storage is the only one on my list that has a plugin for SRM today, but we are going to talk about the built in features to the array and not VMware SRM. Remember, all of these require more than one like array if you want to take advantage of replication. And also, replications and snapshots do not always give you an orchestrated failover and failback architecture. Most of these will include features with the array and others may charge to turn on a feature. Let me first say that all of these storage arrays are fantastic. They are very forward thinking and each have a great place in the enterprise and cloud space.

Tintri logo

Tintri is built on a application aware storage architecture. The array is purpose built for virtualization. Tintri has three main features. Clone, snapshot and protect. The cloning feature is pretty nice because it allows you to clone form an existing, past or present snapshot (Tintri storage snapshot) to another Tintri array or to the same one.

The array can give you a view from the vCenter web client of all the snapshots from Tintri.

Tintri snapshots







You can setup a schedule to “protect” the VM with a snap and make it crash consistent. You can keep this local to the array or remote to another Tintri array.

Tintri protect






If you create the snap to another array you can view the bytes remaining to replicate and the throughput.

Tintri replication throughput

Tintri replication status






All of these features are great, but what are you going to do with that replicated copy once it is on the other side? There is no orchestrated way of bringing it online or doing reverse protection once you have it up.  I’m sure there is a way to work with the Powershell cmdlets to get something working, but that would require many man hours. Zerto does this for you. To replicate VMs from one location to another, two separately license product must be purchased. Tintri Global Center and ReplicateVM. In my use case, I would use something like this to replicate VMs to another datacenter so that I could import them in to vCloud catalogs or work with a production VM offline. The cloning feature would be great for creating VDI sessions as well.


Tegile logo

I had a hard time finding any technical documentation on the Tegile web site. Most of what I found was marketing material. You will find plenty of whitepapers, solution briefs and customer stories. But there is not much on how replication functions. What I did find was a lot of product demos on YouTube. One goes through the demo of dedupe, compression and recovery. On the recovery piece, you can see that it is still a manual process. Nothing automated like Zerto provides. Tegile did partner with Voonami to provide offsite replication with its array.

Tegile does has a file level protocol that is SMB 3.0, which can be used as a Zerto backup target. They have partnered with Veeam to provide the backup solution. Veeam does have a great set of tools.




Solidfire logo

Only SolidFire delivers native snapshot-based backup and restore functionality usable with any object store or device that has an S3 or SWIFT compatible API. SolidFire now offers the SF2405 and the SF4805  with enhancements to SolidFire’s real-time replication offering with a storage replication adapter (SRA) for integration with Site Recovery Manager (SRM).

The real time replication built in to the array is not a full DR solution. Investments must be made into SRM and you must have like arrays at the sources and destination. On the VMware side, this would require a 25 pack of licenses for SRM or vCloud suite licensing on all of your hosts.


Nimble logo


I think Nimble has the most comprehensive tools when it comes to replication. Nimble has a post on Nimble OS 2.0 that does a walk through of how to configure replication. This covers only the array based replication. With the backup solution, Nimble has included a set of tools for backup and recovery. They have also partnered with CommVault Simpana to provide a more comprehensive back and recovery process. You will need to register on the Nimble website to get all the details with a best practice document. What it comes down to is that the CommVault solution is still a backup process. It is not a real time replication product like Zerto. This tool relies on array based snapshots and is not at the kernel level like Zerto’s VRA. The recommended snapshot duration from CommVault is 15 minutes. A lot like vSphere replication out of the box. Although Nimble can handle snapshots up to a minute. The recovery process is still manual.

Nimble also integrates with VMware SRM. This is the DR method listed for the array in a VMware environment. They have also included a webinar demo of the array with SRM.


To sum it up, I think that each one of these storage vendors has great potential to replace some virtualization backup solutions but not replace an orchestrated BC/DR solution like Zerto. When you look at one of these storage vendors, think about what your current backup solution does and how it compares to what the array provides. Zerto does provide an offsite backup solution with the product, but it does not provide dedupe or compression at the source today like Avamar. However, you do get dedup when you use a backup target like Windows 2012 R2 and turn on those features or with a storage array that offers this. The target must be an SMB share. Or you can just use a TNT drive and backup to the cloud. What I do like about the Zerto offsite backup product is that it does the backup against the replicated VM at the target site. This reduces resource overhead at the source site. You would not need dedup at the source since the backup does not happen there.

Think of it this way, if you get fed up with your current storage vendor and you want to move to something else, how would you go about reconfiguring an BC/DR plan? If you just use Zerto you will not have to worry about losing any features because the product is storage agnostic! If you made investments in to SRM you might find yourself locked in to sticking with the same storage vendor. The storage array vendors treat SRM like a car dealer does the 3rd party warranties they sell when you buy a new car, they may not tell you about another vehicle warranty company if it is not a built in feature of the product line they are selling, it is up to you to know there are other options out there. It is in the storage vendors best interest to get you tied in with SRM so that you will either buy more than one or stick with the array vendor down the road to depend on the array replication features. If a storage vendor requires you to pay an additional license for a VM protection feature, it may be in your best interest to just stick with one solution like Zerto to reduce overhead. Some times technology overlap is unavoidable, but look to Zerto as the BC/DR solution for your virtualization environment.

I will be adding more vendors to this list as I have time.

Dell Compellent vSphere Web Client Plugin 2.0 permissions

I was working with the VMware vSphere web client plugin 2.0 for Compellent storage and I came across a small roadblock with the service account permissions. Getting the virtual appliance setup is pretty straight forward. Just get your head wrapped around what the CITV is and how it interacts with the Enterprise Manager server “EM Server” (which talks to your storage controllers). CITV does not talk directly to your Compellent controllers.

Looking at the administration guide on page 3, you configure the vSphere web client plugin in vCenter as the service account you want to use.

Compellent account

The first thing you need to do is configure credentials for the CITV to launch tasks in vCenter. I’m a fan of using specified service accounts for virtual appliances. This is a windows account that needs to be specified. However, in the administrators guide it does not specify what level of permissions in vCenter this account needs. I sent an email over to Jason Boche at Dell and he did confirm that the account needs to have top level “administrator” permissions for now. They are working on changing this in the next release.

Fiber status view in the web client

Now that I’m getting use to the vCenter web client, one status feature in particular caught my eye. The Fiber channel status. In the web client, you actually get the status of the link whether it is up or down. Wouldn’t it fantastic to see more about the link status like congestion or fabric speed? That info may come from 3rd party tools.

Typically you would check the path policy on a datastore to see if one side of the link is down (or both).  This is something you could create an alarm for in vCenter, vCOPS or even 3rd party monitoring tools.

When checking the C# client you can view the HBA, but it does not give a status on the device.

HBA unknown - client

If you take a look in the web client at the same level view, you get a “status” column for the HBA.

HBA unknown - webv2

In my particular case I found the fiber cards were set to the default speed of “auto”. In this fiber fabric, the blade module was 8GB, the brocade directors were 16GB and the storage controllers were 8GB. It is best to set the HBA to the lowest speed in the fabric.

For a Dell M620 blade, this is how you set the fiber card speed. These options may vary slightly based on your version of firmware.

Enter the BIOS of the server (not the “ctrl+q” during the boot process).

enter bios

Go to the device settings.

configure device settings

You will then get a list of devices. Make sure to update both of the fiber channel cards.

device list

Select the port configuration page.

configure device port settings

The last step is to set the port speed for the HBA.

configure device port speed settings

After you have saved these updates, the vCenter web client should show the status as “online”. Different conditions may exist as to why your HBA is in an unknown state. It could be a bad fiber cable, it could be unplugged or  a link configuration mismatch.

VMworld 2014 wrap up

Wow, another VMworld just went by? This year was great. So much is changing. These are the highlights from VMworld this year:

1. VMware introduced (ROBO) remote branch office server licensing. You can compare the two editions here.

2. VMware workspace suite for Horizon and Airwatch.

3. With 5.5 U2 you can now modify version 10 virtual machines with the vSphere client. The update will also include a few feature updates.

4. Bummer, vSphere 6 is still beta. You can sign up for the beta and test it out! Nice stuff like 4 vCPU FT, vMotion changes, the web client has improved, vMotion across vCenters and virtual datacenters (just to name a few things)

5. Virtual Volumes (vVols).

6. VMware integrated OpenStack. (VIO)

7. New network certifications, VCP-NV, VCDX-NV and VCIX-NV. More certifications to add to the list!

8. vCloud Air. There will be a government services platform coming in September. In 2015 there will be an on demand version, but there will be a beta available soon. EMC Viper has a tech preview of object storage.

9. EVO:Rack and EVO:RAIL. 6 vendors so far will be selling these setups.

10. VMware vSphere data protection version 5.8 and vSphere replication. SRM is also at version 5.8.

11. VMware vCAC 6.1 will arrive in September.

12. VSAN version 2.0 is out, but in beta.

13. vRealize operations suite.

There are also a lot more announcements based on development of Horizon DaaS, vGPU from Nvidia and Project Fargo.

This was a great VMworld this year. I had a chance to meet up with some really smart guys. It was a great pleasure to see what all the storage vendors are doing in the market place. It will take some time to digest all of these great announcements coming from VMware.





VMworld 2014 goals

Whether this is your first time or twelfth time at VMworld, it is important to have a goal when going to VMworld. It is not just about the swag and attending the parties. Remember, this is a huge learning opportunity. At the solutions exchange, ask questions, put them on the spot. The specialists are there to answer questions. See if you can get away with pulling the drive on some SAN arrays.

For me this year it is all about DR and storage. The market for host based storage has exploded over the last few years. From PernixData to VSAN. All flash and hybrid arrays have really taken off as well. There are some really awesome storage arrays like Tintri that makes life really simple. There are new players on the market and old dogs with new tricks.

Take things one day at a time. Take notes on technologies you need to follow up on. You may forget them by the end of the day due to the information overload at the conference.

When good vCenters go bad

The idea of virtualizing the vCenter server is not new. I believe it was version 4.x that really started to push the virtual vCenter hard (eat your own dog food approach). 5.x gave us the Linux vCenter virtual appliance. Even with the virtual appliance, there are special considerations to keep in mind when having a virtual vCenter. Although resource requirements have changed since ver 4.x, best practices around creating and placing the virtual vCenter have not really changed. Typically it comes down to understanding your vSwitch configurations when it comes to getting out of a jam with vCenter. In the past some have relied on vCenter server Heartbeat, but that is EoA as of June 2nd 2014.

Mandvis has a couple of good posts on recovering a vCenter during an outage and also special considerations around using hardware ver 10 on your vCenter server.

I would also like to point out a couple of other scenarios to keep in mind when placing your virtual vCenter server on a host and when it comes to recovery during an outage.

Scenario 1:

You have a blade chassis with different fabrics. Fiber, 10 GB and 1GB management. Virtual machines are connected to the 10GB fabric and host management is connect to 1GB fabric. Fiber channel storage is the primary storage for virtual machines, which traverses the fiber fabric. NFS volumes are mounted to house ISO files and templates, which traverses the 1GB network. I had a situation where the network admin used 4 uplinks from each 1GB fabric and properly split them between upstream switches. This would be a proper design (see diagram below). But, instead of bonding the four 1GB cables from each switch, only 1 cable out of 8 was active to the upstream switch. From a blade perspective, all NICs look active. So when we lost the network on the upstream switch, we lost management to the entire enclosure hosting VMware blades.

blade connections

This also effected the vCenter server that had a CD-ROM attached ISO file. The NFS mount was over the 1GB network. This caused the VM to “pause” with a warning message…

Message on vCenter01: Operation on CD-ROM image file
5.5.0-1991310-20140201-update01.iso has failed. Subsequent
operations on this file are also likely to fail unless the image file
connection is corrected. Try disconnecting the image file, then
reconnecting it to the virtual machine’s CD-ROM drive. Select
Continue to continue forwarding the error to the guest operating
system. Select Disconnect to disconnect the image file.

As you can see, the VM would not resume until action was taken on the CD-ROM from the host console. This required knowing which host the vCenter VM lived on. It is still best practice to create a DRS rule to keep the vCenter VM on a known host (sometimes the first host in the cluster is best). We could not acknowledge this prompt from vCenter, because the VM was in a paused state. Once the message was acknowledged from the VM, vCenter came out of its pause state.

Scenario 2:

The host freeze. Not a PSOD, but the hypervisor going into a hung state. I have only seen this happen once. Even from the DCUI you are unable to restart the management services. But virtual machines continue to run. You are unable to log in to the host console to take action on any virtual machines. It is in a “zombie host” state. I’m not sure if host isolation elections even kicked in.

We accepted the only course of action was to pull the power cord on the host server to force a fail over. With doing this, HA should kick in and fail over the virtual machines. But even after powering off the host, the virtual machines stayed registered to the host. Even a manual “unregister” was not accepted while the host was powered off. The host would not release the locks on the VMDK files. We had to remove the host from vCenter and then re-register the virtual machines to new hosts in the cluster. So it may have been a combination of the vCenter DB and the host isolation response. This was the first time I have seen HA not work properly. Even VMware support could not pinpoint the issue.

So what do you do when you are in a scenario like this and vCenter is on the host that is in a hung state and will not release the locks on the VMDK files even after a host is powered off? I would image you would need to do something nasty to the storage volume to release those connections. Or possibly restore vCenter from backups to another host server.

Of course there are other recovery scenarios you have to keep in mind with vCenter… DB becomes full, OS corruptions, miss-configuration by other admins (like deleting the wrong SQL tables),  no DB backups or issues with any of the other components (like SSO) installed on your vCenter server.


VCAP-DCD 5.1 vs 5.5

 — Edit 8/26/2014 –

So I had a chance to sit the VDCD550 exam on 8/24. Unfortunately the exam crashed on me twice with only 30 minutes to go. I decided to continue on with the exam and receive a modified grade in a week, omitting the question that crashed the exam (I cannot say which one). I have now taken the 5.1 and the 5.5 exam. The biggest question that has come up is “is it easier”. Not really. Just because there are not as many questions as the 5.1 does not make it easier.

I cannot go in to detail on the type of questions I had. But I will say, read the exam blue print. Pay attention to section 1.2 “a mixture of drag-and-drop items and design items using an in-exam design tool”. Do not expect any questions that will give you a single radio button and you just move on. The blueprint tells you the style of questions you will have.

I found the design questions were not as hard on the 5.5 like they were on the 5.1 exam. They did a good job on cutting out the fluff and getting to the point. It was easy to read the scenario while you did the design. The important details will pop out at you.

I don’t think I am allow to say what the master design question is. Again, read the blue print. You have 5 “design” questions and one “master design”.

As far as the content goes, the blueprint is what you need to focus on. Just reading about the topics in each section will not help you. You need to install and configure each component like VSAN, VMware storage appliance, Auto deploy, vCenter virtual appliance, vCenter heartbeat, update manager and any other core product the ties in with vSphere. It helps to read the “what’s new” guide and go from there. Anything that can tie into 5.5 is fair game. Also brush up on the ITIL v3 documents listed in the blueprint. Be aware of storage architectures and how each one is different. Again, the blueprint tells you what you need to focus on. Get the theme? Study the blueprint.

I prepared by reading the vSphere design books, doing the PluralSight videos, reading blogs, watching the vBrownbag sessions and making myself flash cards. I’m telling you though, you need to go through the design process yourself. Use your home lab and come up with a scenario to create concept, logical and physical designs. As crazy it may sound, come up with a project to deploy vSphere and oracle on a laptop and go through the design process. Yes, it will not work, but document the process. If you have a good home lab with multiple whitebox hosts and central storage, that will help.

If for some reason I do receive a failing grade, I will be creating a comprehensive study guide made from the exam blue print for VDCD550. I think the exam was a tough but good one.

Who do I think this exam is good for? Anyone who wants to achieve it. I have been doing architecture for the past 5 years at the same company. I have been in IT for the past 18 years. I have had projects from time to time that require some form of project documentation, but nothing as intense as a vendor coming in to deploy a new technology.

Who do I think would have no problem passing this exam? Consultants who work with multiple different customers to deploy solutions. It is they’re bread and butter to come up with a design to win business. Someone who does consulting for a living would have not problems with this exam.

Who do I think would have a tough time passing the exam? Admins who are not involved in the design process. Usually a process is handed over and tasked to the admins to build. Architects who have been at the same place for a long time and know the infrastructure. Is is not a challenge to think about storage or network architecture you work with every day unless the workloads for a project require something else. It would also be tough for admins or architects who do not have VMware as they’re only focus. Someone who must work on Microsoft or other platforms 50% of the time outside of the VMware infrasture may have a hard time. If you have a project to deploy a large SharePoint  environment leading up to the exam, it may be a little tough passing the exam.

But hey, I could be wrong. There could be some super smart guys out there who are helpdesk pursuing CCIE or VCDX. It is up to you to know what you feel confident with. Rise to the challenge and defeat the exam!




– 225 minutes
– 100 questions
– 6 design questions

– 195 Minutes
– 46 questions
– 5 design questions
– 1 Master design question

One big change I also see – there is no mention in blocking you from going back to review flagged questions. This is a big change. Although time management may not allow a whole lot of time to review flagged questions. I am guessing the design questions will still take 15 to 20 minutes a piece. The Master design question needs 30 minutes.

The “Master design question” still remains a mystery.

If you are sitting the exam, you still have the option to cancel your current exam and reschedule for the 5.5 version. I had to problems at all with scheduling the new exam. Just make sure you do it before your cancellation window.

VDCD550 https://mylearn.vmware.com/lcms/web/portals/certification/VCAP_Blueprints/VCAP-DCD-VDCD550-Exam-Blueprint-v3_2.pdf

VDCD551: https://mylearn.vmware.com/lcms/web/portals/certification/VCAP_Blueprints/VCAP5-DCD-Exam-Blueprint-v3_0.pdf


Exam discounts: http://www.vmworld.com/community/conference/us/learn/training

You will still need to request authorization for the exam even if you were approved for 5.1. https://mylearn.vmware.com/


VCAP-DCD VMworld 2014 study group

I want to see how many people would be interested in getting together on 8/23 for a study session on VCAP-DCD. I am not a certified instructor, this will be a group discussion. Or we can cut out a bunch of VMware visio shapes and slap stuff together for a design practice session (pin the vSwitch on the donkey). This is very informal, no sponsors and no budget. If there are just a handful of people, I think whom ever has the best hotel lounge, that’s where we can meet. Let’s make it a couple of hours, 6PM to 8PM. If we get hundreds or thousands of people, I’m going to jump in the river. Or we can try to relocate to a park. I will put the word out on twitter.

I am staying at the Triton Hotel. After looking at pictures of the lounge, maybe 5 or 10 people can fit. Email me if you are interest: james.burd@burdweiser.com

VCP IaaS exam experience

It has been a year or two since I’ve had time to sit down and actually take an exam. I knew with the announcement this year that VCP certifications would need to be renewed before March 2015, I had to do something.

My background leading up to the decision to take the VCP IaaS exam: I have been working with vCloud for the past 7 months. I deployed a single cell in our development environment for the company I am with. It has been great for controlling VM sprawl. I built it as a model for test and production. I kept it simple at first with easy catalogs that joined the development domain, no wild double NAT’s or crazy vApps with multi networks. It is best to get your feet planted firmly in the concepts of vCloud before you take off with that stuff.

My study materials:

– My first exposure to vCloud was last year at VMworld when I won a book from the VMUG group titled “VMware vCloud Architecture Toolkit (vCAT)“. It is a beast of a book. It is not something you want to sit down with and just read from end to end. It is a collection of reference documents. I jumped back and forth with the book to overview material that was important to me. I used it often when designing my architecture.

– I spent time watching the VMware blogs on vCAT.

– I spent plenty of time watching the PluralSight videos from Chris Wahl: VMware vCloud Director 5.1 Essentials “Installing and Configuring” and “Managing and Monitoring“. Also Jake Robinson’s “VMware vCloud Director Organizations” and “VMware vCloud Director Essentials” with David Davis. These are the best training videos! They are worth every bit of money you pay for a subscription to PluralSight.

– I did have a chance to attend the VMware vCloud Director: Install, Configure, Manage course a few months ago. It was really cut and dry material. It was not anything really new for me. It was great to get my hands on the class material though. The book really helped with studying for the exam.

– As a part of the VMware vCloud course, the instructor Shawn Bolan gave us access to Measureup.com practice exams for VCP-Cloud. For $100 you can get two months of access to this practice exam. For me, I was taking the IaaS exam, so it was a little different taking this practice exam based on a different exam (VCP-Cloud).

– I did plenty of the practice exams. 1. The VMware VCP IaaS mock exam. If I missed questions, I would research the answers. 2. Practice exams from Paul McSharry here, here and here. 3. Measureup.com Mock exams. For each mock exam I would actually have my vCAT book and my class material to go over questions and answers. It is not just an exercise to pass a mock exam, but an experiance to learn something new.

– Read plenty of PDF’s. 1. The vShield Installation and Upgrade Guide. 2. The vCloud Director User’s Guide. 3. The vCenter Chargeback Manager Users’s Guide.

The Exam itself:

– I really hate sitting exams. For me it is like sitting there waiting for an electric shock at the end. I had 90 minutes for 85 questions. I think I averaged 1 question every 45 seconds. The questions were not super wordy like a VCAP exam. The mix of questions between vCloud, chargeback, vCloud connector and vShield seemed pretty balanced like it was in the mock exam. You will see every topic from the exam blueprint! I had marked about 10 questions for review. At the end I had about 15 minutes left. I spent a few minutes going over anything that I might have missed. I felt fairly confident in my answers, so I ended the exam. No electric shock at the end, I passed!

You really need hands on with the products. I have to admit, I have not work with chargeback or vCloud connector yet. Those are optional products. I do not have a need for them in my environment. I only reviewed the PDF documents for these. These are probably questions that I missed. If you have a home lab, try to build all of these components out!

You really need time to deploy not just simple networking and vApps, but some of the more complicated items as well. Networking is a big focus in the exam. It is also vital to know when using vCloud. Knowing how to navigate the vCloud administrative options is vital!

Why the VCP-IaaS exam and not VCP-vCloud? Gregg Robertson has a pretty good post on the differences between the two.

Good luck to anyone seeking to take the exam!

Working with XenDesktop 7.x printing policies for external sessions

This post is specifically looking at printer policies to block external users from enabling printers. There is a lack of documentation and a bit of confusion when looking at the policies in XenDesktop.

Lets say you have a simple deployment, internal and external users.


Citrix StoreFront Deployment

Let’s focus on printing for now. You want to allow internal users to map printers and deny external users from mapping home printers. This would be a DLP strategy to keep data from leave the session from a remote location. Looking at the policies, you would think that just allowing internal and deny internal would work.

assign ctx policy 2 - incorrect

Wrong. Printing is actually enabled by default without a policy in place. I could not find this documented anywhere! That should be the first disclaimer on http://support.citrix.com/proddocs/topic/xendesktop-7/ps-console-policies-rules-printer-clients-v2.html.

It took working with support for weeks to find the proper configuration to block printing from external clients. I even worked with the NetScaler teams, thinking the policy had to be tied to the SmartHost name.

The correct configuration is almost a double negative. Set “Client printer redirection – Prohibit” and “Auto-create client printers – Do not create client printers”.

define ctx policy

Set the access controller filter to “Allow with Netscaler” using asterisk (if you have just one NetScaler) as the farm name and access condition. This is allowing the “deny printers” policy to apply to anyone who uses the NetScaler. If you want to use a specific NetScaler, use the SmartHost name for the Farm Name.

assign ctx policy 3

Also Apply a second filter to the Delivery Group with “Allow”.

assign ctx policy 4The final configuration should look like this:

assign ctx policy 5This will block external users from using home printers to printer data from a Citrix session. Users will also need to authenticate against the NetScaler when logging on from an external network. This is best practice, but you do have the option to authenticate directly against storefront (not recommended).