With Microsoft’s announcement of the new Azure Stack HCI, there is one major feature that has many IT admins excited: stretched clusters. With Azure Stack HCI stretched clusters you essentially have a solution for disaster recovery with automatic failover. Previously you could combine Storage Replica and failover clusters, both proven technologies, to provide business continuity should a site suffer an outage or failure. But now they can work together to ensure that VMs can spin up and run at the second location with very little down time. To understand how stretched cluster worked and if it really is an “automatic” disaster recovery I tested it in our DataON lab.
Microsoft has a video that shows the concepts behind stretching an Azure Stack HCI cluster between two sites for disaster recovery that you can see on our YouTube channel.
To understand how stretched cluster worked and if it really is an “automatic” disaster recovery I tested it in our DataON lab.
To stage the environment for testing I created sites in DataON’s test Active Directory.
Using DataON S2D-5212 server nodes I racked two servers in rack 1 and two servers in rack 2. These servers were directly connected using one port from each Mellanox RDMA adapter – essentially implementing a DataON Kepler 2-node (K2N-212) solution.
From my previous experience deploying Storage Spaces Direct I had a good understanding of the necessary steps needed to perform prior to building the stretched cluster on the new Azure Stack HCI. The first task was to join the hosts to the domain and install the roles and features required on every server for the cluster. It’s not necessary to install Storage Replica in order to deploy Storage Spaces Direct so this was the first difference I noticed in the new Azure Stack HCI. Storage Replica supports synchronous replication or asynchronous replication and is a crucial component with deploying stretched cluster/stretched volumes.
Windows Server roles and features:
- Data center bridging (for RoCEv2 network adapters)
- Failover clustering
- File server
- FS-Data-Deduplication module
- Hyper-V
- RSAT-AD-PowerShell module
- Storage Replica (only for stretched clusters)
Once the necessary features were installed, I continued to prep the nodes by configuring the networking which consisted of creating virtual switches and virtual adapters, and configuring each set of nodes to their respective subnets to represent the sites they resided in.
Since I created the sites in Active Directory, I was able to see that the nodes were auto-assigned accordingly when I created the cluster. With the cluster of four servers now created, I continued to enable Storage Spaces Direct by running the following PowerShell command:
Enable-ClusterStorageSpacesDirect
There were things I noticed right away that indicated that this was a ‘stretched cluster’.
- Two storage pools, one for each site.
- There were four volumes automatically created: “Cluster Performance History” and “Cluster Performance HistorySRLog” from the storage pool at one site and the same volumes from the storage pool at the second/replication site.
With two nodes in each rack simulating two nodes at two separate locations, the resiliency of each of the storage pools would follow the convention of traditional Storage Spaces Direct, meaning that the two nodes at each site would be limited to two-way mirror or nested resiliency.
With the stretched cluster created and Storage Spaces Direct enabled on the cluster, I needed to create stretched volumes to validate the feature. Stretch volumes are essentially volumes replicated from one fault domain (another word for site) to another. If you are familiar with Storage Replica, the task of creating the “partnership” is done with PowerShell where you have to create a data volume and a log volume at one site, and the same size volume and log volume at the other site. This task gets tedious very quickly especially when you must create multiple volumes you want replicated. With Windows Admin Center, Microsoft allows you to create these volumes with a click of a button through a wizard. No more worrying about ensuring the size of both volumes and log volumes are equal, no more moving “Available Storage Group” from one node to another, and finally no more entering PowerShell cmdlets yourself! There are options to choose which mode you prefer as well as the direction you want the replication to go. I spun up some VMs in the cluster to see what would happen if I shut off both servers at one site (or in a rack in my case). For this specific test I chose to view the sequence from Failover Cluster Manager since Windows Admin Center is a browser-based interface and can tend to have a delay.
Pre-fail state
Using the out of band management of both servers at the “Anaheim” location I executed an ungraceful shutdown. Slowly but surely, I started to see the effects of the two nodes being abruptly turned off: the VMs showed “Unmonitored” in Failover Cluster Manager and I saw the volumes under the “Disks” tab start to change their statuses. After about 2 minutes I could see the status of both Anaheim nodes being offline and all the resources now “assigned to” and “owned” by one of the nodes at the second location (Rancho Cucamonga). Without any intervention from my end, both of my test VMs were up and running at the second location, showing a successful site failover, while both nodes at the Anaheim site were still offline.