How do AKS and AKS on Azure Stack HCI compare?

 

AKS on Azure and has been generally available for years. AKS on Azure Stack HCI is an on-premises implementation of AKS, it runs in customer environments on customer managed hardware. Here’s a list of general similarities and differences between AKS-HCI and AKS.

Features in preview are marked by (*) 

Feature Set AKS on Azure Stack HCI & AKS on Windows Server AKS
Kubernetes Management Cluster/AKS host AKS on Azure Stack HCI and Windows Server is a Cluster API based hosted Kubernetes offering. A management Kubernetes cluster is used to manage Kubernetes workload clusters. The management Kubernetes cluster runs in customer datacenters and is managed by the infrastructure administrator. AKS is a managed Kubernetes offering. AKS control plane is hosted and managed by Microsoft. AKS worker nodes are created in customer subscriptions.
     
Kubernetes Target Cluster (lifecycle operations)
Cloud Native Computing Foundation (CNCF) certification Yes Yes
Who manages the cluster? Managed by you Managed by you
Where is the cluster located? In your datacenter alongside your AKS hybrid management cluster.

Azure Stack HCI 21H2

Windows Server 2019 Datacenter

Windows Server 2022 Datacenter

Windows 10/11 IoT Enterprise*
Windows 10/11 Enterprise*
Windows 10/11 Pro*

Azure cloud
K8s cluster lifecycle management tools (create, scale, update and delete clusters) PowerShell (PS)

Windows Admin Center (WAC)

Az CLI*

Azure Portal*

ARM templates*

Az CLI

Az PowerShell

Azure Portal

Bicep

ARM templates

Can you use kubectl and other open-source Kubernetes tools? Yes Yes
Workload cluster updates K8s version upgrade through PowerShell or WAC. Initiated by you.

Node OS image update initiated by you;

Updates in a target cluster happen at the cluster level – control plane nodes + node pools updated.

Azure CLI, Azure PS, Portal, ARM templates, GitHub Actions;

OS image patch upgrade;

Automatic upgrades;

Planned maintenance windows;

Kubernetes versions Continuous updates to supported Kubernetes versions. For latest version support, visit AKS hybrid releases on GitHub. Continuous updates to supported Kubernetes versions. For latest version support, run az aks get-versions.
Can you start/stop K8s clusters to save costs? Yes, by stopping the underlying failover cluster Yes
Azure Fleet Manager integration Not yet. Yes*
Terraform support Not yet. Yes
     
Node Pools
Do you support running Linux and Windows node pools in the same cluster? Yes!

Linux nodes: CBL-Mariner

Windows nodes:

Windows Server 2019 Datacenter, Windows Server 2022 Datacenter

Yes.

Linux nodes: Ubuntu 18.04, CBL-Mariner

Windows nodes:

Windows Server 2019 Datacenter

Windows Server 2022 Datacenter

What’s your container runtime? Linux nodes: containerd

Windows nodes: containerd

Linux nodes: containerd

Windows nodes: containerd

Can you scale node pools? Manually

Cluster autoscaler

Vertical pod autoscalar

Manually

Cluster autoscaler

Vertical pod autoscalar

Horizontal pod autoscalar Yes Yes
What about virtual nodes?    
Azure container instance No Yes
Can you upgrade a node pool? We do not support upgrading individual node pools. All upgrades happen at the K8s cluster level. You can perform node pool specific upgrades in an AKS cluster.
GPU enabled node pools Yes* Yes
Azure Container Registry Yes Yes
KEDA support Not yet Yes*
     
Networking
Who creates and manages the networks? All networks (for both the management cluster and target K8s clusters) are created and managed by you By default, Azure creates the virtual network and subnet for you. You can also choose an existing virtual network to create your AKS clusters
What type of network options are supported? DHCP networks with/without VLAN ID

Static IP networks with/without VLAN ID

SDN support for AKS on Azure Stack HCI

Bring your own Azure virtual network for AKS clusters.
Load balancers HAProxy (default) runs in a separate VM in the target K8s cluster

kubeVIP – runs as a K8s service in the control plane K8s node

Bring your own load balancer

Load balancers are always given sIP addresses from a customer vip pool to ensure application and K8s cluster availability.

You can create multiple instances of a LB (active-passive) for high availability

Azure load balancer – Basic SKU or Standard SKU

Can also use internal load balancer

By default, load balancer IP address is tied to load balancer ARM resource. You can also assign a static public IP address directly to your Kubernetes service

CNI/Network plugin Calico (default)

Note: Network policies are covered in the Security and Authentication section.

Azure CNI

Calico

Azure CNI Overlay

Bring your own CNI

Note: Network policies are covered in the Security and Authentication section.

Ingress controllers No but you can use 3rd party addons – Nginx. 3rd party addons are not supported by Microsoft’s support policy.

Support for Nginx with web app routing addon.

Egress controls Egress is controlled by Network policies, by default all outbound traffic from pods is blocked. You can deploy additional egress controls and policies. You can use Azure Policy and NSGs to control network flow or use Calico policies. You can also use Azure FW and Azure Security Groups.
Egress types Egress types and options depend on your network architecture. Azure load balancer, managed NAT gateway and user defined routes are the supported egress types.
Customize CoreDNS Allowed Allowed
Service Mesh Yes, Open Service Mesh (OSM) through Azure Arc enabled Kubernetes.

3rd party addons – Istio, etc. 3rd party addons are not supported by Microsoft’s support policy.

Open Service Mesh

Marketplace offering available for Istio

     
Storage
Where is the storage provisioned? On-premises Azure Storage.

Azure Files and Azure Disk premium CSI drivers deployed by default. You can also deploy any custom storage class.

What types of persistent volumes are supported? Read Write Once

Read Write Many

Read Write Once

Read Write Many

Do the storage drivers support Container Storage Interface (CSI)? Yes Yes
Is dynamic provisioning supported? Yes Yes
Is volume resizing supported? Yes Yes
Are volume snapshots supported? No Yes
     
Security and Authentication
How do you access your Kubernetes cluster? Certificate based kubeconfig (default)

AD based kubeconfig

Azure AD and Kubernetes RBAC

Azure AD and Azure RBAC*

Certificate based kubeconfig (default)

Azure AD and Kubernetes RBAC

Azure AD and Azure RBAC

Network Policies Yes, we support Calico network policies Yes, we support Calico and Azure CNI network policies
Limit source networks that can access API server Yes, by using VIP pools. Yes, by using the “-api-server-authorized-ip-ranges” parameter and private clusters.
Certificate rotation and secrets encryption Yes Yes
Support for private cluster Not supported yet Yes! You can create private AKS clusters
Secrets store CSI driver Yes Yes
Support for disk encryption Yes, via bitlocker Disks are encrypted on the storage side with platform managed keys and with support for customer provided keys.

Hosts and locally attached disks can also be encrypted with encryption at host.

gMSA v2 support for Windows containers Yes Yes
Azure Policy Yes, through Azure Arc enabled K8s Yes
Azure Defender Yes, through Azure Arc enabled K8s* Yes
     
Monitoring and Logging
Collect logs Yes, through PS and WAC. All logs – management cluster, control plane nodes, target K8s clusters are collected. Yes, through Azure Portal, Az CLI, etc
Support for Azure Monitor Yes, through Azure Arc enabled K8s. Yes
3rd party addons for monitoring and logging   AKS works with Azure managed Prometheus* and Azure managed Grafana*
Subscribe to Azure Event Grid Events Yes, via Azure Arc enabled Kubernetes* Yes
     
Develop and run applications
Azure App service Yes, through Azure Arc enabled K8s* Yes
Azure Functions Yes, through Azure Arc enabled K8s* Yes
Azure Logic Apps Yes, through Azure Arc enabled K8s* You can directly create App Service, Functions, Logic Apps on Azure instead of creating on AKS
Develop applications using Helm Yes Yes
Develop applications using Dapr Yes, through Azure Arc enabled K8s* Yes
DevOps Azure DevOps via Azure Arc enabled K8s.

GitHub Actions via Azure Arc enabled K8s.

GitOps Flux v2 via Azure Arc enabled K8s.

3rd party addon: ArgoCD. 3rd party addons are not supported by Microsoft’s support policy.

GitOps Flux v2 through Azure Arc enabled Kubernetes is free for AKS-HCI customers.

Azure DevOps

GitHub Actions

GitOps Flux v2

     
Product Pricing
Product pricing If you have Azure Hybrid Benefit, you can use AKS-HCI at no additional cost.

If you do not have Azure Hybrid Benefit pricing based on number of workload cluster vCPUs. Management cluster, control plane nodes, load balancers are free.

Unlimited free clusters, pay for on-demand compute of the worker nodes.

Paid tier available with uptime SLA, support for 5k nodes.

Azure Support AKS-HCI is supported out of the Windows Server support organization aligned with Arc for Kubernetes and Azure Stack HCI. You can open support requests through the Azure portal and other support channels like Premier Support. AKS in Azure is supported through enterprise class support in the Azure team. You can open support requests in the Azure portal.
SLA We do not offer SLAs since AKS-HCI runs in your environment. Paid uptime SLA clusters for production with fixed cost on the API + worker node compute, storage and networking costs.

This article appears courtesy of Microsoft.