Scenario: Provision a New VM from Scratch
Why This Scenario Matters
Provisioning a new VM is the single most frequent infrastructure operation in any enterprise. At 5,000+ VMs, the current VMware environment processes provisioning requests daily -- from dev/test environments spun up by CI pipelines to production database servers requested through ITSM tickets. The provisioning workflow touches every layer of the stack: the API entry point, the scheduler, storage allocation, network plumbing, and hypervisor boot sequence. Understanding this workflow end-to-end on each candidate platform reveals where automation is mature, where manual steps still exist, where latency hides, and where failure modes lurk.
This scenario traces a single, concrete request: "Provision a RHEL 9 VM with 4 vCPUs, 16 GiB RAM, a 100 GiB boot disk cloned from a golden image template, attached to VLAN 200 (application tier), with cloud-init injecting hostname, SSH keys, and network configuration." The same request is executed on all three platforms so the evaluation team can compare the operational experience.
End-to-End Flow: VMware vSphere (Baseline)
This is what the team does today. It is the baseline against which every replacement is measured.
Step-by-Step
VMware vSphere: VM Provisioning Flow
======================================
Request vCenter ESXi Host vSAN / SAN
(Terraform/ Server Storage
vRA/Manual)
| | | |
| 1. API call | | |
| (SOAP/REST) | | |
+---------------------> | | |
| | | |
| 2. Validate: | | |
| - RBAC perms | | |
| - Resource pool | | |
| quota check | | |
| - Template exists | | |
| ~200 ms | | |
| | | |
| 3. DRS placement: | | |
| - Score each host | | |
| - CPU/RAM fit | | |
| - Affinity rules | | |
| - Datastore space | | |
| ~500 ms | | |
| | | |
| | 4. Clone template | |
| +----------------------->| |
| | - Linked clone: | 5. Storage alloc |
| | metadata only |--------------------->|
| | ~2 sec | - vSAN: CLOM policy |
| | - Full clone: | evaluation |
| | copies all blocks | - Allocate objects |
| | ~30-120 sec | - Apply FTT policy |
| | (100 GiB) | ~1-3 sec (linked) |
| | | ~30-120 sec (full) |
| | | |
| | 6. Configure VM: | |
| | - vCPU, RAM | |
| | - vNIC -> port group| |
| | (VLAN 200) | |
| | - CD-ROM (cloud-init| |
| | ISO) | |
| | ~500 ms | |
| | | |
| | 7. Power on | |
| +----------------------->| |
| | - VMkernel creates | |
| | vCPU world threads| |
| | - Memory allocated | |
| | - vNIC connected | |
| | to vDS port | |
| | ~2-3 sec | |
| | | |
| | 8. VM boots: |
| | BIOS/UEFI -> GRUB -> |
| | kernel -> cloud-init |
| | ~30-60 sec |
| | | |
| 9. VM ready | | |
|<----------------------+ | |
| Total (linked): ~40-70 sec | |
| Total (full): ~90-180 sec | || Step | Component | Action | Duration |
|---|---|---|---|
| 1 | vCenter API | Receive REST/SOAP request, parse spec | ~100 ms |
| 2 | vCenter | RBAC, quota, template validation | ~200 ms |
| 3 | DRS | Host placement scoring | ~500 ms |
| 4 | ESXi | Clone template VMDK | 2 sec (linked) / 30-120 sec (full) |
| 5 | vSAN/SAN | Allocate storage, apply policy | 1-3 sec (linked) / included in clone time (full) |
| 6 | vCenter | Configure vHW, attach network | ~500 ms |
| 7 | ESXi | Power on, create VM world | ~2-3 sec |
| 8 | Guest OS | BIOS -> kernel -> cloud-init | ~30-60 sec |
| Total | ~40-70 sec (linked) / ~90-180 sec (full) |
Storage Detail
- Linked clone: vCenter creates a delta disk (redo log) that references the template VMDK as a parent. No data is copied. Writes go to the delta disk; reads fall through to the parent. Instant provisioning, but the template becomes a shared dependency -- deleting it orphans all linked clones.
- Full clone: vSAN copies every block of the template VMDK to a new VMDK. Duration depends on disk size and vSAN tier (NVMe vs SSD). For a 100 GiB disk on NVMe vSAN at ~1 GB/s throughput, expect ~100 seconds.
- vSAN policy: CLOM evaluates the Storage Policy Based Management (SPBM) profile attached to the VM. If the policy requires FTT=1 with RAID-1 mirroring, vSAN allocates two copies of every component (doubling the physical space consumed). CLOM confirms sufficient capacity on at least two fault domains before allowing the clone.
Networking Detail
- Port group assignment: The VM's vNIC is connected to a dvPortGroup configured for VLAN 200. The vDS trunk port on the ESXi host's physical uplink carries VLAN 200 tagged traffic.
- IP assignment: cloud-init reads network config from the seed ISO (NoCloud datasource) or from DHCP on VLAN 200. Static IP: cloud-init configures
/etc/sysconfig/network-scripts/or Netplan. DHCP: the guest's DHCP client requests a lease on VLAN 200. - NSX (if present): If NSX-T is deployed, the VM's port is registered in the NSX management plane. The Distributed Firewall applies the security policy assigned to the VM's NSX segment. The N-VDS handles overlay encapsulation if the VM is on an overlay segment rather than a VLAN-backed segment.
End-to-End Flow: OpenShift Virtualization Engine (OVE)
Request Entry Point
On OVE, a VM provisioning request can arrive through three paths:
- GitOps (ArgoCD): A
VirtualMachineYAML manifest is committed to a Git repository. ArgoCD detects the change and applies the manifest to the cluster. This is the recommended production workflow for auditability and drift detection. - Terraform/OpenTofu: The
kubevirtTerraform provider creates theVirtualMachineCR via the Kubernetes API. Used for integration with existing IaC pipelines. - OpenShift Console: The web UI provides a guided form for VM creation with template selection. Used for ad-hoc requests and troubleshooting.
All three paths converge on the same point: a VirtualMachine custom resource is created in the Kubernetes API server.
Step-by-Step
OVE: VM Provisioning Flow
===========================
Request kube-apiserver virt-controller kube-scheduler
(GitOps/ + etcd (KubeVirt)
Terraform/
Console)
| | | |
| 1. Create VM CR | | |
| kubectl apply -f | | |
+---------------------> | | |
| | | |
| 2. Admission: | | |
| - virt-api webhook| | |
| validates spec | | |
| - RBAC check | | |
| - ResourceQuota | | |
| - LimitRange | | |
| ~300 ms | | |
| | | |
| | 3. virt-controller | |
| | watches VM CR | |
| +----------------------->| |
| | Creates: | |
| | - DataVolume (CDI) | |
| | - VMI CR | |
| | - virt-launcher Pod | |
| | ~500 ms | |
| | | |
| | | 4. kube-scheduler |
| | +--------------------->|
| | | Scores nodes: |
| | | - CPU/RAM requests |
| | | - nodeSelector |
| | | - affinity rules |
| | | - topology spread |
| | | - device plugins |
| | | (kvm, vhost-net) |
| | | ~200 ms |
| | | |
CDI Controller CSI Driver Ceph/ODF OVN-Kubernetes
(storage import) (rbd.csi.ceph.com) Cluster CNI
| | | |
| 5. DataVolume | | |
| processing: | | |
| - Creates PVC | | |
| - Spawns importer | 6. CSI CreateVolume | |
| pod or cloner +----------------------->| |
| pod | - rbd create | 7. Ceph RBD: |
| ~1 sec | ~500 ms | - PG allocation |
| | | - Replicate |
| If clone from | | (3x or EC) |
| golden image PVC: | | ~1-2 sec |
| - CSI clone (fast) | | |
| - Ceph rbd clone | | |
| (COW, instant) | | |
| ~2-3 sec total | | |
| | | |
| If import from URL: | | |
| - CDI importer pod | | |
| downloads QCOW2 | | |
| - Converts to raw | | |
| - Writes to PVC | | |
| ~60-300 sec | | |
| (depends on image | | |
| size and network) | | |
| | | |
kubelet (target node) virt-handler virt-launcher Pod OVN / Multus
(DaemonSet) (per-VM)
| | | |
| 8. kubelet starts | | |
| virt-launcher pod | | |
| - CRI-O pulls | | |
| container image | | |
| (if not cached) | | |
| - Creates pod | | |
| sandbox | | |
| ~2-5 sec | | |
| | | |
| | | 9. CNI execution: |
| | +--------------------->|
| | | - OVN-K8s: creates |
| | | logical port on |
| | | node's logical |
| | | switch |
| | | - Assigns Pod IP |
| | | from Pod CIDR |
| | | - Programs OVS |
| | | flows |
| | | ~500 ms |
| | | |
| | | 10. Multus (VLAN): |
| | | - Reads NAD for |
| | | VLAN 200 |
| | | - cnv-bridge plugin |
| | | creates bridge + |
| | | VLAN subinterface |
| | | - Attaches to pod |
| | | network namespace |
| | | ~300 ms |
| | | |
| | 11. virt-handler | |
| | detects new VMI | |
| +----------------------->| |
| | Configures: | |
| | - libvirt domain | |
| | XML | |
| | - Disk attachments | |
| | (RBD block | |
| | device) | |
| | - NIC passthrough | |
| | to QEMU | |
| | ~1 sec | |
| | | |
| | 12. virt-launcher: |
| | - Starts libvirtd |
| | - libvirtd starts QEMU |
| | - QEMU boots VM: |
| | UEFI -> GRUB -> |
| | kernel -> cloud-init |
| | ~30-60 sec |
| | | |
| 13. VM ready | | |
| VMI status: | | |
| phase: Running | | |
| Total: ~40-75 sec (clone) / ~120-360 sec (import) |Timing Summary (OVE)
| Step | Component | Action | Duration |
|---|---|---|---|
| 1 | kube-apiserver | Receive and persist VM CR | ~100 ms |
| 2 | virt-api webhook | Admission validation, RBAC, quota | ~300 ms |
| 3 | virt-controller | Create DataVolume, VMI, virt-launcher Pod | ~500 ms |
| 4 | kube-scheduler | Score nodes, bind Pod | ~200 ms |
| 5 | CDI controller | Create PVC, spawn importer/cloner | ~1 sec |
| 6-7 | CSI + Ceph | Create RBD volume, replicate | ~1-2 sec (clone) / ~60-300 sec (import) |
| 8 | kubelet + CRI-O | Pull image, create pod sandbox | ~2-5 sec |
| 9 | OVN-Kubernetes | Create logical port, assign Pod IP, program OVS | ~500 ms |
| 10 | Multus | Attach VLAN 200 secondary interface | ~300 ms |
| 11 | virt-handler | Generate libvirt domain XML, configure disks/NICs | ~1 sec |
| 12 | virt-launcher | Start libvirtd -> QEMU -> guest boot -> cloud-init | ~30-60 sec |
| Total | Clone from golden image PVC | ~40-75 sec | |
| Total | Import from HTTP URL | ~120-360 sec |
Storage Detail (OVE)
The golden-image workflow is the key to fast provisioning at scale:
Golden Image Workflow (OVE)
============================
One-time setup:
1. CDI imports base QCOW2 into a PVC:
DataVolume "rhel9-golden" -> CDI importer pod downloads from HTTP
-> converts QCOW2 to raw -> writes to Ceph RBD volume
-> PVC "rhel9-golden" is ready (takes 5-10 min for a 10 GiB image)
2. Create a VolumeSnapshot of the golden PVC:
VolumeSnapshot "rhel9-golden-snap" -> CSI calls rbd snap create
-> instant metadata operation
Per-VM provisioning:
3. DataVolume with dataSource referencing the snapshot:
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
spec:
source:
snapshot:
name: rhel9-golden-snap
namespace: golden-images
pvc:
accessModes: [ReadWriteMany]
resources:
requests:
storage: 100Gi
storageClassName: ocs-storagecluster-ceph-rbd
4. CSI driver calls rbd clone:
rbd clone pool/rhel9-golden@snap pool/vm-new-boot-disk
-> COW clone: only metadata, zero data copied
-> VM gets a thin-provisioned 100 GiB volume that reads from parent
-> Only new writes consume physical space
-> Takes ~2-3 seconds total (PVC creation + CSI clone + Ceph PG mapping)
Storage layout after cloning 10 VMs:
+----------------------------------------------------------+
| Ceph RBD Pool |
| |
| rhel9-golden (parent image, 10 GiB actual data) |
| +-- @snap (snapshot, metadata only) |
| | |
| +-- vm-001-boot (clone, COW, ~0 GiB initially) |
| +-- vm-002-boot (clone, COW, ~0 GiB initially) |
| +-- vm-003-boot (clone, COW, ~0 GiB initially) |
| ... |
| +-- vm-010-boot (clone, COW, ~0 GiB initially) |
| |
| Physical space used: ~10 GiB (shared parent) |
| Logical space provisioned: 1,000 GiB (10 x 100 GiB) |
+----------------------------------------------------------+This is fundamentally more space-efficient than a VMware full clone and comparable to VMware linked clones, with one important advantage: Ceph RBD clones are independent images that can be snapshotted, resized, and managed independently, while VMware linked clones retain a dependency chain to the parent that complicates lifecycle management.
Networking Detail (OVE)
The VM gets two network interfaces:
VM Networking on OVE
=====================
NetworkAttachmentDefinition for VLAN 200:
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: app-vlan200
namespace: production
spec:
config: |
{
"cniVersion": "0.3.1",
"name": "app-vlan200",
"type": "cnv-bridge",
"bridge": "br-vlan200",
"vlan": 200,
"ipam": {}
}
VirtualMachine spec (network section):
spec:
template:
spec:
networks:
- name: default
pod: {} # Primary: OVN-K8s Pod network
- name: app-net
multus:
networkName: app-vlan200 # Secondary: VLAN 200 via Multus
Result on the node:
+-------------------------------------------------------------------+
| Worker Node |
| |
| Physical NIC (bond0) -- trunk: VLAN 100, 200, 300, 400 |
| | |
| +-- br-ex (OVS bridge for overlay) |
| | | |
| | +-- GENEVE tunnel to other nodes |
| | +-- ovn-k8s-mp0 (management port) |
| | |
| +-- br-vlan200 (Linux bridge for VLAN 200) |
| | |
| +-- bond0.200 (VLAN subinterface) |
| +-- veth-vm-001 (connected to virt-launcher pod) |
| |
| virt-launcher Pod (vm-001) |
| +-------------------------------------------------------------+ |
| | eth0 (OVN Pod network) -- IP: 10.128.2.15/23 | |
| | net1 (Multus VLAN 200) -- passed to QEMU as virtio NIC | |
| | | |
| | QEMU Process: | |
| | VM sees: | |
| | eth0: 10.128.2.15 (pod network, used for K8s services) | |
| | eth1: 10.200.1.50 (VLAN 200, cloud-init static config) | |
| +-------------------------------------------------------------+ |
+-------------------------------------------------------------------+Step-by-step network plumbing during provisioning:
- Pod sandbox created: CRI-O creates a new network namespace for the virt-launcher pod.
- OVN-Kubernetes CNI ADD: Creates a veth pair. One end goes into the pod namespace as
eth0, the other connects to the OVS integration bridge (br-int). OVN allocates a logical switch port and assigns a Pod IP from the node's subnet. OVS flow rules are programmed to forward traffic for this port through the GENEVE overlay. - Multus CNI ADD: Reads the
NetworkAttachmentDefinitionforapp-vlan200. Invokes thecnv-bridgeplugin, which creates a veth pair connecting the pod namespace to thebr-vlan200Linux bridge. The bridge is pre-configured withbond0.200as an uplink, providing direct Layer 2 access to VLAN 200 on the physical network. - virt-handler passes NICs to QEMU: Both interfaces (
eth0from OVN,net1from Multus) are presented to the VM as virtio-net devices. The VM sees two NICs: one for the cluster network, one for the application VLAN. - cloud-init configures guest networking: The seed data (injected via a ConfigMap-backed disk or the NoCloud datasource) configures
eth1inside the guest with the static IP10.200.1.50/24, gateway, and DNS for VLAN 200.
End-to-End Flow: Azure Local
Request Entry Point
On Azure Local, VM provisioning integrates with the Azure control plane:
- Azure Portal / Azure CLI: The operator creates an Azure Arc VM resource. The request goes through Azure Resource Manager (ARM).
- Terraform (azurerm/azapi provider): ARM API call via Terraform to create an Arc VM.
- Windows Admin Center: On-premises GUI for cluster management.
All paths converge on the Azure Arc Resource Bridge, which translates ARM requests into local Hyper-V operations.
Step-by-Step
Azure Local: VM Provisioning Flow
====================================
Request Azure Resource Arc Resource Hyper-V Host
(Portal/CLI/ Manager (ARM) Bridge (on-prem (Failover
Terraform) (Azure cloud) appliance VM) Cluster node)
| | | |
| 1. ARM API call | | |
| PUT /providers/ | | |
| Microsoft.AzureStack| | |
| HCI/virtualMachines | | |
+---------------------> | | |
| | | |
| 2. ARM validates: | | |
| - Azure RBAC | | |
| - Subscription | | |
| quota | | |
| - Template/image | | |
| gallery ref | | |
| ~500-1000 ms | | |
| (cloud roundtrip) | | |
| | | |
| | 3. ARM dispatches | |
| | to Arc Resource | |
| | Bridge via | |
| | Arc connected | |
| | cluster agent | |
| +----------------------->| |
| | ~1-2 sec | |
| | (depends on | |
| | connectivity) | |
| | | |
| | | 4. Arc Bridge |
| | | translates ARM |
| | | spec to local |
| | | Hyper-V config: |
| | | - Select cluster |
| | | node (best fit) |
| | | - Create VHDX |
| | | from gallery |
| | | image |
| | +--------------------->|
| | | ~1 sec |
| | | |
| | | 5. Storage:
| | | - S2D CSV
| | | allocation
| | | - VHDX copy
| | | from gallery
| | | image
| | | - ReFS block
| | | cloning
| | | (if same CSV)
| | | ~5-30 sec
| | | (block clone)
| | | ~60-180 sec
| | | (full copy)
| | | |
| | | 6. Networking:
| | | - VMSwitch
| | | port
| | | created
| | | - VLAN 200
| | | tag set
| | | - SDN Network
| | | Controller
| | | programs
| | | VFP rules
| | | ~1-2 sec
| | | |
| | | 7. VM created:
| | | - Hyper-V
| | | creates VM
| | | - Assigns
| | | vCPUs, RAM
| | | - Attaches
| | | VHDX
| | | - Powers on
| | | ~2-3 sec
| | | |
| | | 8. Guest boot:
| | | UEFI -> BCD
| | | -> kernel ->
| | | cloud-init/
| | | Sysprep
| | | ~30-90 sec
| | | |
| 9. ARM resource | | |
| provisioning | | |
| state: | | |
| "Succeeded" | | |
|<----------------------+ | |
| Total: ~45-130 sec (block clone) |
| ~120-300 sec (full copy) |Timing Summary (Azure Local)
| Step | Component | Action | Duration |
|---|---|---|---|
| 1 | ARM API | Receive and validate request | ~200 ms |
| 2 | ARM | Azure RBAC, subscription quota, image validation | ~500-1000 ms |
| 3 | Arc agent | Relay to on-prem Arc Resource Bridge | ~1-2 sec |
| 4 | Arc Resource Bridge | Translate ARM spec, select target node | ~1 sec |
| 5 | S2D / ReFS | Allocate CSV, create VHDX (block clone or full copy) | ~5-30 sec (clone) / ~60-180 sec (copy) |
| 6 | SDN Network Controller | Create VMSwitch port, program VFP/VLAN rules | ~1-2 sec |
| 7 | Hyper-V | Create VM, assign resources, power on | ~2-3 sec |
| 8 | Guest OS | UEFI -> boot -> cloud-init/Sysprep | ~30-90 sec |
| Total | Block clone from gallery image | ~45-130 sec | |
| Total | Full copy from gallery image | ~120-300 sec |
Storage Detail (Azure Local)
Azure Local Storage: VHDX on S2D CSV
======================================
Gallery image workflow:
1. Admin uploads VHDX to Azure Arc image gallery
-> Stored on a Cluster Shared Volume (CSV) path:
C:\ClusterStorage\Volume01\Gallery\rhel9-golden.vhdx
2. Per-VM provisioning:
Option A -- ReFS Block Cloning (same CSV):
- ReFS copies metadata pointers, not data blocks
- New VHDX references same physical extents
- Writes trigger COW at the ReFS extent level (64 KiB)
- Duration: ~5-30 seconds (metadata + differencing disk setup)
Option B -- Full copy (different CSV or non-ReFS):
- Block-by-block copy of VHDX
- 100 GiB at ~500 MB/s = ~200 seconds
- Duration: ~60-180 seconds
S2D replication:
+------------------------------------------------------+
| Cluster Shared Volume (CSV) on S2D |
| |
| Volume01 (3-way mirror, ReFS) |
| | |
| +-- Gallery/rhel9-golden.vhdx (10 GiB actual) |
| +-- VMs/vm-001/boot.vhdx (differencing from |
| | golden, initially ~0) |
| +-- VMs/vm-002/boot.vhdx |
| +-- VMs/vm-003/boot.vhdx |
| |
| S2D ensures 3 copies across 3 nodes (mirror) |
| Write path: VM -> ReFS -> CSV -> S2D Storage Bus |
| -> write to local SSD/NVMe + replicate to 2 peers |
| -> acknowledge when majority (2/3) written |
+------------------------------------------------------+Networking Detail (Azure Local)
Azure Local uses the Hyper-V Virtual Switch (VMSwitch) combined with Microsoft SDN Network Controller (optional) or simple VLAN tagging:
- VMSwitch: Each Hyper-V host has one or more virtual switches bound to physical NICs (typically SET -- Switch Embedded Teaming, which combines NIC teaming with the virtual switch).
- VLAN tagging: The VM's virtual NIC is configured with VLAN 200. The VMSwitch applies the VLAN tag on egress and strips it on ingress.
- SDN Network Controller (if deployed): The Network Controller programs Virtual Filtering Platform (VFP) rules on each VMSwitch to enforce ACLs, QoS, and overlay networking (VXLAN). This is the equivalent of NSX DFW.
- IP assignment: cloud-init (Linux) or Sysprep/unattend.xml (Windows) configures the guest NIC. Azure Local also supports Azure-style DHCP via the Arc VM networking extension.
Azure Dependency: Internet Connectivity
A critical architectural difference: Azure Local requires connectivity to Azure for the ARM control plane. Steps 1-3 involve a roundtrip to Azure cloud services. If the internet connection to Azure is down, new VM provisioning through the ARM path is blocked. Existing VMs continue to run (the data plane is fully on-premises), but management operations are degraded. Failover Cluster Manager and PowerShell can provision VMs locally without Azure, but this bypasses the Arc management layer and loses the unified Azure portal experience.
For a Tier-1 financial institution, this dependency on external cloud services for management operations must be evaluated against operational resilience requirements. FINMA expects that critical infrastructure can be operated independently during connectivity outages.
Three-Platform Comparison
Provisioning Timing Comparison
Provisioning Duration Comparison (Clone from Template)
========================================================
VMware OVE Azure Local
(linked clone) (RBD clone) (ReFS block clone)
API/admit |====| 0.7s |=====| 0.8s |=========| 1.5-3s
Schedule |===| 0.5s |==| 0.2s (included in Arc
Bridge step)
Storage |==| 2s |===| 2-3s |=======| 5-30s
Network |=| 0.1s |=====| 0.8s |==| 1-2s
VM create |===| 2-3s |=====| 2-6s |===| 2-3s
Guest boot |===========| |===========| |===========|
30-60s 30-60s 30-90s
___________ ___________ ___________
TOTAL ~40-70s ~40-75s ~45-130sFeature Comparison
| Capability | VMware vSphere | OVE (KubeVirt) | Azure Local |
|---|---|---|---|
| API model | SOAP/REST (vCenter) | Kubernetes API (kubectl) | ARM REST API (Azure) |
| GitOps native | No (requires wrapper) | Yes (ArgoCD, Flux) | Partial (ARM templates) |
| Clone mechanism | Linked clone / full clone | Ceph RBD clone (COW) | ReFS block clone / VHDX differencing |
| Clone speed (100 GiB) | ~2 sec (linked) | ~2-3 sec (RBD clone) | ~5-30 sec (ReFS clone) |
| Template management | vCenter template library | DataVolume + VolumeSnapshot | Azure Arc image gallery |
| Scheduler | DRS (proprietary) | kube-scheduler (pluggable) | Failover Cluster + Arc Bridge |
| Network model | vDS + port group + VLAN | OVN (overlay) + Multus (VLAN) | VMSwitch + VLAN + optional SDN |
| Multi-NIC | Multiple vNICs on port groups | Multus NetworkAttachmentDefinition | Multiple vNICs on VMSwitch |
| cloud-init support | Via GOSC / OVF properties | Native (NoCloud datasource) | Via Sysprep (Windows) / cloud-init (Linux) |
| RBAC granularity | vCenter roles + permissions | Kubernetes RBAC (namespace-scoped) | Azure RBAC (subscription/resource group) |
| Quota enforcement | Resource pools (soft/hard) | ResourceQuota + LimitRange | Azure subscription quotas |
| Offline provisioning | Full (vCenter on-prem) | Full (cluster is self-contained) | Degraded (ARM path requires Azure) |
Storage Efficiency Comparison
Provisioning 100 VMs from a 10 GiB Golden Image (100 GiB logical per VM)
==========================================================================
Logical provisioned: 100 VMs x 100 GiB = 10,000 GiB (all platforms)
Physical consumed at provisioning time:
VMware (linked clone):
Parent VMDK: 10 GiB (shared)
Delta disks: ~0 GiB (initially)
Overhead: ~10 GiB per FTT=1 mirror (vSAN)
Total: ~20 GiB physical
Ratio: 500:1 thin provisioning ratio
OVE (Ceph RBD clone):
Parent image: 10 GiB (shared, 3x replicated = 30 GiB raw)
Clone COW: ~0 GiB (initially)
Total: ~30 GiB raw physical (3x replication)
Ratio: ~333:1 thin provisioning ratio
Note: Replication factor higher than VMware FTT=1
but raw efficiency is comparable
Azure Local (ReFS block clone):
Gallery VHDX: 10 GiB (shared, 3-way mirror = 30 GiB raw)
Diff disks: ~0 GiB (initially)
Total: ~30 GiB raw physical (3-way mirror)
Ratio: ~333:1 thin provisioning ratioFailure Modes During Provisioning
| Failure | VMware | OVE | Azure Local |
|---|---|---|---|
| Template not found | vCenter returns error, no VM created | CDI DataVolume enters ImportNotFound phase |
ARM returns 404 for gallery image |
| Insufficient storage | CLOM rejects placement, error in vCenter task | PVC stays Pending, CSI driver reports capacity error in events |
S2D reports insufficient CSV space |
| No suitable host | DRS returns "no compatible host", VM not created | Pod stays Pending, scheduler reports Insufficient cpu/memory in events |
Failover Cluster cannot find node with capacity |
| Network config error | Port group missing: VM created but vNIC disconnected | NAD missing: Multus CNI fails, pod enters CrashLoopBackOff or ContainerCreating with event error |
VMSwitch not configured for VLAN: NIC connected but no Layer 2 path |
| cloud-init failure | VM boots but unconfigured. Requires console access to debug. | Same -- VM boots but unconfigured. Debug via virtctl console or virtctl ssh. |
Same -- VM boots but unconfigured. Debug via Azure Serial Console or RDP. |
| Timeout / partial failure | vCenter task shows partial progress. Can retry or clean up manually. | Kubernetes resources left in intermediate state. kubectl delete vm cleans up (virt-controller handles cascading deletion of VMI, Pod, PVC if ownerReferences are set). |
ARM deployment shows failed. Arc Bridge may have partially created resources on-prem. Manual cleanup via PowerShell may be needed. |
Automation at Scale: Provisioning 100 VMs
For the migration factory (5,000+ VMs in waves), provisioning speed and parallelism matter. Here is a realistic estimate for provisioning 100 VMs simultaneously:
Batch Provisioning: 100 VMs (clone from template, 4 vCPU / 16 GiB each)
==========================================================================
VMware (vCenter):
- vCenter processes API calls serially per datacenter (task queue)
- Clone tasks run in parallel on different hosts (limited by
concurrent provisioning operations setting, default 8 per host)
- 10 hosts x 8 parallel clones = 80 concurrent operations
- Linked clone: ~2 sec per clone + 30-60 sec boot = ~70 sec total
- But: vCenter task queue can bottleneck at 100+ simultaneous tasks
- Realistic total: ~2-3 minutes for all 100 VMs to reach running state
OVE (KubeVirt):
- kube-apiserver handles 100 VM CRs concurrently (etcd write throughput
is the bottleneck, typically >1000 writes/sec)
- CDI processes DataVolumes in parallel (configurable concurrency)
- Ceph RBD clones are independent and parallelizable
- kube-scheduler binds 100 pods in <5 seconds
- Guest boot is fully parallel across nodes
- Realistic total: ~1-2 minutes for all 100 VMs to reach Running phase
- Bottleneck: CDI importer concurrency if importing (not cloning)
Azure Local:
- ARM API has throttling limits (per-subscription, per-resource-provider)
- Arc Resource Bridge processes requests serially or with limited
concurrency (implementation-dependent)
- S2D block clones can run in parallel across different CSV owners
- Realistic total: ~3-5 minutes for all 100 VMs to reach running state
- Bottleneck: ARM API throttling and Arc Bridge concurrency
- Workaround: Use PowerShell directly on-cluster to bypass ARM
(loses Azure portal visibility)Key Takeaways
-
Clone speed is comparable across platforms. All three support COW/thin cloning from golden images. The guest OS boot time (30-90 seconds) dominates total provisioning time, making the infrastructure overhead (2-30 seconds) a secondary concern for individual VMs.
-
OVE has the best automation story. Native Kubernetes API means GitOps (ArgoCD), Terraform, and CI/CD pipelines work without adapters. Every provisioning operation is a declarative YAML manifest versioned in Git. VMware requires vCenter-specific tooling; Azure Local requires ARM integration.
-
Azure Local has a cloud dependency for management. The ARM roundtrip adds latency and introduces a dependency on internet connectivity. This is architecturally different from VMware and OVE, both of which operate fully on-premises.
-
Storage efficiency is a function of replication factor, not cloning mechanism. All three platforms achieve near-zero initial space consumption for clones. The real space difference comes from the replication factor: vSAN FTT=1 (2x), Ceph replica=3 (3x), S2D 3-way mirror (3x). At 5,000 VMs, this difference is significant for raw capacity planning.
-
Networking complexity is highest on OVE. The OVN overlay + Multus secondary network model is more components than VMware's vDS + port group or Azure Local's VMSwitch + VLAN. The VMware team will need to learn OVN logical switch/router concepts, Multus NAD configuration, and OVS flow debugging. The networking training budget should reflect this.
-
Failure recovery is most declarative on OVE. Failed provisioning on OVE leaves Kubernetes resources in known states with clear events explaining the failure. Deleting and re-creating is straightforward. VMware task failures may require vCenter task investigation. Azure Local failures may leave split state between ARM and on-prem.
Discussion Guide
These questions should be raised with engineering teams and vendors during the PoC:
-
For OVE: What is the CDI DataVolume concurrency limit in the current configuration? Can it handle 50+ simultaneous clone operations without backpressure? What is the Ceph cluster's IOPs ceiling for parallel RBD clone creation?
-
For Azure Local: What is the ARM API throttling limit for VM creation? If we need to provision 100 VMs in a wave, will we hit the subscription-level rate limit? Can we use local PowerShell provisioning during the migration factory and reconcile with Arc afterwards?
-
For all platforms: What is the end-to-end provisioning time for a Windows Server VM with Sysprep (not just Linux with cloud-init)? Windows Sysprep specialization typically adds 5-10 minutes to the boot sequence. Is this acceptable for the migration factory's cadence?
-
For all platforms: How does the golden image lifecycle work? When the RHEL 9 base image is updated (security patches), how is the golden image refreshed, and how are existing clones affected? On OVE, updating the parent image does NOT retroactively update existing clones (they are independent after cloning). A new golden image requires new clones for new VMs.
-
For the networking team: In the OVE model, VMs on VLAN 200 via Multus communicate directly on the physical VLAN (no overlay). VMs on the OVN Pod network communicate via GENEVE overlay. What is the impact on existing firewall rules, monitoring tools, and network segmentation policies that assume all VM traffic is on physical VLANs?