Scenario: Live Migrate a VM During Node Maintenance

Why This Scenario Matters

Live migration is the single most operationally critical capability in a virtualized environment. At 5,000+ VMs spread across 100+ hypervisor hosts, the infrastructure team performs planned maintenance weekly -- kernel patches, firmware updates, DIMM replacements, NIC swaps, BIOS upgrades. Every one of these operations requires evacuating running VMs from the host being serviced. Without live migration that completes reliably within minutes and with near-zero guest downtime, each maintenance window becomes a negotiation with application owners, a change advisory board ticket, and an after-hours event.

In the VMware baseline, vMotion handles this transparently. An administrator puts a host into maintenance mode, DRS migrates all VMs to other hosts, the patching proceeds, and the host returns to the cluster. The team has done this thousands of times. It works. The replacement platform must replicate this operational pattern or provide an equivalent that is at least as reliable and at least as fast.

This scenario traces a single, concrete operation: "Drain worker-07 for a kernel security patch. The node runs 48 VMs, including a 64 GiB Oracle database VM on VLAN 300. Migrate all VMs to other nodes with zero unplanned downtime, complete the patch, and return the node to service." The same operation is executed on all three platforms.

End-to-End Flow: VMware vSphere (Baseline)

Trigger and Per-VM vMotion Sequence

The administrator right-clicks the host in vCenter and selects "Enter Maintenance Mode" (or automates via PowerCLI). vCenter validates the host, DRS computes an evacuation plan (scoring every target host by CPU/RAM fit, NUMA, datastore accessibility, affinity/anti-affinity rules), and executes migrations in parallel (default: 4 concurrent outbound per host).

For each VM, the following traces a 64 GiB Oracle database VM on shared vSAN:

  vMotion: oracle-prod-01 (64 GiB RAM, 8 vCPUs, shared vSAN)
  ==============================================================

  Source ESXi (worker-07)                     Target ESXi (worker-03)

  T=0.0s    Pre-flight checks                                  ~200 ms
  --------  Target has 64 GiB free, vSAN accessible, EVC compat, no
            anti-affinity violations.

  T=0.2s    Create shadow VM on target                          ~300 ms
  --------  Allocate memory, open TCP port for migration stream.

  T=0.5s    Pre-copy iteration 0: bulk transfer                 ~13 sec
  --------  Copy ALL 64 GiB. VM continues running on source.
            Transfer rate: ~5 GB/s on 2x 25 GbE vMotion NICs.

  T=13.5s   Pre-copy iteration 1: dirty pages                  ~1.6 sec
  --------  Oracle dirtied ~8 GiB (SGA buffer cache, redo, PGA).
            Send 8 GiB at 5 GB/s.

  T=15.1s   Pre-copy iteration 2                               ~0.3 sec
  --------  1.5 GiB dirtied. Dirty set shrinking.

  T=15.4s   Pre-copy iteration 3: converging                   ~40 ms
  --------  200 MB dirtied. Small enough for switchover.

  T=15.5s   Switchover (VM stun)                                ~100 ms
  --------  Source PAUSES VM. Final 200 MB + CPU state + device
            state sent. THIS IS THE ACTUAL GUEST DOWNTIME.

  T=15.6s   Resume on target                                   ~50 ms
  --------  Target RESUMES VM. dvSwitch port re-bound. Source sends
            RARP to update physical switch MAC table.

  T=15.9s   Migration complete
  --------  Total wall-clock: ~16 sec. Guest downtime: ~100 ms.

Storage During vMotion

Shared storage (vSAN, SAN, NFS): No disk migration. Both hosts access the same datastore. Only memory + CPU state transfer. This is the fast path.

Storage vMotion (no shared storage): Memory pre-copy runs in parallel with disk block-level mirroring. Source writes to both old and new datastores. A 100 GiB disk at 500 MB/s adds ~200 seconds, and migration cannot complete until both memory and disk converge.

Networking During vMotion

1. Before switchover: VM's vNIC on dvPortGroup (VLAN 300) on source host.
2. At switchover: vCenter moves dvPort binding to target. Target host's dvSwitch now handles traffic.
3. RARP broadcast: Target ESXi sends RARP with VM's MAC on VLAN 300. Physical switches update MAC tables within milliseconds.
4. IP preservation: VM IP unchanged. TCP connections survive (pause < retransmission timeout).
5. ARP caches: MAC address itself does not change -- only the switch port association. GARP resolves the switching fabric; end hosts' ARP caches remain valid.

Full Node Evacuation (48 VMs)

  Configuration: 48 VMs, 4 concurrent vMotion, avg 32 GiB, 2x25 GbE, shared vSAN

  Wave 1:  VMs 1-4    T=0:00  -> T=0:30    (largest VMs first)
  Wave 2:  VMs 5-8    T=0:30  -> T=0:55
  ...
  Wave 12: VMs 45-48  T=4:30  -> T=4:50

  Evacuation: ~5 min (best) / ~8-12 min (typical) / ~15-20 min (worst, 256 GiB VMs)
  Patching:   ~5-10 min
  Rebalance:  DRS rebalances automatically within ~5 min
  Total:      ~15-30 minutes per node

End-to-End Flow: OpenShift Virtualization Engine (OVE)

On OVE, the equivalent operation is kubectl drain. Kubernetes evicts pods (including virt-launcher pods), which triggers KubeVirt's live migration controller. The flow is structurally different: instead of DRS computing an evacuation plan, Kubernetes evicts pods sequentially (limited by PDBs), and each eviction triggers an independent scheduling + migration decision.

Trigger: Cordon and Drain

# Step 1: Cordon (prevent new scheduling, existing VMs keep running)
$ kubectl cordon worker-07

# Step 2: Drain (evict all pods, triggering live migration for VMs)
$ kubectl drain worker-07 --delete-emptydir-data --ignore-daemonsets --timeout=3600s

Drain Sequence (Orchestration Layer)

OVE: Node Drain -> Live Migration
====================================

  kubectl drain         kube-apiserver      virt-controller       kube-scheduler
     |                       |                    |                    |
     | 1. Cordon: taint      |                    |                    |
     |    NoSchedule         |                    |                    |
     +---------------------> |                    |                    |
     |                       |                    |                    |
     | 2. For each VM pod:   |                    |                    |
     |    Eviction API call  |                    |                    |
     |    -> check PDB       |                    |                    |
     |    -> if allowed:     |                    |                    |
     +---------------------> |                    |                    |
     |                       | 3. virt-controller |                    |
     |                       |    intercepts via  |                    |
     |                       |    EvictionStrategy:|                   |
     |                       |    "LiveMigrate"   |                    |
     |                       +-------------------> |                    |
     |                       |                    |                    |
     |                       | 4. Creates VMIM CR |                    |
     |                       | 5. Creates NEW     |                    |
     |                       |    virt-launcher   | 6. Schedule target |
     |                       |    pod             +-------------------> |
     |                       |                    |    Score nodes:     |
     |                       |                    |    CPU/RAM fit,     |
     |                       |                    |    affinity, taints,|
     |                       |                    |    device plugins   |
     |                       |                    |    ~200 ms          |
     |                       |                    |                    |
     | 7. kubelet starts pod, CNI setup (OVN + Multus), migration     |
     |    handshake, then QEMU pre-copy begins                        |

Per-VM Migration (KubeVirt, 64 GiB Oracle VM, RWX Ceph)

  KubeVirt Live Migration: oracle-prod-01 (64 GiB, RWX Ceph volume)
  ===================================================================

  Source virt-launcher (worker-07)            Target virt-launcher (worker-03)
  Both pods mount the SAME Ceph RBD volume (RWX). No disk migration needed.

  T=0.0s    Pre-flight / pod setup                              ~2-3 sec
  --------  VMIM created, target pod created, scheduled, started.
            CNI: OVN logical port + OVS flows on target. Multus: VLAN
            bridge attached. Migration TCP handshake established.

  T=2.5s    MigrationPolicy applied                             ~100 ms
  --------  Policy "production-migration-policy" matches VMI labels:
            bandwidthPerMigration: 1Gi, allowAutoConverge: true,
            completionTimeoutPerGiB: 150, allowPostCopy: false.

  T=2.6s    Pre-copy iteration 0: bulk transfer                 ~64 sec
  --------  All 64 GiB sent at ~1 GB/s (bandwidth cap). VM runs on source.
            (Capped at 1Gi because multiple VMs migrate concurrently.)

  T=66.6s   Pre-copy iteration 1: dirty pages                  ~6 sec
  --------  Oracle dirtied ~6 GiB during bulk copy (~100 MB/s dirty rate).

  T=72.6s   Pre-copy iterations 2-3: converging                ~0.7 sec
  --------  600 MB -> 60 MB. Dirty set small enough for switchover.

  T=73.3s   Switchover                                          ~150 ms
  --------  Source QEMU pauses vCPUs. Final 60 MB + CPU + device state
            sent. Target QEMU resumes. Guest downtime: ~150 ms.

  T=73.5s   Network switchover + GARP                           ~100-500 ms
  --------  GARP on VLAN 300 updates physical switch MAC tables.
            OVN SB DB Chassis binding updated -> GENEVE tunnels redirect.

  T=74.0s   Cleanup                                             ~500 ms
  --------  VMI status updated (nodeName: worker-03). Source pod terminated.
            OVN/OVS source port cleaned up.

  Total wall-clock: ~75 sec. Guest downtime: ~150 ms.

Why slower than vMotion? Bandwidth capped at 1 Gi for concurrent drain. With 2x25 GbE and bandwidthPerMigration: 3Gi, bulk copy takes ~22 sec, total ~28 sec -- comparable to vMotion.

Storage: RWX vs. RWO -- The Critical Distinction

  Path A: RWX (ReadWriteMany) -- FAST PATH
  ==========================================
  Source Pod          Target Pod
  +--------+         +--------+
  | /dev/  |         | /dev/  |    Both mount SAME Ceph RBD image.
  | rbd0   |         | rbd0   |    No data migration. Only memory transfer.
  +---+----+         +---+----+    Equivalent to vMotion with shared vSAN.
      |                  |
      +-- Ceph RADOS ----+         PVC: accessModes: [ReadWriteMany]
                                         volumeMode: Block

  Path B: RWO (ReadWriteOnce) -- SLOW PATH (storage migration)
  ==============================================================
  Source Pod          Target Pod
  +--------+         +--------+
  | vol-A  | ======> | vol-B  |    CDI copies blocks to NEW PVC.
  +--------+  copy   +--------+    Writes mirrored during migration.
                                   100 GiB at 500 MB/s = ~200 sec EXTRA.

  RECOMMENDATION: Use RWX Block mode for ALL VM disks.
  Enforce via StorageClass defaults and admission webhooks.

Networking: Two Layers

Pod Network (OVN overlay): Pod IP changes (new pod gets new IP from target node's CIDR). KubeVirt updates VMI pod reference; Kubernetes Services with selectors continue to work. OVN SB DB Chassis binding updated; GENEVE tunnels redirect within ~100 ms.

VLAN Network (Multus/cnv-bridge): This is the production-visible network. Guest IP on VLAN 300 is preserved. At switchover, target QEMU sends GARP through br-vlan300 -> bond0.300. Physical switches update MAC tables in ~10-50 ms. Identical to vMotion RARP behavior.

EvictionStrategy: The Make-or-Break Setting

spec:
  template:
    spec:
      evictionStrategy: LiveMigrate    # WITHOUT THIS, kubectl drain KILLS the VM
      # Options: LiveMigrate | LiveMigrateIfPossible | External | None

If evictionStrategy is not set, kubectl drain terminates the VM ungracefully. Every VM in the estate must have this set. A mutating admission webhook should enforce it for all production namespaces.

Full Node Drain (48 VMs)

  Configuration: 48 VMs, parallelOutbound=5, bandwidthPerMigration=1Gi,
                 25 GbE migration network, Ceph RBD RWX, PDBs for DB clusters

  Effective concurrency: 5 at 1 Gi = 5 GB/s (25 GbE provides ~3.1 GB/s,
  so bandwidth-limited to ~3 concurrent at full speed)

  Batch 1:  VMs 1-5    T=0:00  -> T=1:10    (PDB may delay DB cluster VMs)
  Batch 2:  VMs 6-10   T=1:10  -> T=2:00
  ...
  Batch 10: VMs 46-48  T=8:00  -> T=8:40

  Drain: ~8-12 min (typical) / ~15-20 min (large VMs or PDB contention)
  Patching: ~5-10 min
  Rebalance: NONE (no DRS equivalent, VMs stay on target nodes)
  Total: ~20-35 minutes per node

End-to-End Flow: Azure Local

Azure Local uses Hyper-V Live Migration orchestrated by Windows Failover Clustering.

Trigger

Suspend-ClusterNode -Name "worker-07" -Drain
# Or: Windows Admin Center > Cluster > Servers > worker-07 > Pause > Drain Roles
# Or: Azure Portal > Maintenance Mode (requires Azure connectivity)

Failover Cluster Manager enumerates all VM roles on the node, selects the best target node for each (CPU/RAM fit, preferred owners, anti-affinity groups), and initiates Live Migrations (default: 1 concurrent, configurable to 4).

Per-VM Hyper-V Live Migration (64 GiB Oracle VM, CSV on S2D)

  Hyper-V Live Migration: oracle-prod-01 (64 GiB, CSV on S2D)
  =============================================================

  T=0.0s    Pre-flight                                          ~500 ms
  --------  Target node healthy, 64 GiB free, CSV accessible,
            processor compat mode OK, Live Migration network reachable.

  T=0.5s    Setup                                               ~500 ms
  --------  Target creates placeholder VM, allocates memory,
            TCP connection on Live Migration network.

  T=1.0s    Memory transfer (pre-copy)                          ~54 sec (10 GbE)
  --------  Same iterative pre-copy as KVM/vMotion.             ~13 sec (RDMA)
            Dirty tracking via SLAT/EPT.
            64 GiB at ~1.2 GB/s (10 GbE) or ~4 GB/s (RDMA).

  T=55.0s   Dirty page iterations (3-5 rounds)                 ~5-8 sec
  --------  Same convergence pattern. Oracle dirty rate ~100 MB/s.

  T=62.0s   Switchover                                          ~200 ms
  --------  Pause, final state, resume. Guest downtime: ~200 ms.
            GARP on VLAN 300. VMSwitch port binding updated.

  T=62.5s   CSV ownership update                                ~100 ms
  --------  If target is not CSV coordinator: redirected I/O
            (~100-200 us latency per I/O until auto-rebalance, ~30 sec).

  Total wall-clock: ~63 sec (10 GbE) / ~20 sec (RDMA).
  Guest downtime: ~200 ms.

Storage: CSV on S2D

CSVs are inherently shared -- no RWX/RWO distinction needed. Both hosts access the same CSV path. S2D does not move data blocks during migration. Only CSV owner metadata is updated. If the target node is not the CSV coordinator, I/O is redirected over SMB to the coordinator node (adds ~100-200 us latency) until automatic rebalance (~30 sec).

Hyper-V also supports Storage Live Migration (VHDX mirrored in real-time), but Azure Local clusters always use S2D CSVs, so this path is not typical.

Full Node Drain (48 VMs)

  Configuration: 48 VMs, 2 concurrent Live Migration, avg 32 GiB,
                 10 GbE Live Migration network, S2D CSV

  Drain: ~18-20 min (10 GbE, 2 concurrent) / ~6-8 min (RDMA, 4 concurrent)
  Patching: ~5-10 min
  Rebalance: Resume-ClusterNode -Failback Immediate (VMs auto-return)
  Total: ~25-35 min (10 GbE) / ~15-20 min (RDMA)

Swisscom ESC Considerations

ESC is a managed IaaS offering on VMware vSphere. The customer does not trigger node maintenance -- Swisscom operates the infrastructure. The evaluation must assess:

• Transparency: Maintenance notification process, shared calendar.
• SLA: Zero-downtime guarantee via vMotion? Terms for maintenance-induced disruption?
• Control: Can the customer set anti-affinity rules? Prevent migration during batch windows?
• Visibility gap: The team currently monitors vMotion tasks in vCenter. In ESC, this visibility is Swisscom's responsibility. The customer sees only "VM is running."

Failure Scenarios

Failure 1: Migration Does Not Converge

A 256 GiB SAP HANA instance with 2 GB/s dirty rate vs. 1 GB/s migration bandwidth will never converge (dirty set grows faster than transfer).

Failure 2: Target Node Fails Mid-Migration

  Pre-copy (VM running on source):
  =================================
  VM is SAFE. Migration TCP breaks. Source QEMU aborts. VM continues
  running on source without interruption. Migration retried to another node.
  Identical on all three platforms.

  Post-copy (VM running on target, KubeVirt only):
  ==================================================
  If TARGET crashes: VM lost. Restart from disk (crash recovery).
  If SOURCE crashes: VM hangs (cannot fault pages). Kill and restart.
  THIS IS WHY allowPostCopy: false IS RECOMMENDED FOR PRODUCTION.

Failure 3: Network Partition

All platforms: migration TCP stream breaks. In pre-copy, VM continues on source (safe). In post-copy (KubeVirt only), VM on target hangs. Mitigation: bonded NICs with LACP fast timeout (1 sec) for migration network.

Failure 4: Insufficient Resources

All platforms: migration blocked. kubectl drain blocks (OVE), maintenance mode blocked (VMware), node drain halted (Azure Local). Resolution: power off non-critical VMs, add capacity, or abort maintenance.

Three-Platform Comparison

Migration Timing (Single 64 GiB VM, Shared Storage)

Live Migration Duration Comparison
=====================================

                  VMware vMotion      KubeVirt (OVE)       Hyper-V (Azure Local)
                  (2x25 GbE)         (25 GbE, 1Gi cap)    (10 GbE)

  Pre-flight      |=| 0.5s            |===| 2.5s            |=| 1.0s
  Bulk copy       |=======| 13s       |=====================| 64s   |=================| 54s
  Dirty iters     |==| 2s             |====| 7s              |====| 7s
  Switchover      |.| 100ms           |.| 150ms              |.| 200ms
                  ____________        ____________           ____________
  TOTAL           ~16 sec             ~75 sec                ~63 sec
  Downtime        ~100 ms             ~150 ms                ~200 ms

  With matched network (all 2x25 GbE, no cap / RDMA):
  TOTAL           ~16 sec             ~28 sec                ~17 sec

Node Drain Timing (48 VMs, avg 32 GiB)

Feature Comparison

Operational Differences: OVE-Specific Configuration Requirements

1. EVICTION STRATEGY
   Every VM MUST have evictionStrategy: LiveMigrate.
   Without it, kubectl drain KILLS the VM.
   VMware/Hyper-V: vMotion is always the default. No equivalent setting needed.

2. STORAGE ACCESS MODE
   PVCs must be RWX Block mode for zero-copy migration.
   RWO triggers slow storage migration path.
   VMware/Hyper-V: shared storage is inherently shared. No distinction.

3. MIGRATION POLICY
   MigrationPolicy CRDs must be defined and matched to VM labels.
   Without policy: no auto-converge, default bandwidth, default timeout.
   VMware: vMotion NIC speed + DRS aggressiveness (cluster-level).

4. PDB MANAGEMENT
   PDBs must exist for VM groups needing availability guarantees.
   Without PDBs: all VMs on a node can be evicted simultaneously.
   VMware: DRS anti-affinity + HA admission control.

5. NO DRS
   After draining and patching, VMs stay where they migrated.
   After patching all 9 nodes: VMs concentrated on 3 nodes, 6 idle.
   Must use Descheduler or build custom rebalancing.

Pre-Maintenance Checklist

Key Takeaways

1. The core algorithm is identical across platforms. Pre-copy with iterative dirty page convergence. Migration time = f(memory, dirty rate, bandwidth). No candidate changes the physics.

2. OVE requires more Day-1 configuration. evictionStrategy, MigrationPolicy, PDBs, RWX access modes, dedicated migration NADs must all be explicitly set. VMware ships with sensible defaults. The OVE team must build this framework before the first production drain.

3. Storage access mode is OVE's biggest migration risk. RWO PVCs (common default) turn every live migration into a storage migration. Standard must be RWX Block for all VM disks, enforced via StorageClass defaults and admission webhooks.

4. No DRS equivalent on OVE is a real gap. After maintenance, VMs are not rebalanced. Over multiple patch cycles, cluster utilization becomes uneven. Azure Local's Failback option partially addresses this.

5. Post-copy is powerful but dangerous. Only KubeVirt supports it. Guarantees convergence but introduces unrecoverable failure modes. Keep allowPostCopy: false for production; use allowAutoConverge: true instead.

6. Azure Local defaults are conservative. Default 1 concurrent Live Migration leads to long drain times. Tuning to 2-4 concurrent with RDMA brings times in line with VMware. Day-1 configuration item.

7. Drain timing at scale drives maintenance window planning. At ~15-30 min per node and 9 worker nodes, full cluster patching takes 2-4.5 hours. Factor into change management scheduling.

Discussion Guide

1. For OVE: What percentage of VM PVCs are RWX vs. RWO? What is the path to convert RWO to RWX online?

2. For OVE: Can a mutating admission webhook enforce evictionStrategy: LiveMigrate on all VMs in production namespaces?

3. For OVE: What is the tested maximum concurrent live migration count? At what concurrency does the migration network or Ceph become the bottleneck?

4. For Azure Local: What is the measured Live Migration throughput (10 GbE vs. 25 GbE vs. RDMA)? Is Quick Migration an acceptable fallback for 256 GiB VMs?

5. For Swisscom ESC: What visibility does the customer have into live migration events? Can migration start/end times and guest downtime be queried via API?

6. For all platforms: Can a 256 GiB SAP HANA instance be live migrated within a 15-minute maintenance window on the proposed hardware?

7. For all platforms: What happens if a node fails while being drained (some VMs migrated, some in-flight, some not started)? What is the recovery procedure?

8. For networking: During a 48-VM drain, is there risk of GARP storms on VLAN segments? Should GARP rate-limiting be configured on switches?

9. For capacity planning: Can the cluster absorb two simultaneous node losses (one planned, one unplanned)? What is the current N+2 headroom?