Scenario: Rolling OS/Platform Update with Zero Downtime
Why This Scenario Matters
Patching is the most frequent disruptive operation in any virtualization estate. At 5,000+ VMs across a 10+ node cluster, the current VMware environment applies ESXi patches, firmware updates, and security fixes on a monthly cadence -- sometimes more frequently when critical CVEs drop. The patching workflow exercises every layer of the stack simultaneously: the orchestrator must decide which node to patch first, the hypervisor must live migrate all VMs off that node, the storage subsystem must tolerate a temporarily missing member, and the network must reprogram forwarding rules as VMs shift between hosts. A single misstep in any of these layers causes VM downtime.
This scenario traces a concrete operation: "Apply a new host OS version (RHCOS 4.16.8 / Windows Server 2025 CU / ESXi 8.0 U3) across a 10-node cluster running 500 VMs, with zero VM downtime, during a planned maintenance window." The same operation is executed on all three platforms so the evaluation team can compare the operational experience, duration, risk surface, and rollback capabilities.
For a Tier-1 financial institution regulated by FINMA, the patching scenario intersects directly with operational resilience requirements. The regulator expects that critical infrastructure can be patched without service interruption, that rollback procedures are tested and documented, and that the patching timeline is predictable enough to fit within change management windows.
The Anatomy of a Rolling Update
All three platforms use the same fundamental strategy: update one node at a time while keeping the rest of the cluster operational. The difference is in the orchestration, the automation maturity, and the failure handling.
Rolling Update: Universal Pattern (All Platforms)
====================================================
Cluster: 10 nodes, each running ~50 VMs
Time ----->
Node 01 [DRAIN======][UPDATE=====][REJOIN]
Node 02 [DRAIN======][UPDATE=====][REJOIN]
Node 03 [DRAIN======][UPDATE=====][REJOIN]
...
Node 10 ... [DRAIN===][UPDATE===][REJOIN]
Each node cycle:
+--------+ +---------+ +-------+ +--------+ +-------+ +--------+
| CORDON |-->| DRAIN |-->| PATCH |-->| REBOOT |-->| VERIFY|-->|UNCORDON|
| (mark | | (migrate| | (apply| | (new | | (health| | (allow |
| unsched) | VMs off)| | update) | kernel)| | checks) | sched)|
+--------+ +---------+ +-------+ +--------+ +-------+ +--------+
~5 sec ~10-30 min ~2-5 min ~5-10 min ~2-5 min ~5 secThe 90% rule: Zero-downtime rolling updates require at least one node's worth of spare capacity at all times. A 10-node cluster running at 100% capacity cannot drain a node because there is nowhere to put the VMs. The operational recommendation is to size clusters at N+1 (one node worth of headroom) or N+2 for additional safety during patching. This applies equally to all three platforms.
VMware vSphere Lifecycle Manager (vLCM) -- Baseline
This is what the team does today. vLCM uses a desired image model. The administrator defines a cluster-level image consisting of an ESXi base version, vendor add-ons (HPE custom image), firmware, and driver packages. vLCM compares each host's current state against the desired image and flags non-compliant hosts.
Step-by-Step: Per-Node Remediation Cycle
VMware vLCM: Rolling Remediation (per host)
=============================================
vCenter / vLCM DRS ESXi Host vSAN
| | | |
1. Compliance check: | | |
scan 10 hosts vs | | |
desired image ~30-60s | | |
| | | |
2. Pre-check: can DRS | | |
evacuate host? Enough | | |
capacity? HA admission? | | |
~1-2 min | | |
| | | |
3. Enter MAINTENANCE MODE | | |
+---------------------> | | |
| 4. DRS evacuates host: | |
| - vMotion plan for ~50 VMs | |
| - Respects affinity rules | |
| - 4-8 concurrent vMotions | |
| - Avg 30 sec/VM | |
| ~5-15 min total | |
| | | |
| | 5. vSAN: "ensure |
| | accessibility" mode |
| | (no data migration, |
| | relies on mirrors |
| | on other hosts) |
| | FTT temporarily reduced |
| | ~30 sec (metadata only) |
| | | |
6. Stage update: upload | | |
ESXi image + firmware | | |
~2-5 min | | |
| | | |
| | 7. Reboot: POST, firmware |
| | flash, ESXi boot, |
| | VMkernel init |
| | ~5-15 min |
| | | |
| | 8. Reconnect to vCenter |
| | 9. vSAN resync:
| | stale objects
| | resynchronize
| | ~1-5 min
| | | |
10. Exit maintenance mode | | |
DRS rebalances (auto) | | |
| | | |
=== REPEAT for host-02 through host-10 ===| Step | Action | Duration |
|---|---|---|
| 3 | Enter maintenance mode | ~5 sec |
| 4 | DRS evacuation (vMotion ~50 VMs, 4-8 parallel) | 5-15 min |
| 5 | vSAN "ensure accessibility" | ~30 sec |
| 6 | Stage update, upload image | 2-5 min |
| 7 | Reboot + POST + firmware + ESXi boot | 5-15 min |
| 8-9 | Reconnect + vSAN resync | 1-5 min |
| 10 | Exit maintenance + DRS rebalance | 1-2 min |
| Total per node | ~15-45 min | |
| Total 10-node cluster | ~2.5-7.5 hours |
Canary strategy: Manual. The administrator can remediate a single host first, run validation tests, then proceed with the rest. There is no automated gate that says "canary passed, proceed." This process relies on the administrator's discipline.
OVE: MachineConfigOperator (MCO) Rolling Update
OpenShift uses the MachineConfigOperator (MCO) to manage the RHCOS operating system on every node. RHCOS is an immutable, image-based OS -- there is no yum update. Instead, the MCO delivers a new OS image via rpm-ostree, and the node reboots into the new image. The previous image is retained on disk for rollback.
Key concepts:
- MachineConfigPool (MCP): Groups nodes sharing the same config. Default pools:
masterandworker. Custom pools (e.g.,worker-canary) enable canary strategies. - MachineConfig (MC): Declares desired OS state -- kernel args, systemd units, files, rpm-ostree image reference.
- maxUnavailable: Controls how many nodes update simultaneously (default: 1).
Step-by-Step: Per-Node Update Cycle
OVE: MCO Rolling Update (per worker node)
============================================
CVO / MCO MCO Daemon kubelet / KubeVirt Ceph/ODF
(on target node)
| | | |
1. CVO downloads release | | |
payload, updates control | | |
plane first (~15-30 min | | |
for 3 masters sequentially) | | |
| | | |
2. MCO selects worker-01 | | |
(respects maxUnavailable=1) | | |
| | | |
3. CORDON worker-01 | | |
(kubectl cordon) ~5 sec | | |
| | | |
4. DRAIN worker-01: | | |
- Evicts all pods | | |
- virt-launcher eviction | | |
triggers LIVE MIGRATION | | |
- Pre-copy memory transfer | | |
- 2-5 concurrent migrations | 5. Per VM: |
- Respects PDBs | - virt-handler |
| | initiates live |
| | migration |
| | - ~30-90 sec/VM |
| | - VM sees <500 ms |
| | pause (converged) |
| | Total: ~10-30 min |
| | | |
6. Worker-01 fully drained | | |
| | | |
| 7. MCO daemon applies | |
| new MachineConfig: | |
| - rpm-ostree rebase | |
| to new RHCOS image | |
| - Write config files | |
| ~1-3 min | |
| | | |
| 8. REBOOT: | |
| - BIOS POST | |
| - RHCOS boots from | |
| new ostree commit | |
| - kubelet, CRI-O, | |
| OVS/OVN, Ceph OSDs |
| all restart | |
| ~5-10 min | |
| | | |
| 9. MCO daemon reports | |
| config matches desired | |
| Node condition: Ready | |
| ~1-2 min | |
| | | |
10. UNCORDON worker-01 | | |
~5 sec | | |
| | | |
=== REPEAT for worker-02 through worker-10 ===| Step | Action | Duration |
|---|---|---|
| 3 | Cordon node | ~5 sec |
| 4-5 | Drain: live migrate ~50 VMs (2-5 parallel) | 10-30 min |
| 7 | Apply MachineConfig (rpm-ostree rebase) | 1-3 min |
| 8 | Reboot (POST + RHCOS boot + all services) | 5-10 min |
| 9 | Node health verification | 1-2 min |
| 10 | Uncordon | ~5 sec |
| Total per node | ~20-45 min | |
| Total 10 workers | ~3.5-7.5 hours | |
| Total with control plane | ~4.5-10 hours |
Canary Strategy (OVE) -- Native
OVE supports a declarative canary strategy through custom MachineConfigPools, significantly more mature than VMware's or Azure Local's manual approaches:
OVE Canary with Custom MachineConfigPool
==========================================
1. Create a canary pool:
kind: MachineConfigPool
metadata:
name: worker-canary
spec:
machineConfigSelector:
matchExpressions:
- key: machineconfiguration.openshift.io/role
operator: In
values: [worker, worker-canary]
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker-canary: ""
maxUnavailable: 1
2. Label one node:
kubectl label node worker-07 node-role.kubernetes.io/worker-canary=""
3. MCO updates canary pool FIRST (1 node only).
Main "worker" pool is paused or waits.
4. Validate canary (24-48h monitoring).
5. If OK -> unpause worker pool -> MCO proceeds with remaining 9 nodes.
If FAIL -> pause pool, debug, fix. Main worker pool never touched.The canary MachineConfigPool approach means a failed update affects exactly one node. The remaining nine nodes continue operating with the previous OS version indefinitely until the issue is resolved.
Azure Local: Cluster-Aware Updating (CAU)
Azure Local uses Cluster-Aware Updating (CAU), a Windows Server Failover Clustering feature, combined with Azure Update Manager for orchestration. Updates come in two flavors: solution updates (full platform: OS + firmware + drivers) and quality updates (monthly Windows cumulative updates via WSUS).
Step-by-Step: Per-Node Update Cycle
Azure Local: CAU Rolling Update (per node)
=============================================
Azure Update Mgr / Failover Cluster Hyper-V Host S2D
CAU Orchestrator Manager
| | | |
1. Download update | | |
package to nodes | | |
~5-15 min | | |
| | | |
2. SUSPEND node-01: | | |
Suspend-ClusterNode | | |
+------------------------->| | |
| 3. Failover Cluster | |
| drains node-01: | |
| - Live Migrate all VMs | |
| - Quick Migration fallback | |
| ~5-20 min for ~50 VMs | |
| | | |
| | 4. S2D: node-01 |
| | storage temporarily |
| | unavailable. Marks |
| | disks "retired" (temp).|
| | Mirror copies on |
| | remaining nodes serve |
| | all I/O. NO rebuild |
| | triggered (<30 min). |
| | ~30 sec |
| | | |
5. Install updates: | | |
Windows Update agent | | |
applies CU + firmware | | |
~5-15 min | | |
| | | |
6. REBOOT: | | |
- Windows boot + "Configuring| | |
updates" phase | | |
- Cluster Service rejoins | | |
- S2D reconnects + resync | | |
~5-15 min (boot) | 7. S2D resync: |
| stale data resync |
| | ~1-5 min |
| | | |
8. RESUME node-01: | | |
Resume-ClusterNode | | |
~5 sec | | |
| | | |
=== REPEAT for node-02 through node-10 ===| Step | Action | Duration |
|---|---|---|
| 2 | Suspend node | ~5 sec |
| 3 | Live migrate ~50 VMs | 5-20 min |
| 4 | S2D marks node absent | ~30 sec |
| 5 | Install updates (CU + firmware) | 5-15 min |
| 6-7 | Reboot + S2D resync | 6-20 min |
| 8 | Resume node | ~5 sec |
| Total per node | ~20-60 min | |
| Total 10-node cluster | ~3.5-10 hours |
Canary strategy: Manual, similar to VMware. Use -NodeOrder in Invoke-CauRun to target a single node first, then validate before continuing.
Swisscom ESC (Managed Service)
Swisscom Enterprise Service Cloud is a managed platform. The customer does not execute patching -- Swisscom's operations team handles platform updates. The customer controls the maintenance window preference and go/no-go approval, but not patch selection, node ordering, canary strategy, or rollback decisions. For a Tier-1 financial institution, the SLA terms and the escalation path during a failed update matter more than the technical details of how Swisscom patches internally.
Cross-Domain Impact: Storage During Node Reboot
Storage Behavior During Node Reboot
======================================
Ceph/ODF (OVE):
================================================================
- Ceph MON detects OSDs down after 10 sec (osd_heartbeat_grace)
- PGs with replicas on rebooting node go DEGRADED (2/3 replicas)
- I/O continues: reads/writes served by remaining 2 replicas
- NO RECOVERY for 10 min (osd_recovery_delay_start, configurable)
- Node returns within 10 min: OSDs rejoin, backfill only the delta
- Node stays down >10 min: Ceph re-replicates to restore 3x
CRITICAL setting: ceph osd set noout
Prevents OSD removal from CRUSH map during planned maintenance.
Without this flag, Ceph rebalances data after 10 min -- flooding
the storage network and degrading VM performance mid-update.
Always set before patching; unset after last node completes.
Performance impact: read I/O normal, write I/O <5% degradation.
Risk: second node failure = some PGs at 1 replica (CRITICAL).
S2D (Azure Local):
================================================================
- S2D marks copies on rebooting node as "stale"
- I/O continues: reads/writes served by remaining 2 copies
- NO REBUILD for 30 min (default repair delay, automatic)
- Node returns within 30 min: resynchronizes delta only
- Node stays down >30 min: S2D re-mirrors to another node
Performance impact: <5% degradation for typical workloads.
Risk: same -- second node failure = some data at 1 copy.Key difference: Ceph's noout flag is a manual but well-understood operation. S2D's repair delay is automatic and less visible. Both achieve the same goal: prevent unnecessary data movement during short, planned absences.
Cross-Domain Impact: Networking During Node Reboot
VMs are already migrated off the node before reboot, so there is no direct VM traffic disruption. The impact is on the network control plane during the node's departure and return.
Networking During Node Reboot
================================
OVN-Kubernetes (OVE):
================================================================
Phase 1 -- VMs migrating off (before reboot):
- Each VM migration moves the OVN logical port to target chassis
- Source ovn-controller removes OVS flows for that port
- Target ovn-controller installs OVS flows
- Per-migration network reprogramming: ~100-500 ms
- VM traffic gap during switchover: <200 ms
Phase 2 -- Node rebooting (no VMs present):
- ovn-controller stops, OVS stops
- OVN Southbound DB marks chassis inactive
- Other nodes remove stale tunnel endpoints
- No VM impact
Phase 3 -- Node returns:
- OVS starts with empty flow tables
- ovn-controller reconnects to Southbound DB
- Re-establishes GENEVE tunnels, programs baseline flows
- Full convergence: ~30-60 sec
- Race condition risk: VMs scheduled before flows converge may
see ~1-5 sec connectivity gap. Conservative: wait 60 sec
after uncordon before migrating VMs back.
Hyper-V VMSwitch (Azure Local):
================================================================
Phase 1 -- VMs migrating off:
- Target node creates VMSwitch port, registers MAC
- Switchover: source port removed, target port active
- Gratuitous ARP updates physical switch MAC tables
- Network switchover: <1 sec per VM
Phase 2 & 3 -- Node reboot and return:
- VMSwitch goes offline, SDN Host Agent disconnects
- On return: VMSwitch initializes, VFP policies re-downloaded
- Full convergence: ~15-30 sec
- Simpler model: each node's VLAN-tagged traffic is independentKey difference: OVN manages a distributed logical network (broader blast radius for control-plane bugs, more powerful policy model). Azure Local's VMSwitch operates largely independently per node (simpler, but less sophisticated network virtualization without the optional SDN Network Controller).
Rollback: What Happens When the Update Breaks Something
Rollback Capabilities Comparison
===================================
VMware vLCM:
- Remediate to previous baseline: define old ESXi image in vLCM,
re-remediate the host. Same rolling process. ~15-45 min/node.
- Break glass: reinstall ESXi from ISO via iLO. ~30-60 min + manual config.
- No previous image kept on host. Admin must know which version to revert to.
OVE (MCO):
- rpm-ostree rollback (instant, built-in):
RHCOS keeps TWO ostree deployments on disk at all times.
"rpm-ostree rollback" + reboot -> node runs previous OS in ~10-15 min.
+--------------------------------------------------+
| /boot/ostree/deploy/ |
| +-- rhcos-4.16.8 (current) |
| +-- rhcos-4.16.7 (previous, always present) |
+--------------------------------------------------+
- Pause MCP: oc patch mcp worker -p '{"spec":{"paused":true}}'
Remaining nodes do NOT update. Debug at leisure.
- GitOps rollback: git revert the MachineConfig -> MCO re-applies.
Azure Local (CAU):
- Uninstall CU: wusa /uninstall /kb:NNNNNNN + reboot. ~15-30 min.
Limitation: not all updates are uninstallable.
- Stop-CauRun: remaining nodes stay at previous version.
- Break glass: reimage node with deployment tool. ~1-2 hours.| Capability | VMware vLCM | OVE MCO | Azure Local CAU |
|---|---|---|---|
| Instant rollback on the node | No | Yes (rpm-ostree) | Partial (wusa /uninstall) |
| Previous image always on disk | No | Yes (two ostree deployments) | No (10-day rollback window) |
| Stop rolling update mid-cluster | Yes (cancel remediation) | Yes (pause MCP) | Yes (Stop-CauRun) |
| Canary-gated rollout | Manual | Native (custom MCP) | Manual |
| GitOps rollback | N/A | Yes (git revert -> MCO re-applies) | N/A |
| Rollback duration (per node) | 15-45 min | 10-15 min | 15-30 min |
Complete Timeline: 10-Node Cluster Update
Full Cluster Update Timeline
===============================
Assumptions: 10 worker nodes, ~50 VMs/node, maxUnavailable=1,
avg VM memory 8 GiB, 25 Gbps migration, OS-only (no firmware).
Time (hours)
0 1 2 3 4 5 6 7 8
|--------|--------|--------|--------|--------|--------|--------|--------|
VMware vLCM (optimistic 15 min/node):
[N01][N02][N03][N04][N05][N06][N07][N08][N09][N10]
|=============================================| ~2.5 hrs
VMware vLCM (realistic 30 min/node):
[==N01==][==N02==][==N03==][==N04==][==N05==][==N06==][==N07==][==N08==][==N09==][==N10==]
|===========================================================================| ~5 hrs
OVE MCO (realistic 35 min/node, workers only):
[===N01===][===N02===][===N03===]...[===N10===]
|====================================================================================| ~6 hrs
Add ~45-90 min for 3 control plane nodes -> ~4.5-7.5 hrs total
Azure Local CAU (realistic 40 min/node):
[====N01====][====N02====][====N03====]...[====N10====]
|================================================================================================| ~6.5 hrs
Per-node breakdown (realistic):
VMware (30 min): |drain(10m)|stage(3m)|reboot(8m) |resync(4m)|rebal(5m)|
OVE (35 min): |drain(15m) |ostree(2m)|reboot(8m) |verify(5m)|uncordon|
Azure (40 min): |drain(12m) |install(8m)|reboot(12m) |resync(5m)|resume|Why Azure Local Takes Longer
Windows Server has a longer reboot cycle than RHCOS or ESXi: the "Configuring Windows updates" phase adds 3-8 minutes post-reboot, and the Failover Cluster Service must rejoin and validate quorum (+1-3 minutes). Over 10 nodes, this adds 50-100 minutes to the total window. For a Tier-1 institution with strict maintenance windows (e.g., Saturday 02:00-06:00), a 10-node Azure Local update may not complete in 4 hours.
Parallel Updates: maxUnavailable > 1
All platforms support updating multiple nodes simultaneously. With maxUnavailable=2, total time is roughly halved, but two nodes are unavailable simultaneously. Recommendation for a 10-node cluster with 3-way replication: keep maxUnavailable=1. With only 10 nodes, two nodes down simultaneously means some objects may have replicas on both absent nodes, leaving only 1 copy available. For larger clusters (20+ nodes), maxUnavailable=2 becomes safer because replica overlap probability drops.
Three-Platform Comparison Summary
Rolling Update Capability Matrix
===================================
VMware vLCM OVE (MCO) Azure Local (CAU)
============ ============ ==================
Automation level High High High
(vLCM auto) (MCO fully auto) (CAU fully auto)
Canary support Manual Native (MCP) Manual
OS update model Image-based Image-based Cumulative
(ESXi image) (ostree commit) update (CU)
Rollback: instant No Yes (ostree) Partial (wusa)
Previous image on disk No Yes (always) No
GitOps integration No Yes No
Live migration maturity vMotion KubeVirt LM Hyper-V LM
(15+ years) (since ~2020) (15+ years)
Concurrent migrations 4-8/host 2-5/node Bandwidth-limited
(default) (configurable)
Reboot (OS only) 5-10 min 5-8 min 5-15 min
Per-node cycle 15-45 min 20-45 min 20-60 min
10-node total 2.5-7.5 hrs 4.5-10 hrs 3.5-10 hrs
(incl ctrl plane)
Storage during reboot vSAN: degraded Ceph: degraded S2D: degraded
no rebuild <60m no rebuild w/ no rebuild <30m
noout flag
Network during reboot vDS: no impact OVN: flow VMSwitch:
(VMs migrated) reprogram on no impact
return ~30-60s (VMs migrated)
Firmware in same cycle Yes (vLCM+iLO) No (separate) Yes (CAU+vendor)Risk Matrix
| Risk | Probability | Impact | Mitigation |
|---|---|---|---|
| VM migration fails (convergence timeout) | Low-Medium | Single VM rebooted | Pre-check dirty page rates. Configure migration timeout/bandwidth. For databases: schedule brief freeze or use post-copy. |
| Node fails to boot after update | Low | One node out; VMs safe on other nodes | rpm-ostree rollback (OVE), ESXi reinstall (VMware), wusa /uninstall (Azure Local). Worst case: reimage. |
| Storage resync overload | Low | Performance degradation | Keep noout set (Ceph) during entire window. Do not start next node until resync completes. |
| Second node failure during update | Very Low | Data at risk (some objects at 1 replica) | maxUnavailable=1. Verify cluster health before each node. |
| Update breaks application | Medium | App downtime despite VM uptime | Canary strategy. Test in non-prod. App-level health checks before proceeding. |
| Maintenance window exceeded | Medium | Change management violation | Use conservative time estimates. Build 50% buffer (6h estimate = book 8-9h). |
| Anti-affinity violation during drain | Low-Medium | Co-located VMs that should be separated | DRS/PDB/Failover Cluster anti-affinity may block drain. Review rules before update. |
Operational Checklist
Before the Update
| # | Check | VMware | OVE | Azure Local |
|---|---|---|---|---|
| 1 | Cluster health | vCenter > vSAN Health | oc get nodes, ceph status |
Get-ClusterNode, Get-StorageHealthReport |
| 2 | N+1 spare capacity | DRS capacity view | oc adm top nodes |
Get-VM \| Measure-Object |
| 3 | Backup critical VMs | Veeam/Commvault | Velero + VolumeSnapshot | Azure Backup / DPM |
| 4 | Review release notes | VMware KB | OpenShift errata, BZ list | Microsoft KB |
| 5 | Test in non-prod | Dev/test cluster | Non-prod OCP cluster | Dev/test Azure Local |
| 6 | Set storage flags | N/A (vSAN auto) | ceph osd set noout |
N/A (S2D auto delay) |
| 7 | Change management | CAB approval | CAB approval | CAB approval |
During the Update: What to Monitor
| Signal | VMware | OVE | Azure Local |
|---|---|---|---|
| Migration progress | vCenter Tasks tab | oc get vmim |
Failover Cluster Manager |
| Storage health | vSAN Health > Resync | ceph status |
Get-StorageJob |
| Node status | vCenter Hosts tab | oc get nodes |
Get-ClusterNode |
| VM health | VM list (heartbeat) | oc get vmi -A |
Get-VM -ClusterObject |
After the Update
Verify: all nodes running new version, all VMs healthy, storage fully clean (no degraded objects), remove noout flag (OVE), spot-check network connectivity, compare IOPS/latency to pre-update baseline.
Practical Recommendations
-
Patching is where OVE's relative immaturity matters most. vMotion has 15+ years of production hardening for maintenance-mode evacuation. KubeVirt live migration has been production-capable since ~2020 but has a shorter track record. The PoC must include a full rolling update with realistic VM workloads (including a memory-dirty database VM) to validate migration reliability under drain conditions.
-
OVE has the best rollback story. The rpm-ostree dual-deployment model means every node always has a known-good previous OS image on disk, ready to boot in minutes. This is architecturally superior to both VMware (requires re-remediation or reinstall) and Azure Local (Windows Update rollback is not always possible). For a regulated environment where rollback capability is a compliance requirement, this is a significant advantage.
-
OVE has the best canary strategy. Custom MachineConfigPools with label-based node selection provide a declarative, automated canary gate. VMware and Azure Local require manual single-node updates and manual validation.
-
Azure Local has the longest update cycle. Windows CU installation and post-reboot configuration add 5-10 minutes per node vs. RHCOS or ESXi. Over 10 nodes, this adds 50-100 minutes. A 4-hour maintenance window may not be sufficient.
-
Storage flags are critical. On OVE with Ceph, always
ceph osd set nooutbefore starting and unset after completion. Forgetting this causes Ceph to rebalance data after 10 minutes, flooding the storage network during the update. Automate this in the pre-update runbook. -
Size for N+1 permanently. All three platforms require at least one node's worth of spare capacity for zero-downtime rolling updates. A 10-node cluster at 95% utilization cannot be patched without VM downtime.
-
Firmware updates are the wildcard. OS patches are predictable (5-10 min reboot). Firmware updates (BIOS, NIC, storage controller) add 10-30 minutes per node. vLCM and Azure Local CAU integrate firmware; OVE does not (separate vendor lifecycle tools required). Raise this gap with Red Hat during the PoC.
-
Swisscom ESC eliminates this entire scenario. Patching becomes Swisscom's responsibility. The team gains operational simplicity but loses control over timing, rollback, and canary strategies.
Discussion Guide
-
For OVE: What is the measured live migration success rate during a rolling MCO update with 50 VMs per node? How many migrations fail and require pod restart (VM cold reboot)? What is the maximum VM memory size that reliably migrates within the MCO drain timeout?
-
For OVE: How does the MCO handle a node that fails to boot after an ostree rebase? Does it auto-trigger
rpm-ostree rollback, or is manual intervention required? -
For Azure Local: What is the realistic total update time for a 10-node cluster applying a monthly CU? Does it fit within a 4-hour maintenance window?
-
For Azure Local: Can the cluster operate indefinitely with some nodes at the old version and some at the new if a CAU run is interrupted mid-cluster?
-
For all platforms: What is the procedure for VMs that cannot be live migrated (GPU passthrough, local disk)? Does the drain block? Is there automated fallback to cold migration?
-
For Swisscom ESC: What is the SLA for patch application after a critical CVE (CVSSv3 >= 9.0)? What is the escalation path if the maintenance window causes unexpected disruption?
-
For change management: Given 3-10 hour update windows, should rolling updates be classified as standard changes (pre-approved) or normal changes (CAB each time)?