VMware vSphere / ESXi -- Current Baseline

Why This Matters

Understanding what you are migrating from is as important as understanding what you are migrating to. Every feature comparison, every migration risk assessment, and every PoC acceptance criterion implicitly references the VMware baseline. If the team cannot articulate exactly how DRS decides where to place a VM, or how vMotion iterates memory pages, or why VMFS uses on-disk locking, then evaluating whether OVE, Azure Local, or Swisscom ESC can replicate those behaviors becomes guesswork rather than engineering.

This document captures the VMware vSphere / ESXi platform at the depth required to make precise comparisons. It covers architecture, execution model, resource management, high availability, live migration, storage, networking, licensing, and operational tooling. The final section -- "What to Preserve vs. What to Leave Behind" -- is the most important: it forces the team to distinguish between capabilities that are genuinely load-bearing for the business and capabilities that are simply muscle memory from years of VMware operation.

The current estate runs 5,000+ VMs across a production VMware environment. Everything described here is the implicit contract that any replacement platform must honor, renegotiate, or explicitly retire.

Concepts

1. ESXi Architecture

Overview

VMware ESXi is a Type-1 bare-metal hypervisor built on a proprietary microkernel called the VMkernel. Unlike KVM (which is a Linux kernel module) or Hyper-V (which runs a thin hypervisor beneath a "root partition" running Windows), ESXi's VMkernel is a purpose-built, monolithic kernel that directly manages CPU scheduling, memory management, storage I/O, and network I/O. There is no general-purpose operating system underneath.

  ESXi Host Architecture

  +---------------------------------------------------------------+
  |  Management Plane                                             |
  |  +------------------+  +------------------+  +-------------+  |
  |  |  hostd (C++)     |  |  vpxa (vCenter   |  |  DCUI       |  |
  |  |  (local mgmt     |  |   agent)         |  |  (Direct    |  |
  |  |   API daemon)    |  |                  |  |   Console)  |  |
  |  +------------------+  +------------------+  +-------------+  |
  |  +------------------+  +------------------+  +-------------+  |
  |  |  CIM Agents      |  |  sfcbd (CIM      |  |  SNMP       |  |
  |  |  (hardware       |  |   broker)        |  |  Agent      |  |
  |  |   monitoring)    |  |                  |  |             |  |
  |  +------------------+  +------------------+  +-------------+  |
  |                                                               |
  |  All above run as "Userworlds" -- ESXi user-space processes   |
  +---------------------------------------------------------------+
  |  VM Execution Layer                                           |
  |  +-------------+  +-------------+  +-------------+            |
  |  | VMX Process |  | VMX Process |  | VMX Process |   ...      |
  |  | (VM 1)      |  | (VM 2)      |  | (VM 3)      |            |
  |  |  +-------+  |  |  +-------+  |  |  +-------+  |            |
  |  |  | VMM   |  |  |  | VMM   |  |  |  | VMM   |  |            |
  |  |  +-------+  |  |  +-------+  |  |  +-------+  |            |
  |  +-------------+  +-------------+  +-------------+            |
  +---------------------------------------------------------------+
  |  VMkernel                                                     |
  |  +----------+ +--------+ +--------+ +--------+ +-----------+  |
  |  | CPU      | | Memory | | VMFS / | | vSwitch| | VMkernel  |  |
  |  | Scheduler| | Mgr    | | NFS    | | Net    | | TCP/IP    |  |
  |  +----------+ +--------+ +--------+ +--------+ +-----------+  |
  |  +----------------------------------------------------------+ |
  |  |  Device Drivers (native .vib drivers)                    | |
  |  +----------------------------------------------------------+ |
  +---------------------------------------------------------------+
  |  Hardware: CPU (VT-x/AMD-V), RAM, NICs, HBAs, Local Disk     |
  +---------------------------------------------------------------+

VMkernel

The VMkernel is not Linux. It is a custom, preemptive, multithreaded kernel optimized exclusively for running virtual machines. Key characteristics:

• No general-purpose syscall interface. User-space processes ("userworlds") communicate with the VMkernel through a private, VMware-internal system call table. There is no POSIX compatibility.
• Custom CPU scheduler. The VMkernel scheduler is a proportional-share scheduler with support for shares, reservations, and limits. It operates on "worlds" -- the VMkernel term for schedulable entities (threads). Each vCPU of a VM is a world. Management daemons are also worlds.
• Custom memory manager. Manages host physical memory with ballooning, transparent page sharing (TPS), compression, and swap tiers. Memory overcommitment is a deliberate design choice, not an accident.
• Custom network stack. The VMkernel has its own TCP/IP stack for management traffic, vMotion, iSCSI, NFS, and vSAN traffic. This stack is separate from any guest networking.
• Custom storage stack. Includes VMFS (a clustered filesystem), NFS client, iSCSI initiator, and the Pluggable Storage Architecture (PSA) for multipathing.

The VMkernel boots from a small boot bank (a USB device, SD card, or local disk partition) containing a compressed image. The boot bank holds two copies of the ESXi image for rollback capability. The VMkernel loads entirely into RAM at boot time -- the boot device is not accessed during normal operation.

Userworlds

Userworlds are ESXi's equivalent of user-space processes. They run in unprivileged mode (Ring 3) and use VMkernel system calls for I/O and resource access. Key userworlds include:

DCUI (Direct Console User Interface)

The DCUI is the text-mode console accessible via the physical server's keyboard/monitor or via IPMI/iLO/iDRAC remote console. It provides minimal configuration capabilities:

• Set management network IP address (vmk0)
• Configure DNS and hostname
• Reset root password
• Restart management agents
• Enable/disable ESXi Shell and SSH
• View system logs

The DCUI is intentionally limited. It exists for emergency recovery and initial bootstrapping, not day-to-day administration. In a locked-down production environment, SSH access to the ESXi shell is typically disabled, and all management flows through vCenter.

CIM Agents

The Common Information Model (CIM) layer is how ESXi exposes hardware health information. Server vendors (Dell, HPE, Lenovo, Cisco) ship CIM provider VIBs (vSphere Installation Bundles) that register with sfcbd. These providers talk to the server's BMC (Baseboard Management Controller) and expose standardized CIM classes for:

• Physical disk health (SMART data, predicted failure)
• DIMM ECC error counts
• Power supply status and power draw
• Fan RPM and thermal zones
• RAID controller battery/capacitor status

vCenter consumes this data to show hardware health in the UI and trigger alarms. Without the correct vendor CIM VIBs installed, the ESXi host reports "Unknown" for hardware health. This is a common operational gap -- during upgrades, teams forget to reinstall the vendor-specific VIBs.

2. vCenter Server Architecture

Overview

vCenter Server is the centralized management plane for a vSphere environment. Since vSphere 6.7, the only supported deployment is the vCenter Server Appliance (VCSA) -- a pre-built Photon OS Linux VM running all vCenter services. The Windows-based vCenter installation was deprecated and removed.

  vCenter Server Appliance (VCSA) Internal Architecture

  +---------------------------------------------------------------+
  |  VCSA (Photon OS Linux VM)                                    |
  |                                                               |
  |  +----------------------------------------------------------+ |
  |  |  vSphere Client (HTML5 UI)                               | |
  |  |  Served by envoy reverse proxy on port 443               | |
  |  +----------------------------------------------------------+ |
  |                                                               |
  |  +------------------+  +------------------+  +-------------+  |
  |  |  vpxd            |  |  SSO / STS       |  |  Inventory  |  |
  |  |  (vCenter core   |  |  (VMware         |  |  Service    |  |
  |  |   service,       |  |   Identity       |  |             |  |
  |  |   written in     |  |   Provider)      |  |             |  |
  |  |   Java + C++)    |  |                  |  |             |  |
  |  +------------------+  +------------------+  +-------------+  |
  |                                                               |
  |  +------------------+  +------------------+  +-------------+  |
  |  |  PostgreSQL      |  |  vPostgres       |  |  Content    |  |
  |  |  (embedded DB    |  |  (VMware-        |  |  Library    |  |
  |  |   for vpxd,      |  |   modified       |  |  Service    |  |
  |  |   SSO, etc.)     |  |   PostgreSQL)    |  |             |  |
  |  +------------------+  +------------------+  +-------------+  |
  |                                                               |
  |  +------------------+  +------------------+  +-------------+  |
  |  |  VAMI            |  |  Certificate     |  |  Lookup     |  |
  |  |  (Appliance      |  |  Authority       |  |  Service    |  |
  |  |   Management)    |  |  (VMCA)          |  |             |  |
  |  +------------------+  +------------------+  +-------------+  |
  +---------------------------------------------------------------+

Key Components

vpxd is the core vCenter daemon. It maintains the inventory of all ESXi hosts, VMs, clusters, networks, and datastores. It communicates with each ESXi host via the vpxa agent. vpxd makes all cluster-level decisions: DRS placement, HA failover, storage DRS. The vpxd process stores its state in the embedded PostgreSQL database (historically called the "VCDB").

SSO (Single Sign-On) / STS (Security Token Service) handles authentication. The Platform Services Controller (PSC) was a separate component in vSphere 6.x that housed SSO, the Certificate Authority (VMCA), and the Lookup Service. Starting with vSphere 7.0, the PSC is fully embedded in the VCSA -- there is no external PSC deployment option. SSO supports:

• Local (vsphere.local) identity source
• Active Directory over LDAP
• Active Directory via Integrated Windows Authentication (IWA)
• OIDC identity providers (added in vSphere 8.0)

VMCA (VMware Certificate Authority) issues certificates for ESXi hosts, vCenter services, and solution users. By default, VMCA is the root CA. Enterprise deployments typically configure VMCA as a subordinate CA under the organization's PKI root, or replace all certificates with enterprise-issued certs. Certificate management is one of the most operationally painful aspects of vSphere -- expired VMCA certificates can take down the entire management plane.

Inventory Hierarchy

  vCenter Inventory Object Model

  vCenter Server
  |
  +-- Datacenter
  |   |
  |   +-- Cluster (group of ESXi hosts with shared DRS/HA policies)
  |   |   |
  |   |   +-- ESXi Host 1
  |   |   |   +-- VM-A
  |   |   |   +-- VM-B
  |   |   |
  |   |   +-- ESXi Host 2
  |   |   |   +-- VM-C
  |   |   |
  |   |   +-- Resource Pool (optional nesting)
  |   |       +-- RP-Production
  |   |       |   +-- VM-D
  |   |       +-- RP-Development
  |   |           +-- VM-E
  |   |
  |   +-- Standalone ESXi Host (not in a cluster -- no DRS/HA)
  |   |
  |   +-- Datastore Cluster (optional, for Storage DRS)
  |   |   +-- Datastore-1 (VMFS)
  |   |   +-- Datastore-2 (VMFS)
  |   |
  |   +-- VM Folder (organizational hierarchy)
  |   |   +-- Folder: Finance
  |   |   |   +-- VM-F
  |   |   +-- Folder: HR
  |   |       +-- VM-G
  |   |
  |   +-- Network Folder
  |       +-- Distributed vSwitch
  |       +-- Port Groups
  |
  +-- Datacenter 2
      +-- ...

Key points about this hierarchy:

• Permissions inherit downward. A role assigned at the Datacenter level propagates to all clusters, hosts, and VMs within it. This inheritance model is central to multi-tenancy.
• VMs have two locations: a compute location (which host/cluster/resource pool they run on) and an inventory location (which VM folder they appear in). These are independent. A VM can sit in the "Finance" folder while running on any host in any cluster.
• Tags and Custom Attributes provide a secondary, non-hierarchical classification system. Many organizations use tags for lifecycle management (environment, application owner, backup policy) since the folder hierarchy can only represent one dimension.

vCenter Availability

vCenter itself is a single VM. If it fails, existing VMs continue running (ESXi hosts operate autonomously), but no management operations are possible -- no vMotion, no DRS rebalancing, no VM power-on to a new host. Options for vCenter availability:

• vCenter HA (Active-Passive-Witness): A native feature that maintains an active VCSA, a passive replica, and a witness node. Failover is automatic but takes 5-10 minutes.
• vSphere Replication / SRM: Site-level DR for the VCSA itself.
• Backup and restore: Most organizations simply back up the VCSA and accept the recovery time.

The fact that the management plane is a single VM is a significant architectural difference from Kubernetes-based platforms (OVE, Azure Local) where the control plane is typically a replicated set of processes across multiple nodes.

3. VM Execution Model on ESXi

The VMX Process

Every powered-on VM on ESXi is represented by a VMX process -- a userworld that acts as the VM's "container" in the ESXi process table. The VMX process handles:

• Device emulation for virtual hardware that is not backed by hardware passthrough (virtual CD-ROM, virtual floppy, legacy virtual NIC emulation for older guests)
• The VM's connection to the vSphere Client console (VNC/BLAST protocol)
• Snapshot management coordination
• Communication with hostd for lifecycle operations

  VM Execution Model on ESXi

  +-----------------------------------------------------------+
  |  Guest OS (Windows, Linux, etc.)                          |
  |  Running in VMX non-root mode (Ring 0/3 within VM)        |
  +-----------------------------------------------------------+
        |           |              |              |
        | vCPU      | Virtual      | Virtual      | Virtual
        | execution | Disk I/O     | NIC I/O      | Device
        |           |              |              | I/O
        v           v              v              v
  +-----------------------------------------------------------+
  |  VMM (Virtual Machine Monitor)                            |
  |  One per VM. Runs in VMkernel context (VMX root mode).    |
  |  Handles VM exits, EPT management, interrupt injection.   |
  +-----------------------------------------------------------+
        |                          |              |
        | CPU world               | I/O world     | MKS world
        | scheduling              | scheduling    | (console)
        v                          v              v
  +-----------------------------------------------------------+
  |  VMX Process (Userworld)                                  |
  |  PID visible in esxtop. One per powered-on VM.            |
  |  Handles device emulation, snapshot ops, console.         |
  +-----------------------------------------------------------+
        |
        v
  +-----------------------------------------------------------+
  |  VMkernel                                                 |
  |  CPU scheduler, memory manager, storage stack, net stack  |
  +-----------------------------------------------------------+
        |
        v
  +-----------------------------------------------------------+
  |  Physical Hardware                                        |
  +-----------------------------------------------------------+

The VMM (Virtual Machine Monitor)

The VMM is a per-VM kernel-mode component that runs within VMkernel context. It is the code that actually handles VM exits. When a guest vCPU executes a trapped instruction:

1. The CPU triggers a VM exit and saves guest state to the VMCS.
2. Control returns to the VMkernel CPU scheduler.
3. The VMkernel dispatches to the VMM for that specific VM.
4. The VMM inspects the exit reason, emulates the instruction or services the event.
5. The VMM prepares modified guest state and issues VMRESUME.

The VMM also manages:

• EPT page tables for the VM's guest-physical-to-host-physical mapping
• Virtual APIC and interrupt injection (posted interrupts on modern CPUs)
• Virtual timer (TSC offsetting, deadline timers)
• Instruction emulation for para-virtualized operations (vmware.tools backdoor port)

vmknic (VMkernel Network Interfaces)

VMkernel NICs (vmk0, vmk1, etc.) are the VMkernel's own IP endpoints. They are not guest-visible -- they belong to the hypervisor itself. Each vmknic is associated with a port group on a vSwitch or Distributed vSwitch and is tagged for a specific traffic type:

The VMkernel TCP/IP stack can be split into multiple named stacks (default stack, vMotion stack, provisioning stack) to provide traffic isolation at the network stack level, not just at the vmknic/VLAN level.

4. Resource Management

DRS (Distributed Resource Scheduler)

DRS is the cluster-level workload balancer. It runs inside vpxd (on vCenter) and makes two types of decisions:

1. Initial placement: When a VM is powered on, DRS selects the optimal host in the cluster based on available CPU and memory resources, affinity/anti-affinity rules, and host compatibility.
2. Load balancing (rebalancing): Every 5 minutes (configurable), DRS evaluates the resource utilization imbalance across hosts and generates a set of vMotion recommendations to reduce the imbalance.

DRS Algorithm Internals

DRS calculates a cluster imbalance metric using the standard deviation of host resource utilization. For each host, it computes a "load" score based on:

• CPU active time (not just CPU demand -- this accounts for actual usage vs. allocation)
• Memory active usage (measured via statistical sampling of page access bits)

DRS then generates a set of candidate VM migrations and evaluates each for its migration cost (vMotion overhead, memory size, current activity level) vs. benefit (reduction in cluster imbalance). Migrations are only recommended when the benefit exceeds the cost by a threshold determined by the DRS aggressiveness setting:

DRS operates in two modes:

• Fully automated: DRS executes vMotion migrations without human approval.
• Manual / Partially automated: DRS generates recommendations that an admin must approve.

DRS and Affinity Rules

DRS honors three types of rules:

• VM-VM affinity ("should" or "must" run together): Keeps VMs on the same host for low-latency inter-VM communication. "Should" rules are best-effort; "must" rules are hard constraints that DRS will not violate.
• VM-VM anti-affinity ("should" or "must" run separately): Spreads VMs across hosts for resilience. Critical for HA -- ensures both nodes of a database cluster do not land on the same host.
• VM-Host affinity/anti-affinity: Restricts VMs to specific hosts or host groups. Used for licensing compliance (e.g., Oracle database VMs must only run on hosts with Oracle-licensed cores) or hardware requirements (e.g., GPU-equipped hosts).

Resource Pools

Resource pools are hierarchical containers within a cluster that partition CPU and memory resources among groups of VMs. They are the primary mechanism for multi-tenant resource governance in vSphere.

  Resource Pool Hierarchy

  Cluster (total: 256 GHz CPU, 2 TB RAM)
  |
  +-- RP: Production (Reservation: 180 GHz, 1.5 TB)
  |   |
  |   +-- RP: Tier-1 (Reservation: 120 GHz, 1 TB)
  |   |   +-- DB Servers
  |   |   +-- Trading Systems
  |   |
  |   +-- RP: Tier-2 (Reservation: 60 GHz, 500 GB)
  |       +-- Web Servers
  |       +-- Batch Jobs
  |
  +-- RP: Development (Limit: 64 GHz, 512 GB, no reservation)
  |   +-- Developer VMs
  |
  +-- RP: Infrastructure (Reservation: 12 GHz, 48 GB)
      +-- vCenter, AD, DNS, Monitoring

Shares, Reservations, and Limits

These three controls operate on both CPU and memory at the VM or resource pool level:

A common misconfiguration is using limits when shares would be appropriate, or failing to set reservations for latency-sensitive workloads. The result is either wasted capacity (limits on idle hosts) or unpredictable performance under load (no reservations for critical VMs).

5. HA and Fault Tolerance

vSphere HA (High Availability)

vSphere HA provides automatic VM restart when an ESXi host fails. It does not prevent downtime -- it detects the failure and restarts the affected VMs on surviving hosts. Typical recovery time is 1-5 minutes depending on VM boot time.

HA Architecture

When HA is enabled on a cluster, all hosts participate in an HA master election. One host becomes the HA master; the rest are HA subordinates.

• The HA master monitors all subordinate hosts via network heartbeats (sent every second over the management network) and datastore heartbeats (written to a file on shared storage every 5 seconds).
• If the master stops receiving network heartbeats from a subordinate, it checks the datastore heartbeat to distinguish between a host failure and a network partition.

  HA Failure Detection Decision Matrix

                           Datastore       Datastore
                           Heartbeat OK    Heartbeat MISSING
  +-------------------+------------------+-------------------+
  | Network Heartbeat | Host is alive,   | Host has FAILED.  |
  | MISSING           | network isolated | Restart VMs on    |
  |                   | (network         | surviving hosts.  |
  |                   | partition).      |                   |
  |                   | Apply isolation  |                   |
  |                   | response policy. |                   |
  +-------------------+------------------+-------------------+
  | Network Heartbeat | Normal           | Storage issue,    |
  | OK                | operation.       | not host failure. |
  |                   |                  | Investigate.      |
  +-------------------+------------------+-------------------+

Isolation response determines what happens to VMs on a host that is network-isolated but still running:

• Leave powered on (default): The VMs keep running. If the host is truly isolated, they become unreachable but are not disrupted.
• Power off: The isolated host powers off its VMs, allowing the HA master to restart them elsewhere.
• Shut down: The isolated host attempts a graceful guest OS shutdown, then powers off.

The choice depends on whether the VMs can tolerate being unreachable vs. whether split-brain (the same VM running on two hosts) is a worse outcome.

Admission Control

Admission control prevents the cluster from becoming so overloaded that there is not enough spare capacity to restart VMs after a host failure. It answers: "If we lose N hosts, can the surviving hosts run all the VMs that need to be restarted?"

Admission control policies:

Admission control can block VM power-on operations. If powering on a VM would consume failover capacity, vCenter will refuse the operation. This is correct behavior but frustrates operators who do not understand the math. A common (dangerous) reaction is to disable admission control rather than right-sizing reservations.

VM Restart Priority

When HA restarts VMs, it does so in priority order:

1. Priority 1 (Highest): Infrastructure VMs (vCenter, AD, DNS)
2. Priority 2: Critical business VMs
3. Priority 3 (Default): Standard VMs
4. Priority 4 (Lowest): Development/test VMs

VMs with higher priority are restarted first. If the surviving hosts do not have capacity for all VMs, lower-priority VMs may remain powered off.

vSphere Fault Tolerance (FT)

FT provides zero-downtime protection for a single VM by maintaining a live, lock-step secondary copy on another host. If the primary host fails, the secondary instantly takes over with no VM reboot and no TCP connection loss.

FT Internals

FT uses VMware vLockstep technology:

1. The primary VM executes on Host A. Every non-deterministic event (interrupts, I/O completions, timer ticks, RDTSC results) is recorded.
2. These events are streamed over a dedicated FT logging network (vmknic dedicated to FT traffic) to Host B, where a secondary VM replays them in lockstep.
3. The secondary VM's output (disk writes, network packets) is suppressed -- only the primary's output reaches the external world.
4. If Host A fails, the secondary on Host B is already at exactly the same execution state. It "goes live" by unsuppressing its output. External clients see a brief pause (typically <1 second) but no TCP connection reset.

Constraints:

• FT is limited to VMs with up to 8 vCPUs (vSphere 7.x+). Earlier versions were limited to 1 vCPU.
• FT VMs cannot use snapshots, Storage vMotion, or hot-add CPU/RAM.
• The FT logging network requires at least 10 Gbps dedicated bandwidth per FT-protected VM, and latency must be under 10ms between hosts.
• FT doubles the resource consumption: two hosts each dedicate the full CPU and memory allocation for the protected VM.

Because of these constraints, FT is used sparingly -- typically only for a handful of VMs where even 60 seconds of downtime is unacceptable (e.g., a session broker, a quorum witness, a real-time monitoring gateway).

6. vMotion Internals

vMotion is live migration of a running VM from one ESXi host to another with near-zero downtime. It is the single most important operational capability in a VMware environment -- it enables maintenance windows, load balancing (DRS), and failure avoidance.

vMotion Process Flow

  vMotion Pre-Copy Memory Migration

  Source Host                                  Destination Host
  ===========                                  ================

  Phase 0: Pre-checks
  - Verify CPU compatibility (EVC baseline)
  - Verify network reachability (vMotion vmknic)
  - Verify shared storage access
  - Reserve resources on destination

  Phase 1: Full memory copy (Background)
  +----------------------------------+
  | Copy all VM memory pages ------> |  Copy all pages
  | VM continues running on source   |  to destination
  | ~seconds to minutes depending    |  (network bandwidth
  |  on VM memory size               |   limited)
  +----------------------------------+

  Phase 2: Iterative pre-copy (Background, multiple rounds)
  +----------------------------------+
  | Round 1: Copy dirty pages -----> |  Pages modified during
  | VM continues running on source   |  Phase 1 are re-sent
  +----------------------------------+
  +----------------------------------+
  | Round 2: Copy dirty pages -----> |  Pages modified during
  | (fewer pages each round)         |  Round 1 are re-sent
  +----------------------------------+
  +----------------------------------+
  | Round N: Dirty set small enough  |  Convergence: the
  | for final switchover             |  working set delta is
  +----------------------------------+  now tiny

  Phase 3: Stun-and-switch (Brief VM pause)
  +----------------------------------+
  | STUN VM on source                |
  | Copy final dirty pages --------> |  Last few MB/KB
  | Transfer device state ---------->|  (virtual device
  | Update network (RARP) --------->|   registers, VMCS)
  | UNSTUN VM on destination         |
  +----------------------------------+
  |                                  |  VM now executes here
  |  VM no longer active on source   |
  +----------------------------------+

  Typical stun time: 10-200ms
  Worst case for very active VMs: up to 1 second

Pre-Copy Memory Iteration -- Why It Works

The key insight is that most VMs have a relatively small "hot" working set -- the pages being actively modified at any point in time. Even a VM with 256 GB of allocated RAM might only be dirtying 50-100 MB per second. The iterative pre-copy exploits this:

• Phase 1 copies 256 GB. Takes ~25 seconds on a 10 Gbps vMotion link.
• Round 1 of Phase 2 copies the ~1.25 GB dirtied during those 25 seconds. Takes ~0.125 seconds.
• Round 2 copies the ~6 MB dirtied during that 0.125 seconds. Takes microseconds.
• Convergence reached. The final stun-and-switch copies only a few KB.

When pre-copy fails to converge: If the VM is dirtying memory faster than the vMotion link can transfer it (e.g., an in-memory database doing bulk inserts at >10 Gbps equivalent rate), the dirty set never shrinks. vMotion will either:

• Time out after a configurable number of iterations (default: 100 seconds total).
• In vSphere 7.0+, switch to stun during page send -- briefly pause the VM to let the copy catch up, then resume. This increases stun time but guarantees completion.

vMotion Network Requirements

• Dedicated network: vMotion traffic should have its own vmknic on a dedicated VLAN or at minimum a high-priority QoS marking. Sharing the management network with vMotion risks saturating management connectivity during migrations.
• Bandwidth: 10 Gbps minimum for production use. 25 Gbps recommended for large-memory VMs. vSphere supports multi-NIC vMotion (multiple vmknics for vMotion traffic) for aggregated bandwidth.
• Latency: The vMotion vmknics on source and destination must have <150ms RTT (for long-distance vMotion). For standard vMotion within a datacenter, sub-millisecond latency is expected.
• MTU: Jumbo frames (MTU 9000) significantly improve vMotion throughput by reducing per-packet overhead.

Enhanced vMotion Compatibility (EVC)

vMotion requires that the destination host's CPU is compatible with the source host's CPU -- the VM must not encounter instructions on the destination that were available on the source but are now missing. EVC solves this by masking CPUID features across all hosts in a cluster to a common baseline:

• EVC is configured at the cluster level.
• All hosts in the cluster present the same CPUID feature set to VMs, regardless of their actual CPU model.
• This allows mixed-generation hosts (e.g., Cascade Lake alongside Ice Lake) to participate in vMotion, at the cost of disabling newer CPU features.

Storage vMotion

Storage vMotion migrates a VM's virtual disk files from one datastore to another while the VM is running. Unlike compute vMotion, it does not require shared storage -- it works by:

1. Creating a destination disk on the target datastore.
2. Mirroring all writes to both source and destination disks.
3. Copying existing data from source to destination in the background.
4. When the copy is complete, switching the VM's disk pointer to the destination and removing the source.

Storage vMotion and compute vMotion can be performed simultaneously (called cross-host, cross-datastore vMotion), enabling full VM relocation without shared storage.

Cross-vSwitch vMotion

When the source and destination hosts use different virtual switches (e.g., different Distributed vSwitch versions, or Standard vSwitch vs. Distributed vSwitch), vMotion must handle the network switchover. The VM's virtual NIC is disconnected from the source port group and connected to the equivalent port group on the destination. The VM sends a RARP (Reverse ARP) broadcast to update the physical network's MAC-to-port mapping, ensuring traffic reaches the VM at its new location.

7. ESXi Storage Stack

Architecture

The ESXi storage stack is a layered architecture that abstracts physical storage into the logical constructs that VMs consume.

  ESXi Storage Stack Layers

  +----------------------------------------------------------+
  |  VM Layer                                                |
  |  +---------------------------------------------------+   |
  |  |  Virtual Disks (.vmdk files on datastores)        |   |
  |  |  SCSI Controller: pvscsi, LSI Logic, AHCI        |   |
  |  +---------------------------------------------------+   |
  +----------------------------------------------------------+
  |  Filesystem / NFS Layer                                  |
  |  +------------------------+  +------------------------+  |
  |  |  VMFS-6                |  |  NFS Client            |  |
  |  |  (block-based          |  |  (NFS v3 or v4.1)     |  |
  |  |   clustered FS)        |  |  (file-based access)   |  |
  |  +------------------------+  +------------------------+  |
  +----------------------------------------------------------+
  |  VMkernel Storage APIs                                   |
  |  +---------------------------------------------------+   |
  |  |  VAAI (vStorage APIs for Array Integration)       |   |
  |  |  Offloads: full copy, block zeroing, ATS lock     |   |
  |  +---------------------------------------------------+   |
  +----------------------------------------------------------+
  |  Pluggable Storage Architecture (PSA)                    |
  |  +---------------------------------------------------+   |
  |  |  NMP (Native Multipathing Plugin)                 |   |
  |  |  +---------------------------------------------+ |   |
  |  |  |  SATP (Storage Array Type Plugin)            | |   |
  |  |  |  Identifies array type, handles failover     | |   |
  |  |  |  Examples: VMW_SATP_ALUA, VMW_SATP_DEFAULT   | |   |
  |  |  +---------------------------------------------+ |   |
  |  |  +---------------------------------------------+ |   |
  |  |  |  PSP (Path Selection Plugin)                 | |   |
  |  |  |  Chooses active path for I/O                 | |   |
  |  |  |  Examples: VMW_PSP_RR (Round Robin),         | |   |
  |  |  |           VMW_PSP_FIXED, VMW_PSP_MRU         | |   |
  |  |  +---------------------------------------------+ |   |
  |  +---------------------------------------------------+   |
  +----------------------------------------------------------+
  |  Device Drivers (HBA drivers: lpfc, qla2xxx, bnx2fc)    |
  +----------------------------------------------------------+
  |  Physical: FC HBAs, iSCSI NICs, Ethernet (NFS/iSCSI)    |
  +----------------------------------------------------------+

VMFS (Virtual Machine File System)

VMFS is VMware's proprietary clustered filesystem designed for shared block storage (FC SAN, iSCSI SAN). Key characteristics:

• Clustered access: Multiple ESXi hosts mount the same VMFS datastore simultaneously. On-disk locking (using SCSI-3 Persistent Reservations or Atomic Test and Set / ATS) prevents concurrent modification of the same file metadata.
• VMFS-6 (current) uses ATS-based locking instead of the coarser SCSI-3 reservations used in VMFS-5. ATS locks only the specific metadata region being modified, not the entire LUN.
• Large file support: VMFS-6 supports files up to 62 TB -- critical for large VMDK files.
• Sub-block allocation: For small files (VM configuration files, log files), VMFS allocates 1 KB sub-blocks within 1 MB file blocks to avoid wasting space.
• VMFS heap: An in-memory structure that tracks open files, locks, and resource ownership. On hosts with many VMs or many open VMDK files, VMFS heap exhaustion is a known issue that causes VM I/O stalls.

VMDK file types:

• Thick Provision Lazy Zeroed: Space is allocated at creation time but not zeroed until first write. Faster creation, but the first write to each block incurs a zeroing penalty.
• Thick Provision Eager Zeroed: Space is allocated and zeroed at creation time. Required for FT, clustered VMDK, and best for latency-sensitive workloads.
• Thin Provision: Space is allocated on demand as the guest writes data. Enables overprovisioning but requires monitoring to prevent datastore out-of-space conditions.

NFS Datastores

ESXi can mount NFS exports as datastores. NFS datastores store VMDK files as regular files on the NFS server.

• NFS v3: Stateless protocol. No client-side caching in ESXi. All I/O goes to the NFS server. Simpler to troubleshoot.
• NFS v4.1: Stateful protocol with session semantics, Kerberos authentication support, and multipathing (session trunking). Introduced in vSphere 6.0 but only reached production maturity in later releases.

NFS eliminates the VMFS locking complexity but introduces NFS server availability as a dependency. NFS performance depends heavily on the NFS server's capability (NetApp, Pure Storage, etc.).

PSA, NMP, SATP, and PSP

The Pluggable Storage Architecture (PSA) is the framework that manages multipathing and path failover for block storage.

NMP (Native Multipathing Plugin) is VMware's built-in implementation of PSA. It uses two sub-plugins:

SATP (Storage Array Type Plugin) identifies the storage array behind a LUN and determines how to handle path failover. ESXi ships with SATPs for all major arrays:

• VMW_SATP_ALUA: For arrays that support Asymmetric Logical Unit Access (active/optimized and active/non-optimized paths).
• VMW_SATP_DEFAULT_AA: For Active/Active arrays where all paths are equally preferred.
• VMW_SATP_DEFAULT_AP: For Active/Passive arrays with explicit failover.
• Vendor-specific SATPs: VMW_SATP_SYMM (Dell EMC Symmetrix), VMW_SATP_CX (Dell EMC VNX/Unity), etc.

PSP (Path Selection Plugin) determines which available path to use for each I/O operation:

• VMW_PSP_RR (Round Robin): Distributes I/O across all active paths. Configurable with bytes or I/O count before switching paths. Best for performance.
• VMW_PSP_FIXED: Always uses a preferred path. Fails over to an alternate if preferred is unavailable, then switches back when preferred recovers.
• VMW_PSP_MRU (Most Recently Used): Uses the current path until it fails, then switches to another. Does not fail back. Avoids path thrashing on A/P arrays.

Third-party multipathing plugins (Dell EMC PowerPath/VE, Veritas DMP) can replace NMP entirely, providing array-specific optimizations.

VAAI (vStorage APIs for Array Integration) offloads certain storage operations from ESXi to the array hardware:

• Full Copy (XCOPY): Cloning a VMDK or deploying from template is done by the array, not by ESXi reading and writing data over the SAN.
• Block Zeroing (WRITE SAME): Zeroing thick-provisioned disks is done by the array.
• ATS (Atomic Test and Set): VMFS metadata locking is done atomically by the array, replacing coarse SCSI-3 reservations.
• Thin Provisioning Stale Read Guard: Prevents reading stale data from thin-provisioned LUNs after space reclamation.

8. ESXi Networking Stack

Standard vSwitch

A standard vSwitch is a software Layer 2 switch local to a single ESXi host. It connects:

• VM virtual NICs (port groups for VM traffic)
• VMkernel NICs (port groups for management, vMotion, storage, vSAN)
• Physical uplink NICs (connection to the physical network)

  Standard vSwitch Architecture

  +------------------------------------------------------------------+
  |  ESXi Host                                                       |
  |                                                                  |
  |  VMs:  [VM-A]   [VM-B]   [VM-C]     VMkernel: [vmk0]  [vmk1]   |
  |          |         |        |                    |        |      |
  |          |  (Port Group:    |        (Port Group: Mgmt)  (PG:   |
  |          |   "Production")  |                            vMotion)|
  |          |         |        |                    |        |      |
  |  +-------+----+----+-------+--------------------+--------+---+  |
  |  |                                                           |  |
  |  |              vSwitch0 (Standard vSwitch)                  |  |
  |  |                                                           |  |
  |  +--------+---------+---------+------------------------------+  |
  |           |         |         |                                  |
  |        [vmnic0]  [vmnic1]  [vmnic2]   (Physical uplinks)        |
  +-----------|---------|---------|----------------------------------+
              |         |         |
       Physical Network (ToR switches)

Standard vSwitch configuration is per-host. Every host must be configured identically if VMs are to vMotion between them. This is a major operational burden at scale -- with 50+ hosts, ensuring consistent standard vSwitch configuration across all hosts is error-prone.

Distributed vSwitch (VDS / dvSwitch)

A Distributed vSwitch is centrally managed from vCenter and spans all hosts in a datacenter (or a subset). Configuration is defined once at the VDS level and pushed to all member hosts automatically.

  Distributed vSwitch Architecture

  +-------------------------------------------------------+
  |  vCenter (VDS Configuration Authority)                 |
  |  Defines: Port groups, VLAN policies, teaming,        |
  |           traffic shaping, security policies           |
  +---+-------------------+-------------------+------------+
      |                   |                   |
  +---v---+           +---v---+           +---v---+
  | Host 1|           | Host 2|           | Host 3|
  | Proxy |           | Proxy |           | Proxy |
  | Switch|           | Switch|           | Switch|
  |       |           |       |           |       |
  | vmnic0|           | vmnic0|           | vmnic0|
  | vmnic1|           | vmnic1|           | vmnic1|
  +-------+           +-------+           +-------+

Each host runs a proxy switch -- the local data-plane instantiation of the VDS. The proxy switch does the actual packet forwarding. vCenter only manages the control plane (configuration). If vCenter fails, the proxy switches continue operating with their last-known configuration.

VDS features beyond standard vSwitch:

• Network I/O Control (NIOC): Bandwidth reservation and shares for different traffic types (VM, vMotion, vSAN, iSCSI, management) on the same physical NIC.
• Port mirroring (SPAN): Mirror traffic from VDS ports to a monitoring port or remote destination.
• NetFlow / IPFIX: Export flow records for network monitoring and troubleshooting.
• Private VLANs (PVLANs): Micro-segmentation within a VLAN.
• Link Aggregation (LACP): Standards-based NIC teaming with the physical switch.
• Traffic filtering and marking: ACLs and DSCP marking at the virtual switch level.
• Health check: Automatic detection of VLAN and MTU mismatches between the VDS and the physical switch.

Port Groups

Port groups define the network policy for connected endpoints:

• VLAN ID (or VLAN trunk range)
• Security policy: Promiscuous mode, MAC address changes, forged transmits
• Teaming and failover: Active/standby uplinks, load balancing algorithm (route based on originating virtual port, route based on source MAC hash, route based on IP hash, route based on physical NIC load)
• Traffic shaping: Ingress and egress bandwidth limits

NSX and Overlay Networking

VMware NSX (now NSX-T / NSX 4.x) extends the networking stack with:

• VXLAN / Geneve overlay networks that decouple logical networks from the physical fabric.
• Distributed firewalling (micro-segmentation) at the vNIC level.
• Distributed routing (Tier-0 and Tier-1 gateways running on each host).
• Load balancing, VPN, and NAT services.

NSX is a separate licensed product with significant additional cost. If the current environment uses NSX, this creates additional migration complexity because NSX policies, distributed firewall rules, and overlay networks must be replicated or replaced on the target platform.

9. Licensing Model Post-Broadcom

The Broadcom Shift (2024)

Broadcom's acquisition of VMware (completed November 2023) fundamentally changed the licensing model:

Before Broadcom:

• Per-socket licensing. vSphere Standard or Enterprise Plus licensed per physical CPU socket.
• Perpetual licenses with annual SnS (Support and Subscription) renewals.
• A la carte add-ons: vSAN, NSX, vRealize Operations, SRM purchased separately.
• Predictable, understood cost model.

After Broadcom:

• Subscription-only. All perpetual licenses terminated. Existing customers must convert to subscriptions at renewal.
• Per-core licensing. Pricing shifted from per-socket to per-physical-core, with a minimum of 16 cores per CPU. A dual-socket server with 64 cores per socket pays for 128 cores, not 2 sockets.
• Bundled SKUs only. Individual products are no longer sold separately. Two main bundles:
  • VMware Cloud Foundation (VCF): Full stack -- vSphere, vSAN, NSX, Aria. Required if you want NSX or vSAN.
  • VMware vSphere Foundation (VVF): vSphere + vCenter + Aria Standard. Lower tier, but missing vSAN and NSX.
• Elimination of lower tiers. vSphere Standard and Essentials Plus (SMB-oriented) were discontinued. Smaller customers faced dramatic cost increases.
• Partner program restructuring. Many VMware partners were terminated. Remaining partners have less pricing flexibility.

Cost Impact Analysis

For a financial enterprise running 5,000+ VMs:

This licensing change is the primary business driver for the IaaS platform evaluation. The technical capabilities of vSphere are well-understood and mature. The question is whether the cost/risk profile under Broadcom's model justifies remaining on the platform or migrating to an alternative.

10. Operational Tooling

PowerCLI

VMware PowerCLI is a PowerShell module that provides cmdlets for automating vSphere operations. It is the primary automation tool for VMware administrators.

Key capabilities:

• Get-VM, Set-VM, New-VM, Remove-VM -- full VM lifecycle
• Move-VM -- trigger vMotion (compute and/or storage)
• Get-VMHost, Set-VMHost -- host configuration
• Get-Cluster, Set-Cluster -- cluster settings (DRS, HA)
• Get-VDSwitch, Get-VDPortgroup -- distributed switch management
• Pipeline-oriented: Get-VM -Location "Cluster-Prod" | Where-Object {$_.PowerState -eq "PoweredOff"} | Remove-VM

PowerCLI connects to vCenter (or directly to an ESXi host) via the vSphere API (SOAP or REST). All operations available in the UI are available via PowerCLI.

Migration consideration: PowerCLI scripts represent significant institutional automation. Every PowerCLI script is a functional specification of an operational workflow that must be replicated on the target platform (using kubectl/oc for OVE, PowerShell + Azure modules for Azure Local, or the ESC provider's API).

ESXCLI

ESXCLI is the command-line interface available directly on the ESXi host (via SSH or ESXi Shell). It provides low-level host management:

• esxcli storage -- storage subsystem (claim rules, NMP, multipath)
• esxcli network -- network subsystem (vSwitch, vmknic, firewall)
• esxcli system -- host settings (hostname, time, syslog, coredump)
• esxcli software -- VIB installation and management
• esxcli vm -- VM operations (list, kill)

ESXCLI is used for troubleshooting and for operations that are not exposed in vCenter, such as configuring NMP path policies, adjusting storage claim rules, or installing third-party VIBs.

MOB (Managed Object Browser)

The MOB is a web-based interface to the vSphere API object model, accessible at https://<vcenter-or-host>/mob. It exposes the raw managed objects, properties, and methods of the vSphere SDK. It is a debugging and exploration tool -- not meant for production operations, but invaluable for understanding the API structure, checking object IDs, and invoking methods manually during troubleshooting.

vRealize Suite / Aria

VMware's operations management suite was rebranded from "vRealize" to "Aria" in 2023:

Migration consideration: If the environment uses Aria Operations for capacity planning and right-sizing, those reports are the best source of actual VM utilization data for migration planning. Export the data before decommissioning the VMware environment.

esxtop / resxtop

esxtop is ESXi's real-time performance monitoring tool, similar to top on Linux but for the VMkernel. It shows:

• CPU: Per-world (per-vCPU, per-VMX) CPU utilization, ready time, co-stop time, system time
• Memory: Per-VM memory consumption, ballooning, swapping, compression, TPS savings
• Disk: Per-device IOPS, latency (DAVG = array latency, KAVG = kernel latency, GAVG = guest-perceived latency)
• Network: Per-vmnic and per-vNIC throughput

esxtop in batch mode (esxtop -b) is the standard method for collecting performance data during troubleshooting. resxtop is the remote equivalent that can be run from a management station.

Key esxtop metrics every VMware admin watches:

What to Preserve vs. What to Leave Behind

This section is the most important part of this document. Not every VMware feature is worth replicating. Some features are genuinely load-bearing for the business. Others are VMware-specific constructs that should be deliberately abandoned during migration.

Must Preserve (Business-Critical Capabilities)

Nice to Have (Operationally Convenient but Not Critical)

Leave Behind (VMware-Specific Lock-In)

Key Takeaways

1. ESXi is a custom microkernel, not Linux. The VMkernel, userworlds, VMFS, and PSA are all proprietary systems with no direct equivalents. Migration is not a "lift and shift" -- it is a platform re-architecture.

2. vCenter is a single-VM management plane. This is an architectural weakness. Kubernetes-based platforms (OVE) distribute the control plane across multiple nodes. Azure Local uses a distributed Azure Arc management model. Both are more resilient than a single VCSA.

3. DRS is more sophisticated than anything available on other platforms today. The proportional-share scheduler with cluster-wide imbalance optimization and multi-constraint rule evaluation is mature and well-tested. OVE's descheduler and Azure Local's placement logic are simpler. Teams accustomed to DRS "just handling it" will need to adjust expectations and supplement with manual operations or custom tooling.

4. vMotion works because of the pre-copy algorithm and dedicated network infrastructure. Any replacement must provide equivalent live migration with acceptable stun times. The PoC must test migration of large-memory, high-change-rate VMs -- not just idle test VMs.

5. The storage stack (VMFS + PSA + VAAI) is deeply integrated and array-aware. Replacing this means adopting a different storage philosophy: software-defined storage (Ceph for OVE, S2D for Azure Local) or continuing with traditional arrays via different multipathing stacks (Linux DM-Multipath, Microsoft MPIO).

6. The licensing change is the forcing function. VMware's technical capabilities are not in question. The per-core subscription model, forced bundling, and reduced partner ecosystem create a cost and vendor-risk profile that justifies the enormous effort of migrating 5,000+ VMs.

7. Institutional knowledge in PowerCLI scripts, esxtop troubleshooting, and ESXCLI procedures is not portable. The team will need to build equivalent operational muscle in the target platform's tooling. Budget for this learning curve.

8. Not everything deserves to be replicated. VMFS, VIBs, the vSphere SOAP API, and the vCenter folder hierarchy are VMware-specific constructs. Migrating away from them is an opportunity to adopt more modern, standards-based alternatives.

Discussion Guide

These questions should be directed inward -- at your own team -- not at vendors. The answers will shape migration requirements, risk assessment, and the PoC acceptance criteria.

Current Usage Patterns

1. How many VMs use resource reservations today? If the answer is "almost none," then the team has been relying on implicit resource availability rather than explicit governance. The replacement platform must either replicate the same implicit behavior or the team must adopt explicit resource requests.

2. What DRS aggressiveness level is configured on each cluster, and has it ever been changed? If DRS is set to Level 3 (default) and has never been tuned, the team may not actually depend on advanced DRS behavior and can accept a simpler placement model.

3. How many VMs are protected by Fault Tolerance (FT)? If the answer is zero or single digits, FT is not a migration requirement. Those VMs should be moved to application-level HA instead.

4. What is the current vMotion success rate and average stun time? Pull this from vCenter events or Aria Operations. This establishes the baseline that the replacement platform must match.

5. How many PowerCLI scripts exist, and who maintains them? Each script is a functional requirement. Inventory them, classify them (provisioning, reporting, compliance, lifecycle), and prioritize which ones must be re-implemented first on the target platform.

Storage Dependencies

6. What storage arrays are in use, and what VAAI primitives do they support? If the migration moves to software-defined storage (Ceph, S2D), VAAI offloads are no longer relevant. If the migration retains the existing arrays, the replacement's multipathing stack must support them.

7. Are there VMFS datastores with known heap pressure or lock contention issues? These pain points will not exist on the replacement platform (since VMFS is abandoned), but they may indicate workloads with unusually high metadata I/O that need attention.

8. How is thin provisioning managed today? If datastores are heavily overprovisioned, the team needs a monitoring plan on the replacement platform to prevent the same problem with different storage.

Networking Dependencies

9. Is NSX deployed? If yes, the migration scope is dramatically larger. Every NSX distributed firewall rule, logical switch, and load balancer must be replicated or replaced. If no, the migration scope is limited to VLAN-based networking, which all candidates handle.

10. Are Distributed vSwitches in use, or standard vSwitches? If the environment uses standard vSwitches, the team has already been managing per-host network configuration manually -- the operational model on the replacement platform will feel familiar. If VDS is in use, the team expects centralized network management and should confirm the replacement offers equivalent central control.

11. What is the current vMotion network bandwidth and MTU configuration? This directly determines whether the replacement platform's live migration will perform comparably. If vMotion runs on a shared 1 GbE management network with MTU 1500, performance expectations are low and easy to exceed. If vMotion runs on dedicated 25 GbE with jumbo frames, the replacement must match that infrastructure investment.

Operational Knowledge

12. When was the last time the team troubleshot a performance issue using esxtop? If the answer is "weekly," the team has deep VMware troubleshooting skills that will not transfer. Budget training time for the replacement platform's equivalent tools (esxtop -> oc adm top + Prometheus for OVE, esxtop -> Performance Monitor + WAC for Azure Local).

13. How are ESXi host patches applied today? If the answer is "vSphere Lifecycle Manager (vLCM) with image-based management," the team expects automated, cluster-aware rolling updates. The replacement must offer equivalent capabilities (OpenShift's MachineConfigOperator for OVE, Azure Update Manager for Azure Local).

14. What CIM providers / hardware monitoring agents are installed? These are vendor-specific and will need to be replaced with the target platform's hardware monitoring approach (Redfish/IPMI polling, vendor management tools, or OS-native hardware agents).

Risk and Governance

15. Which VMs are running unsupported guest OS versions? VMs running Windows Server 2008, RHEL 6, or other end-of-life guests may have compatibility issues on newer hypervisors. Identify them now and plan for OS upgrades or extended support arrangements.

16. Are there VMs with physical device passthrough (GPUs, FPGAs, USB devices)? Passthrough VMs require special handling during migration. The replacement platform must support the same passthrough capabilities with compatible IOMMU configuration.

17. What is the current RBAC model in vCenter, and who are the permission holders? Map the vCenter roles and permissions to the replacement platform's RBAC model. Identify any permissions that do not have a direct equivalent and plan for how to handle the gap.

18. Are there external systems that integrate with the vSphere API? Backup software (Veeam, Commvault), monitoring tools (SolarWinds, Datadog), CMDB synchronization, and custom portals all integrate via the vSphere API. Each integration point is a migration dependency that must be addressed.

19. What is the contractual situation with Broadcom? When does the current vSphere license/subscription expire? What is the renewal pricing? This sets the hard deadline for migration and the cost baseline that alternatives must beat.

20. Does the team have experience with Kubernetes or Linux administration? If the answer is no, OVE requires the largest skill-set pivot. Azure Local requires Windows Server and Azure skills. ESC requires the least change since it remains on VMware. The team's existing skills should influence the candidate ranking.