Scenario: Site B Takes Over After Site A Failure

Why This Scenario Matters

Disaster recovery is not a feature checkbox -- it is an operational capability that must be designed, built, tested, and maintained. For a Tier-1 financial institution under FINMA regulation, the ability to fail over 5,000+ VMs from a primary site to a secondary site within defined RTO/RPO targets is not optional. FINMA Circular 2023/1 (Operational Risks and Resilience) requires documented BCM plans, defined recovery objectives per critical system, and annual DR testing with auditable results.

This scenario traces a complete DR failover from steady-state replication through failure detection, storage promotion, VM restart, application validation, and eventual failback. The same disaster -- total loss of Site A (power, cooling, or network) -- is executed on all three candidate platforms so the evaluation team can compare the operational experience, automation maturity, and timing at each step.

The scenario is deliberately severe: Site A is completely lost. This is not a single-host failure (handled by HA/Failover Clustering) or a single-disk failure (handled by replication within the cluster). This is the scenario that justifies the cost of a second site, a second set of hardware, and a replication infrastructure running 24/7.

Replication Topology: Steady State

Before any failure occurs, data must already be replicating. The three platforms use fundamentally different replication mechanisms.

Steady-State Replication Topology (All Platforms)
=====================================================

  Site A (Primary)                              Site B (Secondary)
  Basel DC1                                     Zurich DC2
  ============================                  ============================

  OVE (OpenShift Virtualization Engine):
  +----------------------------+                +----------------------------+
  | OCP Cluster A              |                | OCP Cluster B              |
  | ODF / Ceph Cluster A       |                | ODF / Ceph Cluster B       |
  | 9 worker nodes, 108 OSDs   |  RBD mirror   | 9 worker nodes, 108 OSDs   |
  | rbd-mirror daemon ---------|--snapshot----->| rbd-mirror daemon          |
  | (primary for pool "prod")  |  based async  | (secondary for pool "prod")|
  +----------------------------+                +----------------------------+

  Azure Local:
  +----------------------------+                +----------------------------+
  | WSFC Cluster A             |                | WSFC Cluster B             |
  | S2D Pool, 4 nodes          |  Storage      | S2D Pool, 4 nodes          |
  | CSV Volumes (VMs)          |--Replica----->| CSV Volumes (replicas)     |
  | Source volumes (R/W)       |  async/sync   | Dest volumes (read-only)   |
  +----------------------------+                +----------------------------+

  Swisscom ESC:
  +----------------------------+                +----------------------------+
  | ESC Twin-DC Site A         |                | ESC Twin-DC Site B         |
  | VMware vSphere + NSX       |  SAN-level    | VMware vSphere + NSX       |
  | Dell PowerMax / PowerStore |--replication->| Dell PowerMax / PowerStore |
  | (primary)                  |  + SnapMirror | (secondary)                |
  +----------------------------+                +----------------------------+
        |                                              |
        +------ RHACM Hub / Azure / Swisscom NOC ------+
               (Orchestration / Monitoring)

  Inter-site: 2x 100 GbE dark fiber (dedicated replication VLAN 50)
  DNS: GSLB with weighted records, TTL=60s for DR-critical entries
  Routing: BGP AS 65100 (Site A) / AS 65200 (Site B), BFD-enabled

Replication Mechanisms in Detail

Phase 1: Failure Detection

The clock starts when Site A fails. How quickly each platform detects the failure determines how soon the failover process can begin.

Failure Detection Timeline
============================

Event: Site A loses all power at T=0

  T+0s     Site A goes dark. All nodes, switches, storage offline.
           No graceful shutdown. VMs crash. Storage I/O stops mid-write.

  OVE (RHACM + ODF):
  T+10s    Ceph MONs on Site B detect loss of Site A MONs.
           If Metro-DR: arbiter node (Site C) breaks quorum.
           MON quorum: 2 of 3 (Site B + arbiter) = quorum maintained.
           If Regional-DR: Site B Ceph cluster is independent, unaffected.
  T+30s    RHACM hub cluster detects OCP Cluster A as unreachable.
           ManagedCluster status: "Unknown" -> "Offline".
           Alert fires in alertmanager / PagerDuty.
  T+60s    RHACM confirms cluster A is down (configurable timeout).
           DRPlacementControl CRD can be updated (manual or automated).

  Azure Local:
  T+5s     WSFC heartbeat between sites fails.
           If stretch cluster: surviving nodes detect quorum loss for Site A.
           Cloud Witness (Azure blob) or File Share Witness breaks tie.
  T+15s    WSFC determines Site A nodes are down.
           Cluster quorum maintained by Site B nodes + witness.
  T+30s    Storage Replica logs replication interruption.
           If sync mode: writes at Site B were already committed.
           If async mode: last replicated point is the recovery target.
  T+60s    Admin alerted via WAC / SCOM / Azure Monitor.

  Swisscom ESC:
  T+0-60s  Swisscom NOC monitoring detects site failure.
           Multiple monitoring systems: infrastructure, network, application.
  T+60s    Swisscom NOC initiates incident management process.
  T+300s   Swisscom contacts customer for failover authorization
           (unless pre-authorized automatic failover is contracted).
           SLA-dependent: typical initial response 15-30 minutes.

Split-Brain Prevention

Split-brain -- where both sites believe they are primary -- is the most dangerous failure mode in DR. Each platform prevents it differently:

Split-Brain Prevention Mechanisms
====================================

Problem: Site A and Site B lose connectivity but both remain operational.
Both sites could accept writes, causing divergent data.

OVE (Ceph / ODF):
+------------------+                           +------------------+
| Site A           |     link down             | Site B           |
| Ceph MONs: 1    |         X                 | Ceph MONs: 1     |
| (no quorum)     |                           | (no quorum)      |
+------------------+                           +------------------+
        |                                              |
        +---------------- Site C ----------------------+
                     | Arbiter MON: 1 |
                     | Votes with B   |
                     | B has quorum   |
                     | A is fenced    |
                     +-----------------+

  Metro-DR: Arbiter node on Site C (or cloud) provides tie-breaking MON.
    Site with arbiter connectivity wins quorum. Losing site's OSDs
    are marked DOWN, cannot accept writes. No split-brain possible.
  Regional-DR: Independent clusters. rbd-mirror cannot promote
    secondary if primary is still reachable. Operator must explicitly
    demote primary or confirm it is down before promoting secondary.


Azure Local (WSFC):
+------------------+                           +------------------+
| Site A           |     link down             | Site B           |
| 2 nodes          |         X                 | 2 nodes          |
| 2 votes          |                           | 2 votes          |
+------------------+                           +------------------+
        |                                              |
        +------------- Cloud Witness ------------------+
                   | Azure Blob: 1 vote |
                   | Accessible from B  |
                   | B has 3/5 votes    |
                   | A has 2/5 votes    |
                   | A loses quorum     |
                   +--------------------+

  WSFC uses a witness (Cloud Witness in Azure, or file share witness)
  as the tie-breaking vote. The site that can reach the witness wins.
  The losing site's nodes stop their cluster service and VMs are halted.
  For Azure Local: Cloud Witness requires Azure connectivity.
  RISK: if both sites lose Azure connectivity, neither has quorum.
  MITIGATION: file share witness on a third site (on-prem).


Swisscom ESC:
  Split-brain prevention is handled at the SAN level by the storage
  controllers (Dell SRDF/Metro uses a witness/bias mechanism) and by
  Swisscom's operational procedures. The customer does not manage
  quorum or witness infrastructure. This is a contractual guarantee,
  not a technical control the customer can verify independently.

Phase 2: Failover Decision

The decision to fail over is the most consequential step. It determines whether recovery begins or whether the team waits for Site A to come back online.

Timing: T+0 to T+5 minutes

Failover Decision Timeline
==============================

  T+0:00   Site A fails
  T+0:30   Failure detected (all platforms)
  T+1:00   Alerts received by on-call team

  OVE:
  T+1:00   On-call engineer reviews RHACM dashboard.
           Confirms Cluster A is "Offline", not a monitoring false positive.
  T+2:00   Engineer applies DRPlacementControl failover:
             kubectl patch drplacementcontrol <name> \
               --type merge \
               -p '{"spec":{"action":"Failover",
                    "failoverCluster":"site-b-cluster"}}'
  T+2:30   ODF DR orchestrator begins storage failover.

  Azure Local (stretch cluster):
  T+0:15   WSFC automatic quorum resolution.
  T+0:30   Cluster roles begin automatic failover to Site B nodes.
  T+1:00   VMs starting on Site B (automatic, no human decision needed).

  Azure Local (cluster-to-cluster):
  T+1:00   On-call engineer reviews WAC / SCOM alerts.
  T+2:00   Engineer initiates Storage Replica partnership switch:
             Set-SRPartnership -NewSourceComputerName SiteB-Node1 ...
  T+2:30   Storage Replica promotes Site B volumes.

  Swisscom ESC:
  T+1:00   Swisscom NOC detects failure.
  T+5:00   Swisscom NOC contacts customer on-call for authorization.
  T+10:00  Authorization confirmed, Swisscom initiates DR runbook.
  T+15:00  SAN-level failover begins (Swisscom-executed).

Phase 3: Storage Failover

Storage failover is the critical path. VMs cannot start until their disks are writable on Site B.

Storage Failover: Promote Replicas to Read-Write
===================================================

OVE (Ceph RBD Mirroring):
  1. Skip demote (Site A is down)
  2. Force-promote on Site B: rbd mirror image promote <img> --force
     -> Image becomes R/W. Data since last mirror-snapshot is LOST (RPO gap).
  3. ODF DR orchestrator automates for all images via DRPlacementControl:
     -> Promotes all RBD images in parallel (~10-30 sec)
     -> Creates PVs/PVCs on Site B cluster

  Timeline (500 VMs, 1000 RBD images):
    T+2:30 promotion begins -> T+3:00 all promoted -> T+3:30 storage ready

Azure Local (Storage Replica):
  Stretch cluster: Automatic. WSFC promotes Site B volumes. ~5-15 seconds.
  Cluster-to-cluster: Manual. Force-remove SR partnership (Site A unreachable):
    Remove-SRPartnership -Force / Remove-SRGroup -Force
    Site B volumes become writable. ~15-60 seconds.
    RISK: replication state lost, must re-establish from scratch.

  Timeline: Stretch T+0:30 ready | C2C T+3:00 ready

Swisscom ESC (SAN-level):
  Swisscom executes SAN failover (SRDF/SnapMirror break), updates LUN
  masking/zoning, rescans datastores on Site B vCenter.
  Duration: ~5-15 minutes. Customer has no visibility into these steps.

  Timeline: T+15:00 begin -> T+25:00 storage ready

Phase 4: Network Failover

Once storage is writable, the network must route traffic to Site B. This involves DNS updates, BGP route changes, and firewall reconfiguration.

Network Failover Architecture
================================

Before Failover (Steady State):
                                    Internet / WAN
                                         |
                                    +----+----+
                                    |  GSLB / |
                                    |  DNS    |
                                    +----+----+
                                    /         \
                             active /           \ standby
                                  /             \
  Site A (BGP AS 65100)          /     Site B (BGP AS 65200)
  +-------------------------+   /     +-------------------------+
  | Border routers          |--+      | Border routers          |
  | Advertise: 10.100.0.0/16|         | NOT advertising         |
  | (workload subnets)      |         | 10.100.0.0/16           |
  +-------------------------+         +-------------------------+
  | Spine-Leaf fabric       |         | Spine-Leaf fabric       |
  | VMs running on VLANs    |         | VMs standby / off       |
  +-------------------------+         +-------------------------+


After Failover:
                                    Internet / WAN
                                         |
                                    +----+----+
                                    |  GSLB / |
                                    |  DNS    |
                                    +----+----+
                                    /         \
                            down   /           \ active
                                  /             \
  Site A (BGP AS 65100)          /     Site B (BGP AS 65200)
  +-------------------------+         +-------------------------+
  | OFFLINE                 |         | Border routers          |
  |                         |         | NOW advertising         |
  |                         |         | 10.100.0.0/16           |
  +-------------------------+         +-------------------------+
                                      | Spine-Leaf fabric       |
                                      | VMs running on VLANs    |
                                      +-------------------------+

Network Failover Steps per Platform

Network Failover Timing

Network Convergence Timeline
===============================

  T+3:00   Storage failover complete (OVE/Azure Local)
           T+25:00 for Swisscom ESC

  BGP Convergence:
  T+3:10   BFD detects Site A link down (if BFD enabled, 3x 300ms = ~1s)
           OR BGP hold timer expires (default 90s, tuned to 10-30s)
  T+3:15   Site A routes withdrawn from upstream routers
  T+3:20   Site B routes announced (pre-staged, activated by DR automation)
  T+3:30   BGP convergence across upstream providers: ~10-30 seconds
  T+3:45   External traffic arriving at Site B

  DNS Propagation:
  T+3:00   DNS records updated (automated or manual)
  T+3:00   Clients with cached records: stale until TTL expires
  to       (TTL=60s: 1 minute; TTL=300s: 5 minutes)
  T+8:00   Most clients have refreshed DNS

  Internal Network:
  T+3:00   Site B fabric is already operational (always running)
  T+3:05   VLANs active, gateway IPs configured (pre-staged)
  T+3:10   ARP tables populated as VMs boot and send gratuitous ARP

Phase 5: VM Restart

With storage writable and network paths established, VMs must be started on Site B. The order matters -- databases before application servers before web frontends.

VM Boot Order and Dependency Management
==========================================

Dependency Tree (Example: Banking Application Stack):

  Priority 1 (T+4:00 - T+6:00): Infrastructure Services
    +-- DNS servers (2 VMs)
    +-- Active Directory / LDAP (2 VMs)
    +-- NTP servers (2 VMs)
    +-- Certificate Authority (1 VM)
    +-- Monitoring collectors (2 VMs)

  Priority 2 (T+6:00 - T+10:00): Database Tier
    +-- Oracle RAC cluster (4 VMs) -- requires shared storage + interconnect
    +-- PostgreSQL primary + standby (2 VMs)
    +-- MS SQL Always On (3 VMs)
    +-- Message broker / Kafka (3 VMs)

  Priority 3 (T+10:00 - T+15:00): Application Tier
    +-- Core banking middleware (8 VMs)
    +-- Payment processing (4 VMs)
    +-- Risk calculation engine (6 VMs)
    +-- API gateway (2 VMs)

  Priority 4 (T+15:00 - T+20:00): Frontend / Integration
    +-- Web application servers (10 VMs)
    +-- Customer portal (4 VMs)
    +-- Partner API endpoints (2 VMs)
    +-- Reporting servers (4 VMs)

Total Critical VMs (Priority 1-4): ~60 VMs
Non-critical (Priority 5): dev/test, batch, archive (100+ VMs, hours later)

VM Restart per Platform

OVE: VM Restart via RHACM + DRPlacementControl
=================================================

  DRPlacementControl manages application placement:
  - Each application group is a DRPlacementControl CR
  - CR references a PlacementRule and a DRPolicy
  - Failover triggers redeployment on Site B cluster

  Boot sequence controlled by:
  1. DRPlacementControl priority ordering (custom annotation)
  2. Kubernetes init containers for dependency checks
  3. Readiness probes for health gating
  4. ArgoCD sync waves (if using GitOps-managed DR)

  Per-VM restart: ~60 seconds (VMI create -> schedule -> kubelet ->
    virt-handler -> QEMU boot -> guest OS -> readiness probe passes)

  Parallelism: kube-scheduler handles hundreds of Pods concurrently.
  60 critical VMs across 9 worker nodes = ~7 VMs/node, all parallel.
  Bottleneck: Ceph IOPS for parallel boot (NVMe-backed OSDs handle this).

  Estimated timeline (Priority 1-4, 60 VMs):
    Priority 1: T+4:00 to T+5:00 (9 VMs, 60s boot each)
    Priority 2: T+5:00 to T+8:00 (12 VMs, databases need recovery)
    Priority 3: T+8:00 to T+12:00 (20 VMs, app startup + healthchecks)
    Priority 4: T+12:00 to T+15:00 (19 VMs, frontend servers)


Azure Local: VM Restart via WSFC / PowerShell
================================================

  Stretch cluster (automatic):
    WSFC auto-restarts VMs on Site B using priority levels:
    3000 (infra) -> 2000 (DB) -> 1000 (app) -> 0 (default).

  Cluster-to-cluster (manual):
    Import VMs from replicated CSV (pre-staged config) or start
    pre-created dormant WSFC roles. Terraform/ARM also possible.

  Estimated timeline (Priority 1-4, 60 VMs):
    Stretch: T+1:00 to T+10:00 (automatic, overlaps with storage failover)
    Cluster-to-cluster: T+5:00 to T+20:00 (manual import + start)


Swisscom ESC: VM Restart via Swisscom Runbook
===============================================

  Swisscom executes the DR runbook:
  1. ESXi hosts on Site B rescan datastores
  2. VMs registered in Site B vCenter (from replicated datastore)
  3. VMs powered on per priority order defined in DR plan
  4. Swisscom verifies infrastructure-level health

  Customer has limited control over boot order granularity.
  DR plan must be pre-agreed and documented in the service contract.

  Estimated timeline (Priority 1-4, 60 VMs):
    T+25:00 to T+60:00 (SAN failover + VM registration + boot)
    SLA-dependent: contractual RTO typically 4-8 hours

Phase 6: Application-Level Health Verification

VMs running does not mean the application is functional. Each layer must be validated.

Health Verification Checklist (by Priority Group)
====================================================

Layer 1 -- Infrastructure (T+5:00):
  [ ] DNS: dig @<dns-vm-ip> bankapp.internal.example.com  -> resolves
  [ ] AD/LDAP: ldapsearch succeeds, authentication works
  [ ] NTP: chronyc tracking -> offset < 100ms
  [ ] PKI: certificates valid, CRL/OCSP accessible

Layer 2 -- Databases (T+8:00):
  [ ] Oracle: SELECT STATUS FROM V$INSTANCE -> OPEN
  [ ] PostgreSQL: pg_is_in_recovery() = false, redo replay complete
  [ ] Kafka: consumer groups show no lag on critical topics
  [ ] Application schemas: SELECT COUNT(*) FROM critical_table -> non-zero

Layer 3 -- Applications (T+12:00):
  [ ] Health endpoints: curl https://<api>/health -> 200 OK
  [ ] Database connectivity: app connects and queries succeed
  [ ] Message processing: test message published and consumed

Layer 4 -- End-to-End (T+15:00):
  [ ] Login flow: AD auth -> token -> session -> portal loads
  [ ] Test transaction: submit through core banking, verify in DB
  [ ] SWIFT interface: send test payment, confirm routing
  [ ] Monitoring: dashboards show Site B active, no critical alerts

Health Check Automation per Platform

Full Failover Timeline: Side-by-Side Comparison

DR Failover Timeline: Complete Comparison
=============================================

Phase                          OVE              Azure Local       Azure Local       Swisscom
                               (Regional-DR)    (Stretch)         (Cluster-to-C.)   ESC
------------------------------ ---------------- ----------------- ----------------- ---------
T+0:00  Site A fails           |                |                 |                 |
                               |                |                 |                 |
T+0:15  WSFC heartbeat fails   |                * detect          |                 |
T+0:30  Failure detected       * detect         |                 * detect          |
T+1:00  NOC alerted            |                | quorum          |                 * detect
                               |                | resolved        |                 |
T+1:30  WSFC auto-failover     |                * storage         |                 |
        begins (stretch)       |                | promoting       |                 |
T+2:00  Decision: failover     * decision       |                 |                 |
T+2:30  Storage promotion      * storage        |                 * decision        |
        begins                 | promoting      |                 |                 |
T+3:00  Storage ready (OVE)    * storage        * VMs starting    * storage         |
                               | ready          | (auto)          | promoting       |
T+3:30  Network failover       * network        |                 |                 |
        (BGP + DNS)            | converging     |                 |                 |
T+4:00  Priority 1 VMs boot   * P1 boot        |                 * storage         |
                               |                |                 | ready           |
T+5:00  P1 VMs ready (OVE)    * P1 ready       * P1 ready        * P1 boot         * NOC
        Infra services up      |                | (auto)          |                 | contacts
T+5:00  Swisscom contacts      |                |                 |                 | customer
        customer               |                |                 |                 |
T+8:00  P2 VMs ready (OVE)    * P2 ready       * P2 ready        * P2 ready        |
        Databases recovered    |                |                 |                 |
T+10:00 Swisscom authorized    |                |                 |                 * auth
                               |                |                 |                 |
T+12:00 P3 VMs ready (OVE)    * P3 ready       * P3 ready        * P3 ready        |
        Applications up        |                |                 |                 |
T+15:00 P4 VMs ready          * P4 ready       * P4 ready        * P4 ready        * SAN
        End-to-end validated    |                |                 |                 | failover
                               |                |                 |                 |
T+20:00 All critical up       * DONE            * DONE            * DONE            |
        (OVE, Azure Local)     |                |                 |                 * VMs
                               |                |                 |                 | starting
T+30:00                        |                |                 |                 |
                               |                |                 |                 |
T+45:00                        |                |                 |                 * P1-P4
                               |                |                 |                 | ready
T+60:00 Swisscom critical     |                |                 |                 * DONE
        VMs ready              |                |                 |                 |

SUMMARY:
  RTO (all critical VMs operational):
    OVE Regional-DR:              ~15-20 minutes
    Azure Local Stretch Cluster:  ~10-15 minutes
    Azure Local Cluster-to-Cluster: ~15-25 minutes
    Swisscom ESC:                 ~45-60 minutes (SLA: 4-8 hours)

Metro-DR vs Regional-DR Comparison

The two DR topologies have fundamentally different tradeoffs. Both may be needed for different workload tiers.

Metro-DR (RPO=0)                        Regional-DR (RPO>0)
================================        ================================
Distance: < 100 km (Basel-Zurich)       Distance: unlimited

Site A  <=== sync ===>  Site B          Site A  ==== async ====>  Site B
  |    every write on     |               |    snapshot-based       |
  |    both sites         |               |    every 5 min          |
  +---- Arbiter (Site C) -+               +---- RHACM Hub ---------+

Write latency: ~2-4 ms (cross-site     Write latency: ~0.1-0.5 ms
  RTT dominates). 4-20x penalty.          (local only, no penalty)

Failure: arbiter breaks quorum,         Failure: operator promotes Site B
  Site B OSDs become primary.             images (--force). Data since
  No data loss.                           last snapshot is LOST.
  RTO: seconds to minutes.               RPO: up to interval + copy time.

Decision Matrix: Metro-DR vs Regional-DR

Platform Support Matrix

Phase 7: Failback -- Return to Site A

Failback is often harder than failover. Site A must be rebuilt, data must be resynced, and the cutover must be planned to minimize disruption.

Failback Procedure (All Platforms)
=====================================

Prerequisites:
  - Site A infrastructure restored (power, cooling, network, servers)
  - Site A cluster rebuilt / repaired
  - Site A storage subsystem operational
  - Network paths between sites re-established


Step 1: Rebuild Site A (4-8 hours)
  OVE:     Reinstall OCP + ODF, bootstrap new Ceph cluster (IPI/UPI).
  Azure L: Rebuild WSFC + S2D, reconfigure Storage Replica (WAC/PS).
  ESC:     Swisscom rebuilds (SLA-dependent, hours to days).

Step 2: Reverse Replication -- Site B -> Site A (hours to days)
  All platforms: establish replication with reversed direction.
  Initial sync is a FULL copy (no prior baseline on rebuilt Site A).
  Duration: data-size dependent. 100 TiB at 10 Gbps = ~22 hours.
  VMs continue running on Site B during resync -- no downtime.

  OVE:  rbd mirror pool enable + peer add (Site A receives replicas)
  AzL:  New-SRPartnership with reversed source/destination
  ESC:  Swisscom re-establishes SAN replication (customer has no control)

Step 3: Verify Sync Completion
  OVE:  rbd mirror image status -> state: up+replaying, recent timestamp
  AzL:  (Get-SRGroup).Replicas -> NumOfBytesRemaining = 0
  ESC:  Swisscom confirms via customer portal


Step 4: Planned Failback (Controlled Cutover)
-----------------------------------------------
  Unlike the disaster failover, failback is PLANNED. This means:
  - Maintenance window scheduled (e.g., Sunday 02:00-06:00)
  - Users notified
  - Graceful VM shutdown (application quiescing, database checkpoint)
  - Final replication sync (delta only, very fast)
  - Role switch (Site A becomes primary, Site B becomes secondary)
  - VM restart on Site A in priority order
  - Validation and traffic cutover

  OVE Failback Timeline:
    T+0:00   Maintenance window opens
    T+0:05   Graceful shutdown of Priority 4 VMs (frontend)
    T+0:10   Graceful shutdown of Priority 3 VMs (applications)
    T+0:15   Graceful shutdown of Priority 2 VMs (databases)
    T+0:20   All VMs stopped. Final mirror-snapshot replicated.
    T+0:25   rbd-mirror: demote Site B images, promote Site A images
    T+0:30   DRPlacementControl updated: preferred cluster = site-a
    T+0:35   Priority 1 VMs boot on Site A
    T+0:40   Priority 2 VMs boot (database recovery)
    T+1:00   Priority 3 VMs boot (applications)
    T+1:15   Priority 4 VMs boot (frontend)
    T+1:30   Validation complete, DNS/BGP cut back to Site A
    T+2:00   Reverse replication re-established (Site A -> Site B)
    T+2:00   Maintenance window closes
    Total failback downtime: ~1.5-2 hours (planned)

FINMA Compliance Requirements for DR

FINMA Circular 2023/1 (Operational Risks and Resilience) establishes specific requirements that affect DR design, testing, and documentation.

What Must Be Documented

What Must Be Tested Annually

DR Testing: Non-Disruptive Approach per Platform

Non-Disruptive DR Testing (Production Replication Continues)
==============================================================

OVE:     Clone replicated PVCs into test namespace -> deploy VMs from
         clones -> run validation -> delete namespace (cascading cleanup).
         Production impact: zero. Duration: 2-4 hours. Replication: uninterrupted.

Azure L: Mount-SRDestination to create test snapshot -> import VMs on
         isolated vSwitch -> validate -> Dismount-SRDestination.
         Production impact: zero. Duration: 2-4 hours. Replication: paused/resumed.

ESC:     Request via Swisscom portal -> Swisscom creates test env from
         replicated storage -> customer validates -> Swisscom tears down.
         Duration: 4-8 hours. KEY RISK: customer cannot independently initiate.

Complete Platform Comparison

DR Capability Matrix

Cost of DR Infrastructure (Order of Magnitude)

Key Takeaways

1. Azure Local stretch cluster offers the fastest automatic failover. With WSFC quorum, Cloud Witness, and Storage Replica synchronous mode, failover is automatic and begins within seconds. No human decision is needed. RTO for critical VMs: 10-15 minutes. The tradeoff: stretch cluster has a 16-node limit, Cloud Witness requires Azure connectivity, and write latency increases for all workloads (sync replication penalty).

2. OVE Regional-DR provides the best balance of control and automation. DRPlacementControl CRDs make failover declarative and GitOps-compatible. The RHACM orchestrator handles storage promotion, workload rescheduling, and can integrate with external DNS/LB automation. RTO: 15-20 minutes. The tradeoff: Regional-DR is async (RPO=5-15 minutes), and ODF Metro-DR (RPO=0) is newer technology (GA since ODF 4.12) with less production track record than WSFC stretch clusters.

3. Swisscom ESC has the longest RTO but the lowest operational burden. The customer does not manage replication, quorum, or failover orchestration. Swisscom handles everything. The tradeoff: the customer cannot independently initiate failover, cannot control timing, and depends on Swisscom's operational responsiveness. For FINMA compliance, the institution must demonstrate oversight of the provider's DR capability -- this requires contractual guarantees, regular test evidence from Swisscom, and ideally independent verification.

4. Split-brain prevention is architecturally different across platforms. OVE uses a Ceph arbiter MON on a third site. Azure Local uses a Cloud Witness in Azure (or a file share witness on-prem). Swisscom manages it at the SAN level. The OVE and Azure Local approaches are transparent and verifiable; the Swisscom approach is opaque to the customer. For FINMA audits, the ability to demonstrate and test split-brain prevention is important.

5. Failback is the forgotten half of DR. All platforms require a full resync (Site B -> Site A) before failback. For 100 TiB of data at 10 Gbps, the initial resync takes approximately 22 hours. During this period, VMs continue running on Site B with no disruption. The planned failback cutover takes 1.5-2 hours of downtime. This resync period must be factored into the overall DR timeline.

6. Non-disruptive DR testing is essential for FINMA compliance. OVE's PVC clone approach and Azure Local's SR destination mount both allow testing without interrupting production replication. Swisscom requires coordination with the provider. The evaluation team should validate that each platform's non-disruptive test mechanism works correctly during the PoC -- not just in theory, but with actual application-level validation.

7. Metro-DR (RPO=0) comes with a performance tax. Synchronous replication between Basel and Zurich adds 2-4 ms to every write. For latency-sensitive workloads (trading systems, real-time payment processing), this penalty may be unacceptable. The recommended architecture is a tiered approach: Metro-DR for Tier-1 systems requiring RPO=0, Regional-DR for all other systems.

Discussion Guide

These questions should be raised with engineering teams and vendors during the PoC:

1. For OVE (Red Hat): What is the production track record of ODF Metro-DR stretched clusters in financial institutions? How many customers run ODF 4.12+ Metro-DR in production? What is the maximum tested cluster size for Metro-DR (number of OSDs, number of mirrored images)? How does the arbiter node handle network partitions where it can reach both sites but the sites cannot reach each other?

2. For Azure Local (Microsoft): What happens to Cloud Witness if Azure has an outage? Can the stretch cluster survive a simultaneous Azure outage and site-to-site link failure? What is the tested maximum for Storage Replica async lag under sustained high write load (e.g., 500 VMs each writing 10 MB/s)? How does the 16-node limit for stretch clusters affect our capacity planning for 5,000+ VMs?

3. For Swisscom ESC: What is the contractual RTO guarantee, and what are the penalties for missing it? Can the customer independently initiate a DR test without Swisscom involvement? What evidence does Swisscom provide for FINMA audits (replication status reports, test results, incident logs)? If ESC migrates off VMware to a new stack, how does that affect the DR architecture and SLA?

4. For all platforms: What is the maximum number of VMs that can be failed over simultaneously? Is there a bottleneck in the orchestration layer (RHACM, WSFC, Swisscom runbook) that serializes VM restarts? What is the actual measured RTO for 500 VMs with a realistic dependency tree?

5. For the networking team: How will BGP route withdrawal and re-announcement be automated during failover? Is BFD configured on all inter-site links to reduce detection time from 90 seconds (BGP hold timer) to sub-second? Are DNS TTLs for all DR-critical records set to 60 seconds? What is the impact on external API consumers who have hardcoded IP addresses instead of using DNS?

6. For FINMA compliance: Does the selected platform's DR architecture satisfy FINMA Circular 2023/1 requirements for BCM, testing frequency, auditability, and third-party oversight? Can the institution produce all required evidence (RPO/RTO measurement, test reports, replication status logs) from the platform's native tooling, or is additional monitoring infrastructure required?