Regional and Cross Region Streams (Supercluster) (CLI)

Regional and Cross Region Streams (Supercluster) in Use Cases

This example demonstrates the configuration required to setup a three regional clusters and a cross-region cluster that form a supercluster.

The use case is for streams that are resilient to regional failures by having replicas spread out across three different regions. This of course comes at a cost of latency, but this trade-off may be appropriate for streams that are critical for availability, but can tolerate slightly higher latencies.

CLI Go Python JavaScript Rust C# .NET V2 Java Ruby Elixir Crystal C

Jump to the output or the recording

$ nbe run use-cases/cross-region-streams-supercluster/cli

View the source code or learn how to run this example yourself

Code

#!/bin/bash


set -euo pipefail


NATS_URL="nats://localhost:4222"

Create a shared configuration which enables JetStream and defines the unique_tag option which enforces all replicas for a given stream or consumer to be placed on nodes with different availability zones (AZ).

cat <<- EOF > regional-shared.conf
jetstream: {
  unique_tag: "az:"
}


accounts: {
  \$SYS: {
    users: [{user: sys, password: sys}]
  }
  APP: {
    jetstream: true
    users: [{user: user, password: user}]
  }
}
EOF

The cross-region cluster defines a unique_tag on the cluster. Any AZ can be chosen as long as the region is unique.

cat <<- EOF > cross-region-shared.conf
jetstream: {
  unique_tag: "rg:"
}


accounts: {
  \$SYS: {
    users: [{user: sys, password: sys}]
  }
  APP: {
    jetstream: true
    users: [{user: user, password: user}]
  }
}
EOF


cat <<- EOF > gateway-routes.conf
gateways: [
  {name: rg1, urls: [
    nats://localhost:7222,
    nats://localhost:7223,
    nats://localhost:7224,
  ]},
  {name: rg2, urls: [
    nats://localhost:7225,
    nats://localhost:7226,
    nats://localhost:7227,
  ]},
  {name: rg3, urls: [
    nats://localhost:7228,
    nats://localhost:7229,
    nats://localhost:7230,
  ]},
  {name: xr, urls: [
    nats://localhost:7231,
    nats://localhost:7232,
    nats://localhost:7233,
  ]},
]
EOF

Define the server configs for region 1 cluster.

cat <<- EOF > "rg1-az1.conf"
server_name: rg1-az1
server_tags: [az:1]
port: 4222
http_port: 8222
include regional-shared.conf
cluster: {
  name: rg1
  port: 6222
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
gateway {
  name: rg1
  port: 7222
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg1-az1
}
EOF


cat <<- EOF > "rg1-az2.conf"
server_name: rg1-az2
server_tags: [az:2]
port: 4223
include regional-shared.conf
cluster: {
  name: rg1
  port: 6223
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
gateway {
  name: rg1
  port: 7223
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg1-az2
}
EOF


cat <<- EOF > "rg1-az3.conf"
server_name: rg1-az3
server_tags: [az:3]
port: 4224
include regional-shared.conf
cluster: {
  name: rg1
  port: 6224
  routes: [
    nats-route://127.0.0.1:6222,
    nats-route://127.0.0.1:6223,
    nats-route://127.0.0.1:6224,
  ]
}
gateway {
  name: rg1
  port: 7224
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg1-az3
}
EOF

Server configs for region 2 cluster.

cat <<- EOF > "rg2-az1.conf"
server_name: rg2-az1
server_tags: [az:1]
port: 4225
http_port: 8223
include regional-shared.conf
cluster: {
  name: rg2
  port: 6225
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}
gateway {
  name: rg2
  port: 7225
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg2-az1
}
EOF


cat <<- EOF > "rg2-az2.conf"
server_name: rg2-az2
server_tags: [az:2]
port: 4226
include regional-shared.conf
cluster: {
  name: rg2
  port: 6226
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}
gateway {
  name: rg2
  port: 7226
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg2-az2
}
EOF


cat <<- EOF > "rg2-az3.conf"
server_name: rg2-az3
server_tags: [az:3]
port: 4227
include regional-shared.conf
cluster: {
  name: rg2
  port: 6227
  routes: [
    nats-route://127.0.0.1:6225,
    nats-route://127.0.0.1:6226,
    nats-route://127.0.0.1:6227,
  ]
}
gateway {
  name: rg2
  port: 7227
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg2-az3
}
EOF

Server configs for region 3 cluster.

cat <<- EOF > "rg3-az1.conf"
server_name: rg3-az1
server_tags: [az:1]
port: 4228
http_port: 8224
include regional-shared.conf
cluster: {
  name: rg3
  port: 6228
  routes: [
    nats-route://127.0.0.1:6228,
    nats-route://127.0.0.1:6229,
    nats-route://127.0.0.1:6230,
  ]
}
gateway {
  name: rg3
  port: 7228
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg3-az1
}
EOF


cat <<- EOF > "rg3-az2.conf"
server_name: rg3-az2
server_tags: [az:2]
port: 4229
include regional-shared.conf
cluster: {
  name: rg3
  port: 6229
  routes: [
    nats-route://127.0.0.1:6228,
    nats-route://127.0.0.1:6229,
    nats-route://127.0.0.1:6230,
  ]
}
gateway {
  name: rg3
  port: 7229
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg3-az2
}
EOF


cat <<- EOF > "rg3-az3.conf"
server_name: rg3-az3
server_tags: [az:3]
port: 4230
include regional-shared.conf
cluster: {
  name: rg3
  port: 6230
  routes: [
    nats-route://127.0.0.1:6228,
    nats-route://127.0.0.1:6229,
    nats-route://127.0.0.1:6230,
  ]
}
gateway {
  name: rg3
  port: 7230
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg3-az3
}
EOF

Server configs for cross-region cluster. In this case, an arbitrary AZ is chosen per region. If may be desirable to have a six or nine node cross-region cluster if more AZs per region are desired.

cat <<- EOF > "rg1-az1-x.conf"
server_name: rg1-az1-x
server_tags: [rg:1, az:1]
port: 4231
http_port: 8225
include cross-region-shared.conf
cluster: {
  name: xr
  port: 6231
  routes: [
    nats-route://127.0.0.1:6231,
    nats-route://127.0.0.1:6232,
    nats-route://127.0.0.1:6233,
  ]
}
gateway {
  name: xr
  port: 7231
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg1-az1-x
}
EOF


cat <<- EOF > "rg2-az2-x.conf"
server_name: rg2-az2-x
server_tags: [rg:2, az:2]
port: 4232
include cross-region-shared.conf
cluster: {
  name: xr
  port: 6232
  routes: [
    nats-route://127.0.0.1:6231,
    nats-route://127.0.0.1:6232,
    nats-route://127.0.0.1:6233,
  ]
}
gateway {
  name: xr
  port: 7232
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg2-az2-x
}
EOF


cat <<- EOF > "rg3-az3-x.conf"
server_name: rg3-az3-x
server_tags: [rg:3, az:3]
port: 4233
include cross-region-shared.conf
cluster: {
  name: xr
  port: 6233
  routes: [
    nats-route://127.0.0.1:6231,
    nats-route://127.0.0.1:6232,
    nats-route://127.0.0.1:6233,
  ]
}
gateway {
  name: xr
  port: 7233
  include gateway-routes.conf
}
jetstream: {
  store_dir: /tmp/nats/storage/rg3-az3-x
}
EOF

Start a server for each configuration and sleep a second in between so the seeds can startup and get healthy.

for c in $(ls rg*.conf); do
  echo "Starting server ${c%.*}"
  nats-server -c $c -l "${c%.*}.log" > /dev/null 2>&1 &
  sleep 1
done

Wait until the servers up and healthy.

echo 'Cluster 1 healthy?'
curl --fail --silent \
  --retry 5 \
  --retry-delay 1 \
  http://localhost:8222/healthz; echo


echo 'Cluster 2 healthy?'
curl --fail --silent \
  --retry 5 \
  --retry-delay 1 \
  http://localhost:8223/healthz; echo


echo 'Cluster 3 healthy?'
curl --fail --silent \
  --retry 5 \
  --retry-delay 1 \
  http://localhost:8224/healthz; echo


echo 'Cluster 4 healthy?'
curl --fail --silent \
  --retry 5 \
  --retry-delay 1 \
  http://localhost:8225/healthz; echo

Show the server lit and JetStream report.

nats --user sys --password sys server info rg2-az1
nats --user sys --password sys server list
nats --user sys --password sys server report jetstream

Create a cross-region stream specifying the xr cluster.

nats --user user --password user stream add \
  --cluster=xr \
  --retention=limits \
  --storage=file \
  --replicas=3 \
  --discard=old \
  --dupe-window=2m \
  --max-age=-1 \
  --max-msgs=-1 \
  --max-bytes=-1 \
  --max-msg-size=-1 \
  --max-msgs-per-subject=-1 \
  --max-consumers=-1 \
  --allow-rollup \
  --no-deny-delete \
  --no-deny-purge \
  --subjects="events.*" \
  EVENTS

Create a regional stream specifying one of the regional clusters, e.g. rg2.

nats --user user --password user stream add \
  --cluster=rg2 \
  --retention=limits \
  --storage=file \
  --replicas=3 \
  --discard=old \
  --dupe-window=2m \
  --max-age=-1 \
  --max-msgs=-1 \
  --max-bytes=-1 \
  --max-msg-size=-1 \
  --max-msgs-per-subject=-1 \
  --max-consumers=-1 \
  --allow-rollup \
  --no-deny-delete \
  --no-deny-purge \
  --subjects="orders.*" \
  ORDERS

Report out the streams.

nats --user user --password user stream report

Output

Starting server rg1-az1-x
Starting server rg1-az1
Starting server rg1-az2
Starting server rg1-az3
Starting server rg2-az1
Starting server rg2-az2-x
Starting server rg2-az2
Starting server rg2-az3
Starting server rg3-az1
Starting server rg3-az2
Starting server rg3-az3-x
Starting server rg3-az3
Cluster 1 healthy?
{"status":"ok"}
Cluster 2 healthy?
{"status":"ok"}
Cluster 3 healthy?
{"status":"ok"}
Cluster 4 healthy?
{"status":"ok"}
Server information for rg2-az1 (NCYRXWEEZTZVL76ACBYAX6GGSFWCKGO7RX6U5EI6CLAJ2YVHDMPSPEPV)

Process Details:

                     Version: 2.10.1
                  Git Commit: 
                  Go Version: go1.21.3
                  Start Time: 2023-10-23 19:31:26
            Config Load Time: 2023-10-23 19:31:26
                      Uptime: 11s

Connection Details:

               Auth Required: true
                TLS Required: false
                        Host: 0.0.0.0:4225
                 Client URLs: 172.29.0.2:4225
                              172.29.0.2:4226
                              172.29.0.2:4227

JetStream:

                      Domain: 
           Storage Directory: /tmp/nats/storage/rg2-az1/jetstream
                  Max Memory: 5.8 GiB
                    Max File: 20 GiB
             Active Accounts: 1
               Memory In Use: 0 B
                 File In Use: 0 B
                API Requests: 0
                  API Errors: 0
  Always sync writes to disk: false
        Write sync Frequency: 2m0s

Limits:

                    Max Conn: 65,536
                    Max Subs: 0
                 Max Payload: 1.0 MiB
                 TLS Timeout: 2.00s
              Write Deadline: 10.00s

Statistics:

                   CPU Cores: 8 0.50%
                      Memory: 17 MiB
                 Connections: 0
               Subscriptions: 174
                    Messages: 145 in 247 out
                       Bytes: 212 KiB in 324 KiB out
              Slow Consumers: 0

Cluster:

                        Name: rg2
                        Tags: az:1
                        Host: 0.0.0.0:6225
                        URLs: 127.0.0.1:6225
                              127.0.0.1:6226
                              127.0.0.1:6227

Super Cluster:

                        Name: rg2
                        Host: 0.0.0.0:7225
                    Clusters: rg1
                              rg2
                              rg3
                              xr
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                      Server Overview                                                      │
├───────────┬─────────┬──────┬─────────┬─────┬───────┬───────┬────────┬─────┬─────────┬───────┬───────┬──────┬────────┬─────┤
│ Name      │ Cluster │ Host │ Version │ JS  │ Conns │ Subs  │ Routes │ GWs │ Mem     │ CPU % │ Cores │ Slow │ Uptime │ RTT │
├───────────┼─────────┼──────┼─────────┼─────┼───────┼───────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│ rg1-az1   │ rg1     │ 0    │ 2.10.1  │ yes │ 1     │ 184   │      8 │   3 │ 17 MiB  │ 2     │     8 │    0 │ 14.14s │ 1ms │
│ rg1-az1-x │ xr      │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 17 MiB  │ 0     │     8 │    0 │ 15.14s │ 1ms │
│ rg3-az1   │ rg3     │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 17 MiB  │ 2     │     8 │    0 │ 7.10s  │ 1ms │
│ rg1-az3   │ rg1     │ 0    │ 2.10.1  │ yes │ 0     │ 184   │      8 │   3 │ 17 MiB  │ 1     │     8 │    0 │ 12.12s │ 1ms │
│ rg1-az2   │ rg1     │ 0    │ 2.10.1  │ yes │ 0     │ 184   │      8 │   3 │ 17 MiB  │ 2     │     8 │    0 │ 13.12s │ 1ms │
│ rg2-az2-x │ xr      │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 16 MiB  │ 0     │     8 │    0 │ 10.11s │ 1ms │
│ rg3-az3-x │ xr      │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 15 MiB  │ 0     │     8 │    0 │ 5.09s  │ 1ms │
│ rg2-az1   │ rg2     │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 17 MiB  │ 0     │     8 │    0 │ 11.11s │ 1ms │
│ rg2-az3   │ rg2     │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 16 MiB  │ 1     │     8 │    0 │ 8.10s  │ 1ms │
│ rg2-az2   │ rg2     │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 16 MiB  │ 2     │     8 │    0 │ 9.11s  │ 2ms │
│ rg3-az2   │ rg3     │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 16 MiB  │ 1     │     8 │    0 │ 6.09s  │ 2ms │
│ rg3-az3   │ rg3     │ 0    │ 2.10.1  │ yes │ 0     │ 174   │      8 │   3 │ 16 MiB  │ 0     │     8 │    0 │ 4.08s  │ 2ms │
├───────────┼─────────┼──────┼─────────┼─────┼───────┼───────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│           │ 4       │ 12   │         │ 12  │ 1     │ 2,118 │        │     │ 198 MIB │       │       │    0 │        │     │
╰───────────┴─────────┴──────┴─────────┴─────┴───────┴───────┴────────┴─────┴─────────┴───────┴───────┴──────┴────────┴─────╯

╭────────────────────────────────────────────────────────────────────────────╮
│                              Cluster Overview                              │
├─────────┬────────────┬───────────────────┬───────────────────┬─────────────┤
│ Cluster │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ xr      │          3 │                 9 │                 9 │           0 │
│ rg3     │          3 │                 9 │                 9 │           0 │
│ rg2     │          3 │                 9 │                 9 │           0 │
│ rg1     │          3 │                 9 │                 9 │           1 │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│         │         12 │                36 │                36 │           1 │
╰─────────┴────────────┴───────────────────┴───────────────────┴─────────────╯
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                         JetStream Summary                                        │
├───────────┬─────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server    │ Cluster │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├───────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ rg1-az1*  │ rg1     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg1-az2   │ rg1     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg1-az3   │ rg1     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg2-az1   │ rg2     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg2-az2   │ rg2     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg2-az3   │ rg2     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg3-az1   │ rg3     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg3-az2   │ rg3     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg3-az3   │ rg3     │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg1-az1-x │ xr      │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg2-az2-x │ xr      │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ rg3-az3-x │ xr      │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
├───────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│           │         │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
╰───────────┴─────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯

╭─────────────────────────────────────────────────────────────────╮
│                   RAFT Meta Group Information                   │
├───────────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name      │ ID       │ Leader │ Current │ Online │ Active │ Lag │
├───────────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ rg1-az1   │ 6s4h5MOv │ yes    │ true    │ true   │ 0s     │ 0   │
│ rg1-az1-x │ PMcvOJvm │        │ true    │ true   │ 656ms  │ 0   │
│ rg1-az2   │ X8wPeL6S │        │ true    │ true   │ 656ms  │ 0   │
│ rg1-az3   │ FozYtzby │        │ true    │ true   │ 656ms  │ 0   │
│ rg2-az1   │ 8ShHFKtZ │        │ true    │ true   │ 656ms  │ 0   │
│ rg2-az2   │ Kx5WF0Q6 │        │ true    │ true   │ 656ms  │ 0   │
│ rg2-az2-x │ UEoET1zo │        │ true    │ true   │ 656ms  │ 0   │
│ rg2-az3   │ NSUWzTzT │        │ true    │ true   │ 656ms  │ 0   │
│ rg3-az1   │ OEQTh4pP │        │ true    │ true   │ 656ms  │ 0   │
│ rg3-az2   │ LJVb57VO │        │ true    │ true   │ 656ms  │ 0   │
│ rg3-az3   │ uEtIi639 │        │ true    │ true   │ 655ms  │ 0   │
│ rg3-az3-x │ cwYBGolT │        │ true    │ true   │ 656ms  │ 0   │
╰───────────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
Stream EVENTS was created

Information for Stream EVENTS created 2023-10-23 19:31:38

              Subjects: events.*
              Replicas: 3
               Storage: File
     Placement Cluster: xr

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: true

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: xr
                Leader: rg3-az3-x
               Replica: rg1-az1-x, current, seen 753µs ago
               Replica: rg2-az2-x, current, seen 1ms ago

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 0
         Last Sequence: 0
      Active Consumers: 0
Stream ORDERS was created

Information for Stream ORDERS created 2023-10-23 19:31:38

              Subjects: orders.*
              Replicas: 3
               Storage: File
     Placement Cluster: rg2

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: true

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: rg2
                Leader: rg2-az1
               Replica: rg2-az2, current, seen 448µs ago
               Replica: rg2-az3, current, seen 46µs ago

State:

              Messages: 0
                 Bytes: 0 B
        First Sequence: 0
         Last Sequence: 0
      Active Consumers: 0
Obtaining Stream stats

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                    Stream Report                                                    │
├────────┬─────────┬───────────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────────────────────────────┤
│ Stream │ Storage │ Placement     │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas                         │
├────────┼─────────┼───────────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────────────────────────────┤
│ EVENTS │ File    │ cluster: xr   │         0 │ 0        │ 0 B   │ 0    │       0 │ rg1-az1-x, rg2-az2-x, rg3-az3-x* │
│ ORDERS │ File    │ cluster: rg2  │         0 │ 0        │ 0 B   │ 0    │       0 │ rg2-az1*, rg2-az2, rg2-az3       │
╰────────┴─────────┴───────────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────────────────────────────╯

Recording

Note, playback is half speed to make it a bit easier to follow.