by Example
Supercluster with JetStream in Topologies
When configuring and deploying a supercluster that contains JetStream-enabled servers, one server across the entire supercluster is elected to handle JetStream API requests specifically for the placement of streams and consumers.
This server is considered the leader of this meta/API layer. Since this is a Raft group as well, electing a leader requires a quorum across servers. What this means in practice is that there needs to be an odd number of servers across the supercluster.
Supercluster deployments with JetStream-enabled servers should include at least three clusters or use the arbiter pattern.
This example will demonstrate how to configure a minimal supercluster of three single-node clusters. Of course, for a production deployment it is recommended to have three nodes per cluster to support R3 streams.
$ nbe run topologies/supercluster-jetstream/cliView the source code or learn how to run this example yourself
Code
#!/bin/sh
set -euo pipefail
Define the system account to be included by all configurations.
cat <<- EOF > sys.conf
accounts: {
\$SYS: {
users: [{user: sys, password: pass}]
}
}
EOF
Create the east, central, and west server configurations. A requirement of JetStream is to have a cluster block with routes defined, even for single node clusters.
In a production deployment, at least three nodes per cluster are recommended which supports creating R3 streams. In this test setup, only R1 streams can be created since streams replicas do not cross gateway connections.
Similarity, across a supercluster, an odd number of servers must exist to support reaching consensus for the meta group leader.
cat <<- EOF > east.conf
port: 4222
http_port: 8222
server_name: n1
include sys.conf
jetstream: {
store_dir: /tmp/nats/storage/n1
}
cluster: {
name: east,
port: 6222,
routes: [
"nats-route://0.0.0.0:6222"
],
}
gateway: {
name: "east",
port: 7222,
gateways: [
{name: "east", urls: ["nats://0.0.0.0:7222"]},
{name: "central", urls: ["nats://0.0.0.0:7223"]},
{name: "west", urls: ["nats://0.0.0.0:7224"]},
]
}
EOF
cat <<- EOF > central.conf
port: 4223
http_port: 8223
server_name: n2
include sys.conf
jetstream: {
store_dir: /tmp/nats/storage/n2
}
cluster: {
name: central,
port: 6223,
routes: [
"nats-route://0.0.0.0:6223"
],
}
gateway: {
name: "central",
port: 7223,
gateways: [
{name: "east", urls: ["nats://0.0.0.0:7222"]},
{name: "central", urls: ["nats://0.0.0.0:7223"]},
{name: "west", urls: ["nats://0.0.0.0:7224"]},
]
}
EOF
cat <<- EOF > west.conf
port: 4224
http_port: 8224
server_name: n3
include sys.conf
jetstream: {
store_dir: /tmp/nats/storage/n3
}
cluster: {
name: west,
port: 6224,
routes: [
"nats-route://0.0.0.0:6224"
],
}
gateway: {
name: "west",
port: 7224,
gateways: [
{name: "east", urls: ["nats://0.0.0.0:7222"]},
{name: "central", urls: ["nats://0.0.0.0:7223"]},
{name: "west", urls: ["nats://0.0.0.0:7224"]},
]
}
EOF
Start the servers and sleep for a few seconds to startup.
nats-server -c east.conf > /dev/null 2>&1 &
nats-server -c central.conf > /dev/null 2>&1 &
nats-server -c west.conf > /dev/null 2>&1 &
sleep 3
Wait until the servers are healthy.
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8222/healthz > /dev/null
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8223/healthz > /dev/null
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8224/healthz > /dev/null
Save a couple NATS CLI contexts for convenience.
nats context save east \
--server "nats://localhost:4223"
nats context save east-sys \
--server "nats://localhost:4223" \
--user sys \
--password pass
nats context save west \
--server "nats://localhost:4223"
Show the server list which will indicate the clusters and gateway connections as well as the JetStream server report.
nats --context east-sys server list
nats --context east-sys server report jetstream
Start a service running in east.
nats --context east reply 'greet' 'hello from east' &
sleep 1
Send a request with a client connected to west.
nats --context west request 'greet' ''Output
NATS Configuration Context "east"
Server URLs: nats://localhost:4223
Path: /root/.config/nats/context/east.json
NATS Configuration Context "east-sys"
Server URLs: nats://localhost:4223
Username: sys
Password: ****
Path: /root/.config/nats/context/east-sys.json
NATS Configuration Context "west"
Server URLs: nats://localhost:4223
Path: /root/.config/nats/context/west.json
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Server Overview │
├──────┬─────────┬──────┬─────────┬─────┬───────┬──────┬────────┬─────┬────────┬───────┬───────┬──────┬────────┬─────┤
│ Name │ Cluster │ Host │ Version │ JS │ Conns │ Subs │ Routes │ GWs │ Mem │ CPU % │ Cores │ Slow │ Uptime │ RTT │
├──────┼─────────┼──────┼─────────┼─────┼───────┼──────┼────────┼─────┼────────┼───────┼───────┼──────┼────────┼─────┤
│ n2 │ central │ 0 │ 2.10.1 │ yes │ 1 │ 62 │ 0 │ 2 │ 14 MiB │ 2 │ 8 │ 0 │ 3.10s │ 2ms │
│ n3 │ west │ 0 │ 2.10.1 │ yes │ 0 │ 70 │ 0 │ 2 │ 14 MiB │ 0 │ 8 │ 0 │ 3.09s │ 2ms │
│ n1 │ east │ 0 │ 2.10.1 │ yes │ 0 │ 61 │ 0 │ 2 │ 13 MiB │ 1 │ 8 │ 0 │ 3.10s │ 2ms │
├──────┼─────────┼──────┼─────────┼─────┼───────┼──────┼────────┼─────┼────────┼───────┼───────┼──────┼────────┼─────┤
│ │ 3 │ 3 │ │ 3 │ 1 │ 193 │ │ │ 41 MIB │ │ │ 0 │ │ │
╰──────┴─────────┴──────┴─────────┴─────┴───────┴──────┴────────┴─────┴────────┴───────┴───────┴──────┴────────┴─────╯
╭────────────────────────────────────────────────────────────────────────────╮
│ Cluster Overview │
├─────────┬────────────┬───────────────────┬───────────────────┬─────────────┤
│ Cluster │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ west │ 1 │ 2 │ 2 │ 0 │
│ east │ 1 │ 2 │ 2 │ 0 │
│ central │ 1 │ 2 │ 2 │ 1 │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ │ 3 │ 6 │ 6 │ 1 │
╰─────────┴────────────┴───────────────────┴───────────────────┴─────────────╯
╭───────────────────────────────────────────────────────────────────────────────────────────────╮
│ JetStream Summary │
├────────┬─────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server │ Cluster │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ n2 │ central │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ n1 │ east │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ n3* │ west │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
├────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ │ │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
╰────────┴─────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯
╭────────────────────────────────────────────────────────────╮
│ RAFT Meta Group Information │
├──────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name │ ID │ Leader │ Current │ Online │ Active │ Lag │
├──────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ n1 │ fjFyEjc1 │ │ true │ true │ 412ms │ 0 │
│ n2 │ 44jzkV9D │ │ true │ true │ 412ms │ 0 │
│ n3 │ BXScrY9i │ yes │ true │ true │ 0s │ 0 │
╰──────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
19:31:04 Listening on "greet" in group "NATS-RPLY-22"
19:31:05 Sending request on "greet"
19:31:05 [#0] Received on subject "greet":
hello from east
19:31:05 Received with rtt 509.833µs