NATS Streaming to JetStream Migration (CLI)

NATS Streaming to JetStream Migration in Operations

This example demonstrates how to perform a migration of NATS Streaming channels and subscriptions to NATS JetStream streams and consumers using the stan2js migration tool. This is a one time exercise and stan2js only needs to be run once for each set of STAN channels that require migration to JetStream streams.

CLI Go Python JavaScript Rust C# .NET V2 Java Ruby Elixir Crystal C

Jump to the output or the recording

$ nbe run operations/stan2js/cli

View the source code or learn how to run this example yourself

Code

#!/bin/bash


set -euo pipefail

Save connection contexts

The NATS CLI can be used to save and manage contexts which declare the server(s) to connect to as well as authentication and TLS options. For this example, we will create two contexts for the NATS and STAN servers. However, if channels and/or subscriptions need to be migrated to different accounts, then multiple contexts can be used.

echo 'Saving contexts...'
nats context save stan \
  --server $STAN_URL


nats context save nats \
  --server $NATS_URL

Generate STAN data

To showcase the migration, three channels and subscriptions will be used. The first channel, foo, has a max_msgs limit of 400 and one subscription. The second channel, bar, has a max_msgs limit of 500 and two subscriptions, one of which is a queue. The third channel, baz, has no limits and no subscriptions. 1000 messages are published to all channels.

echo 'Generating STAN data...'
generate-stan-data \
  --context stan \
  --cluster $STAN_CLUSTER \
  --client app

Define the migration config

Define a config file required by stan2js to declare which channels and subscriptions are to be migrated, as well as some associated stream and consumer options. These options are not exhaustive since many options on streams and consumers can be changed after the migration. Refer to the stan2js README for the full set of options.

echo 'Creating config file...'
cat <<-EOF > config.yaml
stan: stan
nats: nats
cluster: test-cluster
client: stan2js


channels:
  foo:
    stream:
      name: FOO
      replicas: 1


  bar:
    stream:
      name: BAR
      max_consumers: 10
      max_bytes: 1GB


  baz:
    stream:
      name: BAZ
      max_age: 1h


clients:
  app:
    context: stan
    subscriptions:
      sub-foo:
        channel: foo
        consumer:
          name: foo
          pull: true
      
      
      sub-bar-q:
        channel: bar
        queue: q
        consumer:
          name: bar-queue
          queue: bar-queue


      sub-bar:
        channel: bar
        consumer:
          name: bar-consumer
EOF

Run the migration!

echo 'Running migration...'
stan2js config.yaml

Verify the migration

Since the NATS CLI supports introspecting JetStream assets, we can use it to verify that the migration was successful. Also we can inspect the streams to confirm the headers and data look correct.

echo 'Report the streams...'
nats --context nats stream report


echo 'View 3 messages from FOO...'
nats --context nats stream view FOO 3


echo 'View 3 messages from BAR...'
nats --context nats stream view BAR 3


echo 'View 3 messages from BAZ...'
nats --context nats stream view BAZ 3


echo 'Report the consumers...'
nats --context nats consumer report FOO
nats --context nats consumer report BAR
nats --context nats consumer report BAZ

Output

Saving contexts...
NATS Configuration Context "stan"

      Server URLs: nats://stan:4222
             Path: /root/.config/nats/context/stan.json

WARNING: Shell environment overrides in place using NATS_URL
NATS Configuration Context "nats"

      Server URLs: nats://nats:4222
             Path: /root/.config/nats/context/nats.json

WARNING: Shell environment overrides in place using NATS_URL
Generating STAN data...
Creating config file...
Running migration...
╭─────────────────────────────────────────────╮
│ Channels -> Streams                         │
├────────────┬────────────────┬───────────────┤
│ NAME       │ FIRST SEQUENCE │ LAST SEQUENCE │
├────────────┼────────────────┼───────────────┤
│ bar -> BAR │ 301 -> 1       │ 1000 -> 700   │
│ baz -> BAZ │ 1 -> 1         │ 1000 -> 1000  │
│ foo -> FOO │ 501 -> 1       │ 1000 -> 500   │
╰────────────┴────────────────┴───────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Subscriptions -> Consumers                                                                             │
├────────┬───────────────────┬─────────────────────────┬────────────┬────────────────────┬───────────────┤
│ CLIENT │ CHANNEL -> STREAM │ NAME                    │ QUEUE NAME │ CONVERTED TO PULL? │ NEXT SEQUENCE │
├────────┼───────────────────┼─────────────────────────┼────────────┼────────────────────┼───────────────┤
│ app    │ bar -> BAR        │ sub-bar -> bar-consumer │            │ false              │ 701 -> 401    │
│ app    │ bar -> BAR        │ sub-bar-q -> bar-queue  │ bar-queue  │ false              │ 351 -> 51     │
│ app    │ foo -> FOO        │ sub-foo -> foo          │            │ true               │ 601 -> 101    │
╰────────┴───────────────────┴─────────────────────────┴────────────┴────────────────────┴───────────────╯
Report the streams...
Obtaining Stream stats

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                       Stream Report                                       │
├────────┬─────────┬───────────┬───────────┬──────────┬─────────┬──────┬─────────┬──────────┤
│ Stream │ Storage │ Placement │ Consumers │ Messages │ Bytes   │ Lost │ Deleted │ Replicas │
├────────┼─────────┼───────────┼───────────┼──────────┼─────────┼──────┼─────────┼──────────┤
│ FOO    │ File    │           │ 1         │ 500      │ 85 KiB  │ 0    │ 0       │          │
│ BAR    │ File    │           │ 2         │ 700      │ 119 KiB │ 0    │ 0       │          │
│ BAZ    │ File    │           │ 0         │ 1,000    │ 170 KiB │ 0    │ 0       │          │
╰────────┴─────────┴───────────┴───────────┴──────────┴─────────┴──────┴─────────┴──────────╯

View 3 messages from FOO...
[1] Subject: foo Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Channel: foo
  Nats-Streaming-Sequence: 501
  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.980419925Z

foo: 500

[2] Subject: foo Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Channel: foo
  Nats-Streaming-Sequence: 502
  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.980773425Z

foo: 501

[3] Subject: foo Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Channel: foo
  Nats-Streaming-Sequence: 503
  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.981072175Z

foo: 502

View 3 messages from BAR...
[1] Subject: bar Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Sequence: 301
  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.886964217Z
  Nats-Streaming-Channel: bar

bar: 300

[2] Subject: bar Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.887295217Z
  Nats-Streaming-Channel: bar
  Nats-Streaming-Sequence: 302

bar: 301

[3] Subject: bar Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Channel: bar
  Nats-Streaming-Sequence: 303
  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.887588425Z

bar: 302

View 3 messages from BAZ...
[1] Subject: baz Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Channel: baz
  Nats-Streaming-Sequence: 1
  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.692320384Z

baz: 0

[2] Subject: baz Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Sequence: 2
  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.693268425Z
  Nats-Streaming-Channel: baz

baz: 1

[3] Subject: baz Received: 2023-10-24T10:50:07Z

  Nats-Streaming-Channel: baz
  Nats-Streaming-Sequence: 3
  Nats-Streaming-Timestamp: 2023-10-24T10:50:06.694051675Z

baz: 2

Report the consumers...
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                Consumer report for FOO with 1 consumers                                 │
├──────────┬──────┬────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬─────────┤
│ Consumer │ Mode │ Ack Policy │ Ack Wait │ Ack Pending │ Redelivered │ Unprocessed │ Ack Floor │ Cluster │
├──────────┼──────┼────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼─────────┤
│ foo      │ Pull │ Explicit   │ 30.00s   │ 0           │ 0           │ 400 / 80%   │ 0         │         │
╰──────────┴──────┴────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴─────────╯

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                  Consumer report for BAR with 2 consumers                                   │
├──────────────┬──────┬────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬─────────┤
│ Consumer     │ Mode │ Ack Policy │ Ack Wait │ Ack Pending │ Redelivered │ Unprocessed │ Ack Floor │ Cluster │
├──────────────┼──────┼────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼─────────┤
│ bar-consumer │ Push │ Explicit   │ 30.00s   │ 0           │ 0           │ 300 / 42%   │ 0         │         │
│ bar-queue    │ Push │ Explicit   │ 30.00s   │ 0           │ 0           │ 650 / 92%   │ 0         │         │
╰──────────────┴──────┴────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴─────────╯

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                Consumer report for BAZ with 0 consumers                                 │
├──────────┬──────┬────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬─────────┤
│ Consumer │ Mode │ Ack Policy │ Ack Wait │ Ack Pending │ Redelivered │ Unprocessed │ Ack Floor │ Cluster │
├──────────┼──────┼────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼─────────┤
╰──────────┴──────┴────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴─────────╯

Recording

Note, playback is half speed to make it a bit easier to follow.