Benchmarks

Benchmark results for Supabase Realtime.

This guide explores what can be expected from Realtime's Postgres Changes, broadcast and presence performance. A set of load tests demonstrates its scaling capabilities.

Methodology#

The benchmarks were conducted using k6, an open-source load testing tool, against a Realtime Cluster deployed on AWS. The cluster configurations used 2-6 nodes, tested in both single-region and multi-region setups, all connected to a single Supabase project. The load generators (k6 servers) were also deployed on AWS to minimize network latency impact on the results. Tests were executed with the full load from the start without warm-up runs.

The metrics collected include message throughput, latency percentiles, CPU and memory utilization, and connection success rates. It's worth noting that performance in production environments may vary based on factors such as network conditions, hardware specifications, and specific usage patterns.

Workloads#

The proposed workloads are designed to demonstrate Supabase Realtime's throughput and scalability capabilities. These benchmarks focus on core functionality and common usage patterns.

The benchmarking results include the following workloads:

1. Broadcast Performance
2. Payload Size Impact on Broadcast
3. Large-Scale Broadcasting
4. Authentication and New Connection Rate
5. Database Events

Results#

Realtime Broadcast Performance#

This workload evaluates the system's capacity to handle multiple concurrent WebSocket connections and message broadcasting. Each virtual user (VU) in the test:

• Establishes and maintains a WebSocket connection
• Joins two distinct channels:
  • An echo channel (1 user per channel) for direct message reflection
  • A broadcast channel (6 users per channel) for group communication
• Generates traffic by sending 2 messages per second to each joined channel for 10 minutes

Payload Size Impact#

This workload tests the system's performance with different message payload sizes to understand how data volume affects throughput and latency. Each virtual user (VU) follows the same connection pattern as the broadcast test, but with varying message sizes:

• Establishes and maintains a WebSocket connection
• Joins two distinct channels:
  • An echo channel (1 user per channel) for direct message reflection
  • A broadcast channel (6 users per channel) for group communication
• Sends messages with payloads of 1KB, 10KB, and 50KB
• Generates traffic by sending 2 messages per second to each joined channel for 5 minutes

1KB Payload#

10KB Payload#

50KB Payload#

Note: The final column shows results with reduced load (2,000 users) for the 50KB payload test, demonstrating how the system performs with larger payloads under different concurrency levels.

Large-Scale Broadcasting#

This workload demonstrates Realtime's capability to handle high-scale scenarios with a large number of concurrent users and broadcast channels. The test simulates a scenario where each user participates in group communications with periodic message broadcasts:

• Each virtual user (VU):
  • Establishes and maintains a WebSocket connection (30-120 minutes)
  • Joins 2 broadcast channels
  • Sends 1 message per minute to each joined channel
  • Each message is broadcasted to 100 other users

Realtime Auth#

This workload demonstrates Realtime's capability to handle large amounts of new connections per second and channel joins per second with Authentication Row Level Security (RLS) enabled for these channels. The test simulates a scenario where large volumes of users connect to realtime and participate in auth protected communications:

• Each virtual user (VU):
  • Establishes and maintains a WebSocket connection (2.5 minutes)
  • Joins 2 broadcast channels
  • Sends 1 message per minute to each joined channel
  • Each message is broadcasted to 100 other users

Realtime's Postgres Changes#

Realtime systems usually require forethought because of their scaling dynamics. For the Postgres Changes feature, every change event must be checked to see if the subscribed user has access. For instance, if you have 100 users subscribed to a table where you make a single insert, it will then trigger 100 "reads": one for each user.

There can be a database bottleneck which limits message throughput. If your database cannot authorize the changes rapidly enough, the changes will be delayed until you receive a timeout.

Database changes are processed on a single thread to maintain the change order. That means compute upgrades don't have a large effect on the performance of Postgres change subscriptions. You can estimate the expected maximum throughput for your database below.

If you are using Postgres Changes at scale, you should consider using a separate "public" table without RLS and filters. Alternatively, you can use Realtime server-side only and then re-stream the changes to your clients using a Realtime Broadcast.

Enter your database settings to estimate the maximum throughput for your instance:

Don't forget to run your own benchmarks to make sure that the performance is acceptable for your use case.

Supabase continues to make improvements to Realtime's Postgres Changes. If you are uncertain about your use case performance, reach out using the Support Form. The support team can advise on the best solution for each use-case.