Scaling Waku: A Progress Update

A glance into how Waku has progressed its scaling capabilities throughout 2023

As we move forward into the last months 2023, our primary goal has been to solidify the scalability of Waku protocols, focusing on Status Communities.

We have set and reached our milestone of providing a Waku network that can scale to approximately 10K active users in Status Communities. While continuing to provide Status the required network services to make such a community viable, we believe that the research, protocols, tools, and solutions we are working on will allow the overall Waku network to scale to over a million nodes.

To meet Status's timeline for the Communities feature, the Waku team made some tough decisions that required tradeoffs in terms of decentralization and other design goals. However, this approach has allowed us to demonstrate the core stability and scalability of Waku's protocols, while continuously working towards long-term solutions that provide better anonymity, robustness, and decentralization features.

This article will continue diving deeper into what we did and where you can learn more about these topics. If you're already keen to get your hands dirty, we have recently published our upgraded Waku documentation, allowing us to build faster, together.

Overview

• Theoretical: Laying the groundwork

  • Theoretical Analysis of Scaling Gossipsub.
  • Static Sharding

• Bridging theory to practice

  • Simulations
  • Status Telemetry Analysis

• Hands-on achievements

  • Status MVP Delivery (Scaled Waku Communities to 10k)
  • Protocol Service Improvements
    • PostgreSQL transition
    • Waku Peer Exchange
    • Filter Protocol Redesign
  • Status DoS mitigation mechanism
  • Building for builders
    • Waku Docs upgrade
    • C API

• Contribute to the decentralisation of communication

Laying the theoretical groundwork

Theoretical Analysis of Scaling gossipsub

We started in Q1 by doing a theoretical analysis (mathematical modelling) of GossipSub scaling under certain conditions.

The numbers for our message rates and sizes chosen for the model come from the telemetric analysis of typical traffic on significantly sizable Discord servers and existing Status Communities. From this, we conclude that the Waku Relay network can scale to 10,000 nodes per shard.

Suppose we further assume that each Status Community operates on a separate shard. In that case, each community can scale to 10,000 active relay-only users while bandwidth usage remains within a reasonable range for each user.

Realistically most community nodes are not active simultaneously, so this approach could scale to ~100,000 nodes per community if roughly 1/10th of users are active at any time.

Static sharding

Based on the analyses above, we published an RFC, bringing together an integrated strategy to scale Status Communities based on static sharding. This requires manually selecting and configuring static shard(s) per community, with no sharing of shards between communities. Static sharding provides a simple way to achieve scalability with reasonable Denial of Service (DoS) protection (more on that below) but also requires some trade-offs in decentralisation and anonymity. Therefore, our longer-term strategy is to provide a way to auto-scale the Waku network with "autosharding" - this allows multiple applications and communities to share the same set of shards without intervening manually in the underlying message routing strategies.

This Github issue provides more information on our network strategy as we advance. In the near future you can expect an elaborated post on our strategy & roadmap.

Bridging theory to practice

Simulations

To test our theoretical conclusions, the Distributed Systems Testing (DST) team ran extensive simulations using a Waku simulation tool built on the Kurtosis framework, Wakurtosis.

The DST team has recently published a summary report detailing these conclusions about scalability, matching our theoretical expectations while showing that the nwaku client can deal with the resource overhead in a growing network.

While achieving full-scale simulations of 10,000 nodes posed challenges, initial simulations of up to 1000 nodes aligned with our mathematical models. This gave us confidence in the viability of our approach.

Status telemetry analysis

Having drawn theoretical conclusions under which a Waku Relay network would scale, we then experimented to verify our conclusions. Afterwards we performed a telemetric analysis of current Status Community traffic.

This allowed us to discover potential bottlenecks when sending large messages over Waku, determine major bandwith usage, and propose strategies to help Status Communities reasonably use Waku without compromising its scalability.

Hands-on achievements

Protocol service improvements

To cater to the needs of a growing network of users on diverse platforms, the team improved (and in some cases wholly redesigned) our existing protocols based on the lessons we have learned from dogfooding the Waku network. Of these, the changes to the store and filter protocols were the most significant.

PostgreSQL transition

To be able to retrieve an increasing amount of large history queries & messages from our store protocol, we have transitioned from a simple SQLite backend to a more robust PostgreSQL archive. This allows store nodes to serve multiple queries in parallel while improving message reliability through redundancy.

Filter protocol redesign

The filter protocol allows resource-restricted devices to receive only a subset of messages routed in the network. We have redesigned the entire filter protocol to be more robust and provide features requested by Waku-based applications, such as refreshing existing subscriptions.

This new filter protocol version is currently being dogfooded within both mobile and desktop production environment, and the initial feedback is prominently positive.

Waku peer exchange

We have designed and then created a discovery mechanism suitable for resource-restricted nodes called Waku peer exchange. Work on all these protocols empowers nodes, including resource-restricted devices, to participate seamlessly, ensuring that the network remains accessible to all.

Connectivity: discovering and managing peers

A Waku network can only function well if we make it efficient for nodes to connect and interact.

During the first half of 2023, we spent significant effort improving how a node discovers peers and manages its connectivity to those peers. We have boosted a node's ability to differentiate between "good" and "bad" connections, including integrating gossipsub peer scoring.

Thanks to the introduction of libp2p rendezvous discovery, nodes behind restrictive NATs can now make themselves discoverable and connect to other nodes by coordinating a NAT hole punch via a circuit-relay connection.

We have also modified discovery methods such as discv5, to consider static sharding when filtering discovered nodes.

DoS protection

One of the most important considerations of any Waku network deployment is protecting the network against spam and DOS attacks. While we are still working on productising a privacy-preserving spam protection mechanism based on rate-limiting nullifiers, we focused in H1 on developing a simple DoS mitigation mechanism for Status Communities.

This requires all messages belonging to a community to be signed with a key distributed to all community members. Since static sharding ensures messages are only routed within a community, each relayer in the Waku network can validate that each routed message truly originated from within the community and discard those that did not.

Building for builders

Docs

We are continuing to make building on Waku more accessible than ever. We have redesigned and improved our docs website, making it easier for you to get acquainted with our work.

C API

The nwaku client has recently seen the introduction of a C API that can be seamlessly wrapped in multiple programming languages (see the Python wrapper as an example), an efficient addition to our existing languages and frameworks support. By simplifying the process of integrating nwaku as a service node into other applications, we are fostering an environment where more projects can actively build on Waku and contribute to the Waku network.

Setting the stage for growth

This post provides a glimpse into the Waku Team's progress in the first half of 2023.

Our sights are currently set on taking what we have learned and building a viable public Waku Network that can scale to even larger numbers while maintaining our core principles of open-access, privacy-preservation, decentralisation, and transparency about trade-offs.

Conquering those research & development challenges will not only scale Waku, but will play a significant role in growing and scaling Status Communities. Where we now have made certain trade-offs, our end-goal and principles are aligned in our end-goals, you'll soon read more about our goals and milestones in our upcoming Roadmap.

As a team, we remain dedicated to this mandate while acknowledging the vast involvement of community members and collaborators who contribute to our journey.

Contribute to the decentralisation of communication.

Help us ensure that we are staying decentralised by running your own Waku Node!

Soon we will also release our upcoming roadmap.

Stay tuned, and follow our Twitter to keep yourself updated on new launches. Say hello to us on Discord, join our team, or crack some of our bounties --- No need to remain shy!

Written by Mf & Hanno 09/2023