go-ricochet server
A store-and-forward messaging server for P2P networks, built on libp2p. Provides reliable message delivery when recipients are offline, following an email MX server architecture.
go-ricochet is the Go implementation of the Ricochet protocol (module github.com/stephanfeb/go-ricochet), designed for decentralized messaging, document storage, and mailbox management over peer-to-peer networks.
Features
Section titled “Features”- Store-and-Forward Messaging – Messages are stored on the server until the recipient comes online to retrieve them
- Mailbox Management – Private, shared, and public mailboxes with ACL-based access control
- Document Store – Key-value document storage with ETag-based conditional operations, versioning, and merge-patch support. Documents, feeds, collections and the directory are readable by any peer; see What Any Peer Can Read
- End-to-End Encryption – NaCl box encryption (X25519 + XSalsa20-Poly1305) derived from Ed25519 identity keys
- LZ4 Compression – Transparent payload compression with configurable threshold
- Push Notifications – Hybrid delivery: direct P2P streams for private mailboxes, GossipSub for shared/public
- Relay Support – Circuit Relay v2 service, on by default: clients behind NAT reach each other through it, and go direct by hole punching where their NATs allow. AutoRelay through static relays, and hole punching for the server’s own connections, are off by default and take their relay candidates from
bootstrap_peers; see NAT Traversal - IMAP-Style Flags – Seen, flagged, deleted, draft flags with expunge support
- Presence Detection – Real-time peer online/offline status with TTL-based caching
- Service Discovery – GossipSub-based server announcements and heartbeats; a server whose self-check fails leaves the list at its next heartbeat, and one that goes silent drops out after two hours
- PostgreSQL Storage – Production-grade storage with BYTEA binary encoding and connection pooling
Architecture
Section titled “Architecture”The system follows an email-inspired agent separation:
+-----------------------+ Client | Server | ------ | ------ | | | SendMessage() --------> | MSA (Submit) | | | | | v | | MTA (Route) | | | | | v | | MDA (Deliver) | | | | RetrieveMessages() <--- | MAA (Access) | | | CreateMailbox() ------> | MMA (Admin) | | | PutDocument() --------> | SDA (Documents) | +-----------------------+| Agent | Protocol ID | Purpose |
|---|---|---|
| MSA (Mail Submission) | /sf-network/submit/1.0.0 |
Message submission (write path) |
| MAA (Mail Access) | /sf-network/access/1.0.0 |
Message retrieval, flags, expunge, claim |
| MMA (Mailbox Management) | /sf-network/admin/1.0.0 |
Mailbox lifecycle, ACLs, capacity |
| MSA batch | /sf-network/submit/batch/1.0.0 |
Many submissions in one request |
| SDA (Store-Document) | /ricochet/store/doc/1.0.0 |
Document CRUD operations |
| SFA (Store-Feed) | /ricochet/store/feed/1.0.0 |
Append-only feeds |
| SCA (Store-Collection) | /ricochet/store/collection/1.0.0 |
Keyed collections with queries |
| MOA (Moderator Operations) | /ricochet/moderation/1.0.0 |
Ban and unban peers, remove posts, read the connection log; moderators only |
| Notify | /ricochet/mailbox-notify/1.0.0 |
Push notifications, server to client |
| MTA (Mail Transfer) | Internal | Message routing and validation |
| MDA (Mail Delivery) | Internal | Local storage and push notifications |
Quick Start
Section titled “Quick Start”Prerequisites
Section titled “Prerequisites”- Go 1.26.10+ (1.26.10 fixes GO-2026-6809 in crypto/x509; an older
godownloads the toolchain itself under the defaultGOTOOLCHAIN=auto) - PostgreSQL 14+
Database Setup
Section titled “Database Setup”The schema grants everything to a ricochet role. Create it with a password
first; the schema creates it without one if it is missing, so the file runs on
a fresh cluster either way.
psql postgres -c "CREATE ROLE ricochet LOGIN PASSWORD 'secret'"createdb ricochetpsql ricochet < schema.sqlBuild and Run
Section titled “Build and Run”# Buildgo build -o ricochet ./cmd/ricochet
# Run in development mode (TLS to PostgreSQL off for a local database)export RICOCHET_PG_PASSWORD=secret./ricochet --development \ --pg-host localhost \ --pg-database ricochet \ --pg-username ricochet \ --pg-sslmode disable
# Run in production mode (TLS to PostgreSQL required by default)export RICOCHET_PG_PASSWORD=secret./ricochet --production \ --pg-host db.example.com \ --pg-database ricochet \ --pg-username ricochetThe PostgreSQL password comes from the RICOCHET_PG_PASSWORD environment
variable or the config file. A --pg-password flag also exists, but the
command line of a running process is readable by every local user, so keep it
to throwaway local runs.
CLI Options
Section titled “CLI Options”--port Listen port (default: 55223)--development Development preset (1 GB storage, 100 connections, TLS to PostgreSQL off)--production Production preset (50 GB storage, 10,000 connections, pool of 50)--high-capacity High-capacity preset (100 GB storage, 50,000 connections, forwarding on, 8 push workers)--config YAML config file (default: /etc/ricochet/config.yaml when it exists)--data-dir Data directory path--identity-file Path to Ed25519 identity key file--transports Comma-separated transports to run (default: udx; see Transports)--listen-addrs Comma-separated multiaddrs to listen on--external-ip Public IP of this host; every listen address is advertised on it--external-addrs Comma-separated external multiaddrs to advertise--debug-dht Verbose DHT logging--pg-host PostgreSQL host--pg-port PostgreSQL port (default: 5432)--pg-database PostgreSQL database name--pg-username PostgreSQL username--pg-password PostgreSQL password (prefer RICOCHET_PG_PASSWORD)--pg-sslmode PostgreSQL SSL mode: require (default), disable--version Print the version and exitClient Library
Section titled “Client Library”The pkg/client package provides a full-featured client for interacting with Ricochet servers. The message model it speaks (messages, flags, priorities, mailbox types, acknowledgements, notifications, status codes) is in pkg/wire, so both packages are importable from any module. The snippets below compile as written; CI checks them.
For a complete program, see examples/mailbox: a command-line client that sends end-to-end encrypted messages to offline peers, collects them, and receives push notifications, with a Docker setup that runs a server and two clients behind NAT routers on one machine.
Creating a Client
Section titled “Creating a Client”import ( "github.com/libp2p/go-libp2p" client "github.com/stephanfeb/go-ricochet/pkg/client" "github.com/stephanfeb/go-ricochet/pkg/wire")
// Create a libp2p hosth, _ := libp2p.New()
// Create the Ricochet clientcl := client.New(h, client.Config{ PreferredServers: []client.ServerPreference{ {PeerID: serverPeerID, Priority: 1}, }, ConnectionTimeout: 10 * time.Second, MessageTimeout: 30 * time.Second,})defer cl.Close() // removes the client's handlers; the host stays yours to closeServers are tried in priority order and a server that cannot be dialled is
passed over for the next, so listing more than one keeps the client working
while one is down. A server that answers is never second-guessed: a refusal
comes back from the first server that gave one. Naming a server explicitly
(WithDocServer and friends) makes exactly one attempt.
Every failure the server reports arrives as a typed error: errors.Is(err, client.ErrForbidden), client.ErrNotFound, client.ErrConflict and so on,
with the status and message on *client.ProtocolError. The delete calls
return false, nil only for something that was already absent; a refusal or
a server fault is an error, never “deleted”. GetDocument is the one call
that reports its status in the result, so a conditional get can see a 304.
Sending Messages
Section titled “Sending Messages”// Simple messageresult, err := cl.SendMessage(ctx, recipientID, []byte("hello"))
// With optionsresult, err = cl.SendMessage(ctx, recipientID, payload, client.WithFolderPath("orders"), client.WithPriority(wire.PriorityUrgent), client.WithExpiry(24 * time.Hour), client.WithCompression(), // LZ4, default 1KB threshold client.WithEncryption(), // NaCl box E2E encryption)Retrieving Messages
Section titled “Retrieving Messages”// Retrieve from inboxmessages, err := cl.RetrieveMessages(ctx)
// With filtersmessages, err = cl.RetrieveMessages(ctx, client.WithRetrieveFolderPath("orders"), client.WithFromSequence(100), client.WithMaxMessages(50), client.WithMinPriority(wire.PriorityHigh),)Message Flags
Section titled “Message Flags”// Acknowledge: removes non-persistent messages, marks persistent ones seen.// Retrieval never removes anything on its own.ack, err := cl.MarkDelivered(ctx, []string{msg.MessageID})
// Page through a large mailbox (default page 100, never more than one frame)page, err := cl.RetrievePage(ctx, client.WithMaxMessages(50))for page.HasMore { last := page.Messages[len(page.Messages)-1] page, err = cl.RetrievePage(ctx, client.WithMaxMessages(50), client.WithFromSequence(last.SequenceNumber+1))}
// Flag a messageflagAck, err := cl.UpdateFlags(ctx, msg.MessageID, uint32(wire.MsgFlagFlagged), 0)
// Mark for deletionflagAck, err = cl.UpdateFlags(ctx, msg.MessageID, uint32(wire.MsgFlagDeleted), 0)
// Permanently remove deleted messagesexpungeAck, err := cl.Expunge(ctx)
// Immediate delete (bypasses flag/expunge)deleteAck, err := cl.DeleteMessages(ctx, []string{msg.MessageID})Mailbox Management
Section titled “Mailbox Management”// Create mailboxescl.CreateMailbox(ctx, "invoices", wire.MailboxPrivate)cl.CreateMailbox(ctx, "team-updates", wire.MailboxShared)cl.CreateMailbox(ctx, "announcements", wire.MailboxPublic, client.WithMailboxMaxMessages(500), client.WithRetentionDays(7),)
// Access controlcl.GrantAccess(ctx, "team-updates", colleaguePeerID, wire.AccessReadWrite)cl.RevokeAccess(ctx, "team-updates", colleaguePeerID)entries, _ := cl.ListACL(ctx, "team-updates")
// List and deletemailboxes, _ := cl.ListMailboxes(ctx)cl.DeleteMailbox(ctx, "old-folder")
// Server capacitycapacity, _ := cl.QueryCapacity(ctx)fmt.Printf("Usage: %.1f%%\n", capacity.UsagePercent())Document Store
Section titled “Document Store”ownerID := cl.PeerID()
// Create/replace a documentputResp, err := cl.PutDocument(ctx, ownerID, "config/settings", []byte(`{"theme":"dark"}`), client.WithContentType("application/json"),)
// Conditional put (optimistic locking)_, err = cl.PutDocument(ctx, ownerID, "config/settings", []byte(`{"theme":"light"}`), client.WithIfMatch(putResp.ETag),)
// Merge-patch update_, err = cl.PatchDocument(ctx, ownerID, "config/settings", map[string]any{"fontSize": 14},)
// Conditional get (returns 304 if unchanged)getResp, err := cl.GetDocument(ctx, ownerID, "config/settings", client.WithIfNoneMatch(putResp.ETag),)
// Metadata onlymeta, err := cl.HeadDocument(ctx, ownerID, "config/settings")
// List all documentsdocs, err := cl.ListDocuments(ctx, ownerID)
// Deletedeleted, err := cl.DeleteDocument(ctx, ownerID, "config/settings")
// A document is private to its owner until told otherwise. Publish one,// or share one with named readers; feeds and collections take the same// options (WithFeedVisibility, WithCollectionVisibility) and methods._, err = cl.PutDocument(ctx, ownerID, "profile", []byte(`{"name":"Ann"}`), client.WithVisibility(wire.VisibilityPublic))_, err = cl.SetDocumentVisibility(ctx, "config/settings", wire.VisibilityShared)access, err := cl.GrantDocumentReader(ctx, "config/settings", friendPeerID)fmt.Println(access.Visibility, access.Readers) // shared [{friend ...}]
// History is off until the owner turns it on with a PUT. From that PUT on// the document keeps up to N earlier versions (clamped to the server's// max_document_history_versions); WithHistory(0) turns it off and deletes// them. A restore is written as a new version, so it can be undone too._, err = cl.PutDocument(ctx, ownerID, "config/settings", []byte(`{"theme":"dark"}`), client.WithHistory(5))versions, err := cl.GetDocumentHistory(ctx, ownerID, "config/settings", 0) // newest first, no bodiesold, err := cl.GetDocumentVersion(ctx, ownerID, "config/settings", versions[0].VersionNumber)_, err = cl.RestoreDocumentVersion(ctx, "config/settings", old.VersionNumber)Push Notifications
Section titled “Push Notifications”cl.RegisterNotificationHandler(func(n *wire.Notification) { // MailboxPath is the folder; OwnerPeerID says whose mailbox it is. fmt.Printf("New message in %s/%s (type: %s)\n", n.OwnerPeerID, n.MailboxPath, n.MailboxType)})Configuration Presets
Section titled “Configuration Presets”| Setting | Default | Development | Production | High-Capacity |
|---|---|---|---|---|
| Storage | 10 GB | 1 GB | 50 GB | 100 GB |
| Connections (hard cap) | 10,000 | 100 | 10,000 | 50,000 |
| Messages/Mailbox | 1,000 | 1,000 | 1,000 | 1,000 |
| Folders/Owner | 100 | 100 | 100 | 100 |
| Retention | 30 days | 30 days | 30 days | 30 days |
| Authentication | off | off | off | off |
| Forwarding | off | off | off | on |
| Rate limits | off | off | off | off |
| Push workers | 4 | 4 | 4 | 8 |
Every row is enforced. Authentication is an allow-list, so no preset can turn
it on: set features.enable_authentication: true and list the peers in
security.trusted_peers, and only those peers may connect. Forwarding lets a
trusted peer submit mail on another sender’s behalf; with nobody trusted it is
on but unusable. Rate limits are off by design — the server bounds
concurrency (admission_control) and connections rather than request rate —
and can be turned on per protocol under rate_limits. Select a preset with
--development, --production or --high-capacity.
A config file (--config, or /etc/ricochet/config.yaml if present) is
applied on top of the preset; config.example.yaml documents every key. A
key the file omits keeps the preset’s value, including feature flags. A file
that cannot be read or parsed, or that contains a key the server does not
know, stops the server at startup rather than running on the preset.
Security
Section titled “Security”Who Can Read What
Section titled “Who Can Read What”Every store but the directory carries the same access model: a resource is private (its owner reads it), shared (the owner and the peers on its reader list) or public (anyone). The peer identity behind every check is the one the connection authenticated, so nothing is claimed in a request. Writes are the owner’s alone, with one exception.
| Store | Default | Readable by | Writable by |
|---|---|---|---|
| Mailboxes | private | Owner; shared mailboxes by the peers on their ACL; public mailboxes by anyone; claim mailboxes are listed by the owner and claimed from by anyone, one message per claim | Per mailbox type and ACL (a claim mailbox: owner only); see doc/MAILBOX_LIFECYCLE_AND_ACLS.md |
Documents (GET, HEAD, LIST, HISTORY) |
private | Per document, every stored version with it | Owner only (PUT, PATCH, DELETE, BATCH_PUT, ACCESS, RESTORE) |
Feeds (GET, LIST, BATCH_GET) |
public | Per feed, every entry with it | Owner only (CREATE, DELETE, ACCESS), except that any peer may APPEND to a feed its owner created as collaborative |
Collections (GET, LIST, QUERY) |
private | Per collection, every item and query with it | Owner only (CREATE, PUT, DELETE, ACCESS) |
| Directory | public | Any peer can browse and search every listing | Each peer joins, updates and leaves only its own listing |
Visibility is set when the resource is made (visibility on a document
PUT or BATCH_PUT, on a feed or collection CREATE) and changed with the
ACCESS operation, whose accessAction is get, set (with
visibility), grant or revoke (with readerPeerId); every action
answers with the visibility and reader list. A grant is kept while the
resource is private and applies once it is shared, as a mailbox ACL does.
Visibility belongs to the resource as a whole: making a document public
publishes its retained history, and making a feed private hides entries a
peer has already read. A LIST shows a caller only what it may read, and a
BATCH_GET reports a feed it may not read as that feed’s error. A refused
read is a 403, like a refused write, so a peer that guesses a path learns
that it exists and nothing more. A feed’s collaborative flag is about
appending only; a private collaborative feed takes entries from anyone and
shows them to its readers.
The server stores document, feed and collection bytes exactly as sent and never encrypts them: visibility is enforced by the server, not by a key. A stolen database volume exposes everything, so encrypt the volume or use PostgreSQL TDE in deployments that need it, and a client that must keep a document from the server itself has to encrypt it before storing it, the way the messaging layer does with NaCl box, accepting that the path, size, content type and version count stay visible.
End-to-End Encryption
Section titled “End-to-End Encryption”Messages can be encrypted client-side using NaCl box:
- Key Agreement: X25519 (derived from Ed25519 identity keys)
- Cipher: XSalsa20-Poly1305 (authenticated encryption)
- Wire Format:
"RCE2" [24-byte nonce] [box(header || payload)], where the header binds the ciphertext to the recipient peer ID, the folder path and the message ID. The client still reads the earlier unbound format,[24-byte nonce][ciphertext + Poly1305 tag], for messages sealed before the binding existed.
The server never sees the payload in plaintext when encryption is enabled; the envelope around it stays visible, as listed below. Keys are derived from the libp2p peer identity, so no additional key exchange is needed.
The Dart client (ricochet-dart-client, lib/crypto/payload_encryption.dart)
implements the same format, and both repositories carry the same test vectors
(fixed seeds, fixed nonce, the resulting ciphertext byte for byte: pkg/client
TestCrossLanguageVectors here, test/payload_encryption_test.dart there), so
a message sealed by either client opens in the other. Change the format on one
side and that side’s vector test is the one that fails first.
What this protects, and what it does not:
- Confidentiality and integrity of the payload against the server and the network. Only the recipient’s identity key opens the box.
- Placement. Because the binding is sealed inside the box, a ciphertext copied from one message cannot be presented as another: not in a different folder, not under a different message ID, not to a different recipient. A verbatim replay of the same message into the same folder, for instance after the recipient deleted it, is not detected; a client that needs that keeps a record of message IDs it has seen.
- No forward secrecy. The keys are the long-term identity keys, so a later compromise of either party’s key decrypts everything they exchanged. This is out of scope for this layer by design: an application that needs forward secrecy runs a ratcheting protocol such as Signal over these messages, as the OverNode client does, and uses this layer, if at all, as a second envelope.
- Deniable, not signed. NaCl box authenticates the sender to the recipient only; a recipient cannot prove to a third party who wrote a message.
- The envelope is not encrypted. Sender, recipient, folder path,
timestamps, priority and flags are visible to the server, and the
encryptedflag itself is not authenticated: stripping it makes the recipient fail to decrypt, which is a denial of service and nothing more. - One key, several uses. The same Ed25519 identity signs the Noise handshake and GossipSub messages and, converted to X25519, does the key agreement here.
Trusted Peers
Section titled “Trusted Peers”security.trusted_peers is one list with two uses. With
enable_authentication on it is the set of peers allowed to connect at all;
an untrusted peer costs one Noise handshake and is refused before any stream
opens. With enable_forwarding on it is the set of peers allowed to submit a
message whose sender is not themselves: the message is marked forwarded, its
hop count advanced, and it is refused past the hop limit.
Moderation
Section titled “Moderation”security.moderators names the peers that may use the moderation protocol,
/ricochet/moderation/1.0.0 (operations ban, unban, getBan,
listBans, removePost, connections); anyone else is refused before the request is read.
A banned peer’s open connections are closed and the server never dials it
again. It may connect, but only to learn it is banned and why: it reaches
only the Ricochet protocols (no DHT, GossipSub or relay), and every request
there is refused with a banned answer except reading, acknowledging and
deleting the messages in its own trust-safety folder, where Trust &
Safety’s ban notice is (with pkg/client, RetrieveMessages with
WithRetrieveFolderPath("trust-safety"), MarkDeliveredInFolder and
DeleteMessages). Its directory listing is removed, and its
documents, feeds and collections are hidden from everyone else. Nothing of theirs is
deleted, so unban brings it all back. Bans are kept in peer_bans and
loaded at start. removePost deletes one microblog post by its id: the
author’s entry and their edits and reactions of it, the post’s replies
thread, and its images. Every change a moderator makes is logged at WARN
with the moderator’s peer ID.
Connection Log
Section titled “Connection Log”The server records every connection: the peer ID, the IP it came from, the
remote multiaddr, the transport (udx, tcp, quic-v1, ws, …), the
direction (inbound or outbound), and when it connected and disconnected.
A relayed connection records the relay’s IP, with transport p2p-circuit.
Rows are kept in peer_connections for audit.connection_log_days (90 by
default) and purged hourly after that; 0 turns the log off. It exists so
an abuse report or a child-safety referral can tie a peer to an IP address
at a time. Nobody reads it but moderators, through the moderation protocol:
{"operation": "connections", "peerId": "12D3KooW...", "since": "2026-10-01T00:00:00Z", "limit": 500}since (RFC 3339) and limit (default 500, at most 2000) are optional. The
answer lists the peer’s connections newest first, with the retention in
force (0 when the log is off):
{"success": true, "retentionDays": 90, "connections": [ {"peerId": "12D3KooW...", "ip": "203.0.113.7", "remoteAddr": "/ip4/203.0.113.7/udp/55223/udx", "transport": "udx", "direction": "inbound", "connectedAt": "2026-10-16T09:12:03Z", "disconnectedAt": null}]}disconnectedAt is null while the connection is open, or when the server
stopped without seeing it close. Recording never holds up a connection: the
rows are written in the background, and dropped with a warning if the
database falls behind.
Relayed Topics
Section titled “Relayed Topics”GossipSub forwards a topic only through peers that announce it. Two peers
connected only through this server – a service publishing on a topic and an
app subscribed to it – exchange nothing on a topic the server does not use.
pubsub.relay_topics lists topics the server joins and relays without
subscribing: it forwards their messages and never delivers them to itself. A
bootstrap shared by OverMedia and the overnode app should relay
/overmedia/service-announce. A listed topic the server already uses is
skipped with a warning; it is forwarded anyway.
Identity Is Free, So Per-Peer Limits Are Fairness Controls
Section titled “Identity Is Free, So Per-Peer Limits Are Fairness Controls”A peer identity is an Ed25519 key, which costs nothing to mint, so every
limit keyed by peer ID – the per-peer rate limiters, the per-peer admission
slots, mailbox quotas – bounds what one well-behaved client can consume,
not what a determined one can. The abuse controls are the ones a client
cannot rotate out of: max_concurrent_connections caps the server as a
whole, and admission control bounds work in flight against the database
regardless of how many identities ask for it.
The one limit tied to a source address is max_connections_per_ip, and it
is off by default. Phones behind a carrier NAT and desks behind an office
gateway share an address, and any cap turns the clients past it away.
go-libp2p applies its own per-address defaults, eight connections and
0.2 new connections a second, to every host; this server replaces them with
no per-address cap and no per-address rate limit, because it bounds
concurrency, never rate. Set max_connections_per_ip only where one address
is known to be one client.
Transport Security
Section titled “Transport Security”UDX, TCP and WebSocket connections use the Noise protocol framework for transport encryption and mutual authentication; QUIC and WebTransport use libp2p’s TLS 1.3 handshake. Either way the peer ID is authenticated by the handshake.
Identity
Section titled “Identity”Server and client identities are Ed25519 keypairs. Identity can be provided via:
RICOCHET_SEED_HEXenvironment variable (highest priority)--identity-fileCLI flag- Auto-generated and persisted to
{data-dir}/peer_identity.key
Dependency Advisories
Section titled “Dependency Advisories”CI runs govulncheck on every push. Two advisories it reports have no
upstream fix, and neither reaches code this server runs:
- GO-2026-4479 (
pion/dtls/v2, AES-GCM nonce reuse) is linked through go-libp2p’s WebRTC transport. The host is built withlibp2p.NoTransportsand only the transports inserver.transports, andwebrtc-directis refused there, so no DTLS handshake ever happens. - GO-2024-3218 (
go-libp2p-kad-dht, content censorship via Sybil peers) concerns IPFS content routing: provider records for a CID. Ricochet uses the DHT for peer routing only and never publishes or looks up provider records, so there is nothing to censor.
Networking
Section titled “Networking”Transport Stack
Section titled “Transport Stack”Application (Ricochet protocols) |Yamux (stream multiplexing) |Noise (transport encryption) |UDX (reliable UDP transport: ordered streams, congestion and flow control) |UDPThat is the default stack, and the one the mobile clients speak. The server
can run other go-libp2p transports beside it or instead of it, named in
server.transports (or --transports):
| Transport | Listen address | Notes |
|---|---|---|
udx (default) |
/ip4/0.0.0.0/udp/55223/udx |
Noise + yamux over UDX. |
tcp |
/ip4/0.0.0.0/tcp/55223 |
Noise + yamux over TCP. |
quic-v1 |
/ip4/0.0.0.0/udp/55224/quic-v1 |
TLS 1.3 and streams from QUIC itself. |
websocket |
/ip4/0.0.0.0/tcp/55225/ws, /ip4/0.0.0.0/tcp/443/tls/ws |
Noise + yamux. Secure WebSocket needs server.websocket_tls (a PEM certificate and key, reloaded when the file changes), or TLS terminated in a reverse proxy that the /ws listener sits behind. |
webtransport |
/ip4/0.0.0.0/udp/55224/quic-v1/webtransport |
For browsers; certificate hashes, no CA. |
Every listen and external address needs its transport in the list; a
transport with no listen address only dials. server.external_ip (or
--external-ip) advertises every listen address on the host’s public IP, so
a server behind a 1:1 NAT or a cloud VM’s private interface needs one setting
whatever transports it runs. UDX and the QUIC family
(quic-v1, webtransport) each hold their own UDP socket, so they need
different UDP ports; QUIC and WebTransport share one, and so do TCP and
WebSocket (go-libp2p’s shared TCP listener, turned on when both run). TCP and
UDP may use the same number. --port moves every listen address to one port, so a server
running UDX beside QUIC sets its ports in listen_addresses instead. The
server refuses at start a configuration that breaks any of these rules.
webrtc-direct is refused (see Dependency Advisories).
The yamux keepalive (15 s) and write timeout (10 s) apply to UDX, TCP and WebSocket. QUIC and WebTransport keep their connections alive on quic-go’s own 15-second period.
The server runs a Kademlia DHT in server mode for peer routing. A peer enters the server’s routing table only while the server holds a direct connection to it that the server opened: a routing table entry is a promise to query that peer on its own, and an inbound connection from a client behind a NAT proves nothing about whether the server can reach it. Relayed connections never qualify. Bootstrap peers are dialled at start, and peers learned from lookups are dialled by the DHT, so the peers that belong in the table arrive over outbound connections; a peer that only ever dialled this server stays out until this server dials it.
NAT Traversal
Section titled “NAT Traversal”The relay service is on by default, so a public server relays for the peers
behind NATs. Two clients that cannot reach each other connect through it, and
then try a hole punch (DCUtR) for a direct connection. Hole punching is the
clients’ to turn on (dart-libp2p has it on by default); the server needs no
setting for it, since the relay carries the clients’ coordination like any
other traffic. Behind a cone NAT the punch succeeds; a symmetric NAT defeats
it and the connection stays on the relay. test/nat/run.sh checks this in the
mailbox example’s Docker setup.
The relay does not cut a relayed connection: it carries it for as long, and as
much, as the two peers want, and they see an ordinary connection rather than a
limited one. An operator who wants to bound what one relayed connection may
cost sets relay_limits.connection_duration_sec or
relay_limits.connection_data (see config.example.yaml); go-libp2p’s own
defaults, 2 minutes and 128 KiB, are only enough to coordinate a hole punch.
A server that is itself behind a NAT reaches the network through other relays; that is off by default and configured in the YAML file, with the relay candidates taken from the bootstrap peers:
features: enable_relay: true # Circuit Relay v2 (default) enable_auto_relay: true # reserve slots on the static relays below enable_hole_punching: true # DCUtR for connections relayed to this serverserver: bootstrap_peers: - /ip4/relay.example.com/udp/55223/udx/p2p/12D3KooW...Service Discovery
Section titled “Service Discovery”Servers announce themselves via GossipSub on the /sf-network/services/announce topic every intervals.service_announcement_min. Announcements include capabilities, storage capacity and region.
Between announcements, every intervals.health_check_min a server runs the readiness checks behind /readyz (today: a database ping) and publishes the verdict on /sf-network/services/heartbeat. Other servers refresh the entry from the heartbeat, so a healthy server never goes stale, and drop a server from the available list as soon as it reports a failed check. The uptime score in both messages is measured: the share of the server’s last 288 checks that passed, 24 hours at the default interval. Both messages are accepted only from the server they name, verified by the GossipSub signature.
Project Structure
Section titled “Project Structure”cmd/ricochet/ Server CLI entry pointcmd/ricochet-bench/ Load testing toolexamples/mailbox/ Example client: offline messages, push, and a Docker setup with NATinternal/ admission/ Admission control (bounded work in flight) and the shutdown drain audit/ Connection log: which IP each peer connected from, and when capacity/ Storage capacity sampling core/ Message types, config, mailbox addressing metrics/ Prometheus collectors moderation/ Banned peers, the moderators, and the gater that refuses bans mda/ Mail Delivery Agent + push notifier mailboxes/ Mailbox types, ACL and retention rules mta/ Mail Transfer Agent (routing, forwarding) opsapi/ Operator HTTP surface (health, readiness, metrics, pprof) opsview/ Stored-data views under /ops presence/ Peer presence detection + cache protocol/ frame/ Length-prefix frame encoding wire/ Shared pipeline middleware, status codes, authorization msa/ Mail Submission Agent (write path, single and batch) maa/ Mail Access Agent (read path) mma/ Mailbox Management Agent (admin) moa/ Moderator Operations Agent (bans, post removal, connection log) sda/ Store-Document Agent (documents) sfa/ Store-Feed Agent (feeds) sca/ Store-Collection Agent (collections) notify/ Push notification protocol protocoltest/ In-process pipeline harness for handler tests ratelimit/ Optional per-protocol rate limiters registry/ Service discovery via GossipSub relay/ Relays GossipSub topics the server does not use (pubsub.relay_topics) server/ Server orchestration and lifecycle storage/ Storage interface and models postgres/ PostgreSQL backend storagetest/ In-memory Storage for unit tests transports/ Transport names, constructors and address checks trust/ Trusted-peer set and connection gatepkg/client/ Public client library client.go Messages, mailboxes, documents collections.go Collection operations feeds.go Feed operations options.go Functional options errors.go Typed errors compression.go LZ4 compression encryption.go NaCl box encryption notifications.go Push notification handlerpkg/wire/ Wire model shared by client and servertest/integration/ Integration tests (requires PostgreSQL)deploy/ Supervisord and Debian deployment filesbuild/ Package build outputdoc/ Design notes, audits and the backlogschema.sql PostgreSQL database schemaStress Testing
Section titled “Stress Testing”ricochet-bench is an Apache Bench-style load testing tool for Ricochet servers. It measures throughput, latency percentiles, and error rates for every protocol the server serves.
go build -o ricochet-bench ./cmd/ricochet-benchricochet-bench [flags] <server-multiaddr> <server-peer-id>Flags:
| Flag | Default | Description |
|---|---|---|
-n |
1000 | Total number of requests |
-c |
10 | Number of concurrent workers |
-protocol |
msa |
Protocol to benchmark (see below) |
-payload-size |
1024 | Payload size in bytes |
-duration |
— | Run for a duration instead of fixed count (e.g. 30s, 1m) |
-warmup |
10 | Warmup requests before measuring |
-batch-size |
100 | Documents or messages per request, for the batch scenarios |
-docs |
500 | Vault size for the sync scenarios |
-v |
off | Verbose output (libp2p and UDX diagnostic logs) |
Protocols:
| Protocol | Operation | Description |
|---|---|---|
msa |
SendMessage |
Message submission throughput |
maa |
RetrieveMessages |
Message retrieval (pre-seeds mailbox data) |
sda |
PutDocument + GetDocument |
Document store round-trip |
sfa |
AppendFeedEntry + GetFeedEntry |
Feed append and read cycle |
sca |
PutCollectionItem + QueryCollection |
Collection write and query cycle |
mma |
CreateMailbox + DeleteMailbox |
Mailbox admin round-trip |
mixed |
Random mix of msa, maa, sda, sfa, sca, mma |
Combined workload |
sda-batch |
PutDocuments |
Batched document writes, -batch-size per request |
msa-batch |
SendMessages |
Batched message submission, -batch-size per request |
sync-cold |
PutDocument × -docs |
First upload of a vault, every document new |
sync-warm |
PutDocument × -docs |
Re-upload after edits, every document replaced |
sync-noop |
ListDocuments + compare |
Nothing changed: list and compare ETags, write nothing |
Examples
Section titled “Examples”# Basic message submission benchmarkricochet-bench -n 1000 -c 10 -protocol msa \ /ip4/127.0.0.1/udp/55223/udx 12D3KooW...
# Document store with larger payloadsricochet-bench -n 5000 -c 20 -protocol sda -payload-size 4096 \ /ip4/127.0.0.1/udp/55223/udx 12D3KooW...
# Duration-based mixed workloadricochet-bench -duration 60s -c 50 -protocol mixed \ /ip4/127.0.0.1/udp/55223/udx 12D3KooW...Sample Output
Section titled “Sample Output”A run against a development server on the same laptop, PostgreSQL local:
======================================== Ricochet Bench - MSA (Message Submission)========================================Server: /ip4/127.0.0.1/udp/55999/udx/p2p/12D3KooW...oY6B3C4R
[1/4] Creating 10 workers... done[2/4] Connecting to server... connected[3/4] Warming up (10 requests)... done[4/4] Benchmarking 1000 requests with 10 workers...======================================== RESULTS========================================Concurrency Level: 10Total Requests: 1000Payload Size: 1024 bytes
Completed: 1000 Failed: 0 Total time: 1.297s Requests/sec: 770.86
Latency Distribution: min: 628.0us p50: 2.0ms p75: 6.8ms p90: 25.8ms p95: 76.6ms p99: 188.7ms max: 397.4ms========================================Ten workers on one machine do not saturate the server; doc/BASELINES.md
records the throughput it reaches with more.
Architecture Notes
Section titled “Architecture Notes”Each concurrent worker creates its own libp2p host and client connection, avoiding yamux stream multiplexing contention. For protocols that require pre-existing data (MAA, SFA, SCA), the tool creates the necessary resources before the warmup and the measured run.
Testing
Section titled “Testing”# Run the unit tests (no database needed; packages that want one skip without it)go test $(go list ./... | grep -v test/integration)
# Run integration tests (requires PostgreSQL)RICOCHET_TEST_POSTGRES_DSN="postgresql://user:pass@localhost:5432/ricochet_test?sslmode=disable" \ go test ./test/integration/...
# Run all testsgo test ./...
# With verbose outputgo test -v ./pkg/client/...Test Coverage
Section titled “Test Coverage”- Unit tests: the six protocol handlers driven in-process against an in-memory store, mailbox and ACL rules, routing, the client against loopback servers, server lifecycle, config parsing, frame codec, compression and encryption. The server, storage and delivery packages also carry tests that run only when
RICOCHET_TEST_POSTGRES_DSNis set. - Integration tests: full client-server workflows over UDX against PostgreSQL: messaging, flags and expunge, mailbox management and ACLs, documents, feeds, collections, the directory, notifications, shutdown and authorization.
Protocol Wire Format
Section titled “Protocol Wire Format”All protocol messages use a length-prefixed JSON frame:
+-------------------+--------------------+| Length (4 bytes) | JSON Payload || big-endian uint32 | (variable length) |+-------------------+--------------------+Message Processing Pipeline
Section titled “Message Processing Pipeline”Send: raw payload -> compress (LZ4) -> encrypt (NaCl box) -> encode frame -> sendReceive: receive -> decode frame -> decrypt (NaCl box) -> decompress (LZ4) -> raw payloadDatabase Schema
Section titled “Database Schema”The PostgreSQL schema (schema.sql) includes:
| Table | Purpose |
|---|---|
mailboxes |
Mailbox records with owner, type, retention policy |
stored_messages |
Messages with BYTEA payload, priority, flags, expiry |
mailbox_acls |
Per-peer access control entries |
reader_cursors |
Per-reader position tracking (public mailboxes) |
documents |
Document storage with ETag versioning and a visibility |
document_versions |
Document version history |
document_acls |
Reader list of a shared document |
directory_listings |
Peer directory entries for search |
peer_bans |
Peers a moderator banned, loaded at start |
peer_connections |
The connection log: peer, IP, transport, connect and disconnect times |
feeds |
Append-only feeds, with a visibility |
feed_entries |
Feed entries with sequence numbers |
feed_acls |
Reader list of a shared feed |
collections |
Keyed collections, with a visibility |
collection_items |
Collection items with JSONB content and versions |
collection_acls |
Reader list of a shared collection |
block_store |
Reserved for content-addressed body offload (see the scale roadmap); unused today |
License
Section titled “License”Apache License 2.0. See LICENSE for the full text.