--- name: cryptopeer-interop-testing description: Use when writing or running tests that span languages or processes in the cryptopeer stack: Dart, Go and JS interop for UDX and libp2p, loopback and in-memory test hosts, Docker NAT and netem setups, browser and TURN tests, timeouts and flaky network tests, and the dart test, go test and vitest commands and CI presets of each repo. triggers: interop test, interop_test.go, go_interop, GoProcessManager, go-peer, dart_test.yaml, vitest, UDX_BULK_PEER, GO_PEER_DIR, netem, holepunch_network, FakeBleNetwork, forgetest, real_net_stack --- # Testing across languages and processes The cryptopeer stack has three implementations of UDX (dart-udx, go-udx, js-udx) and two libp2p stacks that talk to each other (dart_libp2p and go-libp2p through go-libp2p-udx-transport, plus js-libp2p through `@go-ricochet/libp2p-udx`). This skill covers the tests that run across these: what exists, how to run it, how to write a new one, and how to keep it stable. For the API of a single package, load its own skill (`udx-dart`, `udx-go`, `udx-js`, `go-libp2p-udx`, `dart-libp2p-host` and so on). When a test fails because peers do not connect, load `cryptopeer-connectivity-debugging`. ## Prerequisites: sibling checkouts Every interop test finds the other implementation by a relative path, so all repos must be checked out side by side in one directory: ``` /dart-udx /go-udx /js-udx /go-libp2p-udx-transport /dart-libp2p /dart-libp2p-kad-dht /dart-libp2p-pubsub /dart-libp2p-webrtc /go-ricochet /ricochet-dart-client ``` | Path used | By | |---|---| | `../go-udx` (from dart-udx) | `dart-udx/test/interop/go_udx_interop_test.dart` | | `../../dart-udx` (from go-udx/interop) | `go-udx/interop/*_test.go` | | `../../dart-libp2p` (from go-udx/interop) | `go-udx/interop/docker-test.sh`, `docker-netem-fullstack.sh` | | `../dart-libp2p` | `go-libp2p-udx-transport/interop_test.go` | | `../../../go-udx`, `../../../go-libp2p-udx-transport` | `replace` lines in `dart-libp2p/interop/go-peer/go.mod` | | `../dart-libp2p/interop/go-peer`, or `GO_PEER_DIR` | kad-dht and pubsub `test/interop/helpers/go_process_manager.dart` | | `../go-udx`, `../dart-udx`, `../go-libp2p-udx-transport`, `../dart-libp2p` | js-udx test helpers | | `../go-ricochet/ricochet` (a built binary) | `ricochet-dart-client/test/support/ricochet_server.dart` | Toolchains: Dart (dart-libp2p's dev dependencies need SDK 3.6; the library allows 3.5), Go (the go.mod files ask for 1.25.7, go-ricochet for 1.26.10), Node 20+ for js-udx (CI uses 22.18), Node 22+ for the WebRTC browser harness, Flutter for dart-libp2p-ble, and Docker for the NAT, netem and TURN setups. `dart-libp2p/pubspec_overrides.yaml` (git-ignored) points `dart_udx` at `../dart-udx`. The demos use the same mechanism; see `dart-libp2p-demos/README.md`. Delete or adjust it when you want to test against the published packages. ## Test commands per repo | Repo | Unit / default | Interop or slow suites | |---|---|---| | dart-udx | `dart test` | `dart test test/interop/` (builds `../go-udx/cmd/interop-server`) | | go-udx | `go test -race ./...`; `go test -short ./...` skips the slow lossy transfers | `go test -v ./interop/`; `UDX_THROUGHPUT=1`; `UDX_NETEM_BASELINE=1` (inside the netem container) | | js-udx | `npm test` (`vitest run`); `npm run ci` = typecheck, build, lint, test | `npm run interop:go-udx`; `UDX_SLOW_TESTS=1 npm test` | | go-libp2p-udx-transport | `go test ./...` | `interop_test.go` runs in the same `go test` and needs Dart | | dart-libp2p | `dart test -P ci` (excludes tags `docker` and `interop`, retries twice) | `dart test test/interop/` or `dart test -t interop`; `dart test -t docker` | | dart-libp2p-kad-dht, dart-libp2p-pubsub | `dart test` | `dart test test/interop/` | | dart-libp2p-webrtc | `dart test` | tags `browser` and `turn` skip themselves when their tools are missing | | dart-libp2p-ble | `flutter test` | none: radios are in memory | | go-ricochet | `go test -count=1 ./...` | integration tests need `RICOCHET_TEST_POSTGRES_DSN`; `test/nat/run.sh`; `test/integration/docker/run_test.sh` | | ricochet-dart-client | `dart test` (one file at a time, `concurrency: 1`) | needs the built server and PostgreSQL (`RICOCHET_TEST_PG`) | | go-p2p-forge | `go test ./...` | none | Run one Dart test by name with `dart test --name ""` or `--plain-name ""`. Run one Go test with `go test -run 'TestName' -v -count=1 ./pkg`. Run one vitest file with `npx vitest run `. ## The interop suites ### UDX wire level - `dart-udx/test/interop/go_udx_interop_test.dart` builds go-udx's `cmd/interop-server`, waits for `READY ` on its stderr, and sends raw packets to it. - `go-udx/interop/dart_interop_test.go` runs `dart run bin/interop_server.dart` in `../../dart-udx` the same way. - `go-udx/interop/bulk_interop_test.go` and `multistream_interop_test.go` drive the Dart bulk peer in `interop/dartpeer`. They skip when `dart` is not on `PATH`. `UDX_BULK_PEER` replaces the peer command; js-udx uses it: ```sh UDX_BULK_PEER="node /path/to/js-udx/tools/bulk-peer/bulk-peer.mjs" \ go test ./interop -run 'TestBulk|TestMultiStream' -count=1 -v ``` `npm run interop:go-udx` in js-udx builds first and runs exactly this. - `js-udx/packages/udx/test/go-interop.test.ts` and `dart-interop.test.ts` build `tools/go-peer` and `tools/dart-peer` and test both directions: handshake, 4 MiB echo, 8 concurrent streams, ping, path MTU discovery. - `js-udx/packages/udx/test/go-vectors.test.ts` checks encoding byte for byte against vectors from go-udx. Regenerate them with `npm run vectors` after a wire change in go-udx. ### libp2p level - `dart-libp2p/test/interop/` (all `@Tags(['interop'])`): host, identify, ping, identify push (`go_interop_host_test.dart`), Noise/Yamux upgrader pipeline (`go_interop_test.dart`), circuit relay v2 with a Go relay (`go_interop_relay_test.dart`), relayed identify and DCUtR, AutoNAT v2, and UDX host and keepalive stability. `interop/README.md` lists the directions. - `go-libp2p-udx-transport/interop_test.go` runs dart-libp2p's `bin/interop_echo_server.dart` (prints `READY ` on stderr) and dials it from go-libp2p over UDX, and the reverse. - `js-udx/packages/libp2p-udx/test/interop.test.ts`: js-libp2p against go-libp2p and dart-libp2p, each side dialing: Noise, Yamux, ping, identify, 1 MiB echo. `compliance.test.ts` runs js-libp2p's transport compliance suite. - `dart-libp2p-kad-dht/test/interop/` and `dart-libp2p-pubsub/test/interop/` reuse dart-libp2p's Go peer. - `dart-libp2p-webrtc/test/interop/`: js-libp2p in Chrome, Firefox or WebKit through Playwright, js-libp2p and go-libp2p relays. Setup: `(cd interop/browser && npm install)`, choose a browser with `BROWSER=chrome|chromium|firefox|webkit`, set `NODE` if `node` is older than 22. TURN tests need `docker compose -f docker/coturn/compose.yaml up -d` or `TURN_URL=turn:user:pass@host:port`. Details: `dart-libp2p-webrtc` skill. ### The Go test peer (`dart-libp2p/interop/go-peer`) One binary with many modes, built by `GoProcessManager.ensureBinary` before each run (`go build -o go-peer .`; incremental, so a stale binary is never used). Flags: `--mode`, `--port`, `--target`, `--relay`, `--message`, `--transport=tcp|udx`, `--config=` (yamux `keepalive_interval`, `connection_write_timeout`), plus `--key`, `--value`, `--cid`, `--topic`. Modes: `server`, `client`, `ping`, `echo-server`, `echo-client`, `push-test`, `relay`, `relay-echo-server`, `relay-echo-client`, `dht-server`, `dht-relay-server`, `dht-put-value`, `dht-get-value`, `dht-provide`, `dht-find-providers`, `pubsub-server`, `pubsub-client`, `autonat-server`, `autonat-client`. It prints `PeerID: `, `Listening: `, `CircuitAddr: ` and `Ready` on stdout; `GoProcessManager` parses them and takes the `127.0.0.1` listen address. It is also a handy known-good peer by hand: ```sh cd dart-libp2p/interop/go-peer && go build -o go-peer . ./go-peer --mode=ping --transport=udx --target=/ip4/127.0.0.1/udp/4001/udx/p2p/12D3KooW... ``` ## Writing a cross-process test Follow the pattern the existing suites share: build the peer, start it, wait for a ready marker with a deadline, read its address from the output, and kill it in teardown whatever happens. ```dart @Tags(['interop']) library; import 'dart:async'; import 'dart:convert'; import 'dart:io'; import 'package:test/test.dart'; void main() { late Process peer; late int port; setUpAll(() async { final build = await Process.run('go', ['build', '-o', 'interop-server', './cmd/interop-server/'], workingDirectory: '../go-udx'); if (build.exitCode != 0) fail('Go build failed: ${build.stderr}'); peer = await Process.start('../go-udx/interop-server', []); final ready = Completer(); peer.stderr.transform(utf8.decoder).transform(const LineSplitter()).listen((line) { print('[peer] $line'); // keep the peer's log in the test output if (line.startsWith('READY ') && !ready.isCompleted) ready.complete(int.parse(line.substring(6))); }); port = await ready.future.timeout(const Duration(seconds: 10)); }); tearDownAll(() async { peer.kill(); await peer.exitCode; }); test('round trip', () async { /* dial 127.0.0.1:$port */ }, timeout: const Timeout(Duration(seconds: 30))); } ``` Rules that the existing tests learned the hard way: - **Keep the child's stdin open if it exits on EOF.** dart-libp2p's `bin/interop_echo_server.dart` stops when its stdin closes. From Go, take `cmd.StdinPipe()` and close it at the end of the test; without a pipe, stdin is `/dev/null` and the server stops as soon as it is ready. - **Print the peer's output into the test log** (`t.Logf("[dart-server] %s", line)`, `print('[go-server] $line')`). A failed interop test without the other side's log is guesswork. - **Bind to port 0 and read the port back** (`READY `, `Listening: ...`). Fixed ports collide between parallel runs. - **Skip, do not fail, when a toolchain or sibling is missing**, as js-udx (`describe.skipIf(goPeerUnavailable() !== undefined)`), go-udx's bulk tests (`exec.LookPath("dart")` then `t.Skip`) and ricochet-dart-client (`skip: await RicochetTestServer.available()`, which returns a reason string) do. Put a reason in the skip message that says how to fix it. - **Tag Dart interop tests** with `@Tags(['interop'])` (or `docker`) and declare the tag in `dart_test.yaml`, so `dart test -P ci` leaves them out. ## Loopback and in-memory setups ### Dart: two real hosts on loopback ```dart Future _host() async { final keyPair = await generateEd25519KeyPair(); final host = await Libp2p.new_([ Libp2p.identity(keyPair), Libp2p.transport(TCPTransport()), Libp2p.security(await NoiseSecurity.create(keyPair)), Libp2p.listenAddrs([MultiAddr('/ip4/127.0.0.1/tcp/0')]), Libp2p.autoNAT(false), Libp2p.relay(false), Libp2p.autoRelay(false), Libp2p.holePunching(false), ]); await host.start(); return host; } // Dial with the listener's real addresses: await a.connect(AddrInfo(b.id, b.network.listenAddresses)); // tearDown: await a.close(); await b.close(); ``` (From `dart-libp2p/test/network/dial_peer_id_mismatch_test.dart`.) Use `host.network.listenAddresses`, not `host.addrs`: the default `addrsFactory` (`defaultAddrsFactory` in `basic_host.dart`) removes loopback and unspecified addresses, so `host.addrs` is empty on loopback. For UDX, listen on `/ip4/127.0.0.1/udp/0/udx`. Turn off AutoNAT, relay, AutoRelay and hole punching in tests that do not test them; they start timers and background dials. Always `close()` hosts in teardown: a live timer keeps the Dart process from exiting. The `test/real_net_stack.dart` helper in dart-libp2p, dart-libp2p-pubsub and dart-libp2p-merkle-crdt builds a full UDX + Noise + Yamux node by hand with `createLibp2pNode(...)` and a `passThroughAddrsFactory` that keeps loopback addresses. Copy it when a test needs a hand-built `Swarm`/`BasicHost`. A hand-built host must call `RecordRegistry.register(...)` itself, as `test/integration/holepunch_network/containers/dart-peer/peer_main.dart` does; `Libp2p.new_` does that for you. ### Dart: no network at all - **BLE**: `package:dart_libp2p_ble/testing.dart` gives `FakeBleNetwork` and `FakeBleRadio`; pass a radio to `BleTransport` and the rest of the stack is real. `FakeBleNetwork(maxValueLength:, latency:)`, `setInRange`, `setRssi`. No permissions, advertising limits or lost links: see `dart-libp2p-ble`. - **Bitswap**: `dart-libp2p-bitswap/test/fake_swarm.dart` (mockito mocks of host, network and peerstore that record every message). - **dart-libp2p mocks**: `dart-libp2p/test/mocks/` (mock transport, connections, multiplexer, host, peerstore). - **Time**: dart-udx's `FakeClock` (`advance(Duration)`) through `UDXMultiplexer(socket, clock: ...)` drives UDX timers by hand. ### Go ```go h, err := libp2p.New( libp2p.NoTransports, libp2p.Transport(udxtransport.NewTransport), libp2p.Security(noise.ID, noise.New), libp2p.Muxer(yamux.ID, yamux.DefaultTransport), libp2p.ListenAddrStrings("/ip4/127.0.0.1/udp/0/udx"), libp2p.ResourceManager(&network.NullResourceManager{}), ) ``` (From `go-libp2p-udx-transport/host_test.go`.) go-udx has `NewMockClock` in `clock.go`. go-p2p-forge's `forgetest` package has `NewMockStream`, `NewMockStreamWithFrame` (preloaded request bytes) and `NewPipeStream` (blocking reads over `net.Pipe`, honours deadlines: use it to test timeouts). go-ricochet's integration helpers (`test/integration/helpers_test.go`) start a real server per test with `newTestServer(t)` and skip without `RICOCHET_TEST_POSTGRES_DSN`. ### TypeScript js-udx tests run against workspace sources, so `npm test` needs no build (the `examples` project does use the build). `simulation.test.ts` shows a deterministic link with delay, loss and reordering on `ManualClock`, without sockets: the pattern for testing loss recovery without real time. ## Multi-host and impaired-network setups | Setup | What it does | Run | |---|---|---| | `dart-libp2p/test/integration/holepunch_network/` | Two peers behind cone or symmetric NAT gateways, a relay, STUN; DCUtR | `dart test test/integration/holepunch_network/holepunch_network_integration_test.dart`; `scripts/run_dcutr_scenario.sh /tmp/out`; `scripts/run_dcutr_scenario.sh --go /tmp/out` for a go-libp2p relay and peer (`--go` must come first) | | `dart-libp2p/test/integration/udx_dht_relay_netem/` | DHT + relay under netem | `./run_test.sh baseline\|good-4g\|poor-4g\|terrible`, or `NETEM_DELAY`, `NETEM_JITTER`, `NETEM_LOSS` | | `go-udx/interop/docker-test.sh` | Go peer in Docker with delay and reorder, Dart echo client | `./docker-test.sh 50 25` | | `go-udx/interop/docker-netem-fullstack.sh` | Same, plus loss, then `dart-libp2p/bin/interop_netem_test.dart` | `./docker-netem-fullstack.sh 50 25 2` | | `go-udx/interop/netem-baseline.sh` | go-udx congestion control over a matrix of delay/loss/reorder | `./netem-baseline.sh 4194304 120` | | `go-ricochet/test/nat/run.sh` | Two clients behind NAT routers, 1 MiB through the relay, then a hole punch | `./run.sh`; `NAT_TYPE=symmetric ./run.sh` | | `dart-libp2p-demos/internet_chat/docker/compose.yml` | Relay + DHT + GossipSub + DCUtR chat behind simulated NATs | `docker compose -f docker/compose.yml up --build`; `NAT_TYPE=symmetric` | | `dart-libp2p-webrtc/docker/` | coturn for TURN tests; `linux-test.Dockerfile` runs the suite on Linux | `docker build -f docker/linux-test.Dockerfile -t dart-libp2p-webrtc-test .` | Facts these setups depend on: - The simulated internet is `100.70.0.0/16` for Dart-only runs, because `MultiAddr.isPublic()` rejects RFC 1918 ranges and DCUtR would have no addresses. The go-libp2p variant uses `11.70.0.0/16`, because go-libp2p treats `100.64.0.0/10` as private and will not hole punch from it. - The NAT gateways add `WAN_DELAY` (default `50ms`). With no delay, one punch always arrives before the other leaves and Linux conntrack rewrites the port, so the punch fails however well it is timed. A failed punch leaves conntrack entries that make a retry within 30 s fail too. - Cold starts need 15 to 20 s before NAT rules and relay connections exist. - netem on `lo` delays each direction, so `delay Xms` gives about 2X RTT. `tc` needs `--cap-add=NET_ADMIN`. - Peer containers take `DEBUG_LOGGERS=p2p-holepunch,UDXTransport` and serve `GET /conns` with a `relayed` flag per connection. ## Timeouts and flakiness - Give every network test an explicit timeout that is longer than the library's own: dart_libp2p's dial timeout is 15 s (30 s through a relay), the multistream read timeout 30 s. A test timeout shorter than these hides the real error behind `TimeoutException`. - Wait for a condition, not for a fixed sleep. js-udx's tests poll with `until(cond, timeoutMs)`; Dart tests wait on futures and streams with `.timeout(...)`. - dart-libp2p's CI preset retries a failed test twice (`retry: 2`) because some network tests depend on timing on slow runners. Do not add retries to hide a deterministic failure: run the test 10 times locally first (`go test -count=10`, or a shell loop around `dart test`). - Two test files that each start a Ricochet server time out when they run in parallel; ricochet-dart-client sets `concurrency: 1` in `dart_test.yaml`. Do the same when tests share a fixed port, a database or a Docker setup. - Hole-punch and NAT results vary by run. The harness counts passes over several runs; treat one failed punch as data, not as a bug, unless it repeats. - Idle and keepalive tests take tens of seconds. Gate them behind an env var (`UDX_SLOW_TESTS=1`) or `-short` (Go), as the repos do. ## CI patterns - **dart-libp2p** (`.github/workflows/ci.yml`): Dart 3.6 and stable, `dart pub get`, `dart analyze` (warnings fail), `dart test -P ci`; a second job removes `dev_dependencies` and runs `dart analyze lib` on 3.5 to check the minimum SDK. Interop and Docker suites do not run in CI. - **js-udx** (`check.yml`): Node 22.18, `npm ci`, `npm run ci`. The interop tests skip themselves because the sibling repos are not checked out. - **go-ricochet** (`ci.yml`): PostgreSQL 16 as a service, `GOTOOLCHAIN: local`, `go build`, `go vet`, README example compile check, govulncheck (informational), load `schema.sql`, `go test -count=1 ./...` with `RICOCHET_TEST_POSTGRES_DSN`. `release.yml` calls it. - The other repos have no workflow: run their suites locally before a release. A new interop suite should follow the same rule: CI runs what needs only the repo itself; cross-repo, Docker and browser suites are tagged or skip themselves, and the README of the suite says how to run it. ## Gotchas - **Some interop tests fail instead of skipping.** dart-udx's `go_udx_interop_test.dart`, go-udx's `TestGoDartInterop`, go-libp2p-udx-transport's `TestInterop*` and the kad-dht and pubsub interop tests have no skip: plain `dart test` or `go test ./...` fails without the sibling repo and toolchain. Exclude them by path when you only want unit tests (`dart test test/`, `go test -run '^TestHost'`). - **go-libp2p cannot parse `/udx` unless `udxtransport` is imported.** The package registers the protocol (code `0x0300`) in `init()`. A Go test helper that only parses addresses still needs the import. - **Both ends must speak UDX wire version 3.** A v2 peer's packets are dropped, so the test sees a dial timeout, not a protocol error. Upgrade every sibling together. - **Crossing dials.** dart_udx 4.0.0+ keeps crossing dials as two connections, like go-udx and js-udx. A test that expects two simultaneous `createSocket` dials to meet as one connection is wrong. - **Relay limits cut data tests.** A go-libp2p relay's defaults limit a relayed connection to 2 minutes and 128 KiB. A 1 MiB echo through a go-libp2p relay fails by design; go-ricochet's relay has no limit unless `relay_limits` sets one. - **Only the dialer's identify is done when `connect()` returns.** A Dart `connect()` waits for its own identify (or throws `IdentifyTimeoutException`); the listener identifies the dialer on its own schedule. On the listening side, wait for the peerstore or an event before asserting on the dialer's protocols or addresses. - **The Go peer's Docker image** (`go-udx/interop/Dockerfile`) rewrites the `replace` paths with `sed`; it builds from the parent directory of all checkouts, so run the scripts from their own directory. ## Red lines - Never change a wire format (UDX frames, Noise, Yamux, protobuf messages) to make one side's test pass. Fix the side that is wrong and check it against the other two implementations. - Never commit a `pubspec_overrides.yaml`, a local `replace` that points outside the repo, or a built peer binary. - Never leave a child process or Docker container running after a test: kill it in `tearDown`/`defer`, and use `trap ... EXIT` in scripts. - Never weaken a test (longer sleeps, more retries, skipped assertions) to hide a failure you have not explained. - Do not point tests at public relays or STUN servers; the Docker setups use internal ones so results are repeatable. ## Reference - UDX interop table: https://cryptopeer.net/udx/js/interop/ - WebRTC interop: https://cryptopeer.net/libp2p/webrtc/interop/ - BLE tests without Bluetooth: https://cryptopeer.net/libp2p/ble/testing/ - PubSub testing: https://cryptopeer.net/libp2p/pubsub/testing/ - Internet chat demo with Docker NATs: https://cryptopeer.net/demos/internet-chat/ - In the repos: `dart-libp2p/interop/README.md`, `dart-libp2p/test/integration/holepunch_network/README.md`, `js-udx/doc/interop.md`, `dart-libp2p-webrtc/doc/interop.md`. - Related skills: `cryptopeer-connectivity-debugging`, `udx-dart`, `udx-go`, `udx-js`, `go-libp2p-udx`, `dart-libp2p-host`, `dart-libp2p-nat-relay`, `dart-libp2p-webrtc`, `dart-libp2p-ble`, `ricochet-server`, `ricochet-dart-client`, `go-p2p-forge`.