Skip to content
cryptopeercryptopeer.net

cryptopeer-connectivity-debugging

Skill cryptopeer-connectivity-debugging. Use when cryptopeer peers do not connect or streams stall: dart_libp2p, go-libp2p with the UDX transport, js-libp2p, UDX, relays and hole punching. A decision procedure from multiaddrs through UDX version, Noise, Yamux, multistream, relay limits, NAT and the resource manager, with the loggers, env vars and tools to enable in Dart, Go and JS.

SKILL.md (Claude Code, Codex, Cursor) · cryptopeer-connectivity-debugging.md (Boubou) · install: curl -fsSL https://cryptopeer.net/skills/install.sh | sh -s -- cryptopeer-connectivity-debugging

A failed connection has one cause at one layer. Work up the layers in order, address, transport, security, muxer, protocol, then relay/NAT and limits, and turn on the log for the layer you suspect before you change code. This skill is the procedure; the per-package skills (udx-dart, udx-go, udx-js, go-libp2p-udx, dart-libp2p-host, dart-libp2p-nat-relay, dart-libp2p-webrtc, dart-libp2p-ble, ricochet-server) hold the API detail. To reproduce a failure in a test, load cryptopeer-interop-testing.

Step 0: collect the facts

Section titled “Step 0: collect the facts”

Before anything else, write down for both peers:

  1. Implementation and version: dart_libp2p and dart_udx from pubspec.lock; go-udx, go-libp2p-udx-transport and go-libp2p from go.mod; @go-ricochet/udx / @go-ricochet/libp2p-udx from package-lock.json.
  2. The exact multiaddr being dialled, with /p2p/<peer id>.
  3. The listen addresses the other side actually bound (not what it advertises).
  4. Which side dials, whether a relay is involved, and the network between them (same host, LAN, NAT, carrier network, Docker).
  5. The first error on each side, with logs turned on (see “Turning on logs”).

Most reports are answered by item 1 or 2.

Step 1: the address

Section titled “Step 1: the address”
Check Why
The multiaddr ends in /p2p/<id> and the id is the peer you mean dart_libp2p 4.x fails the dial with PeerIdMismatchException: dialed X, but the remote peer is Y when someone else answers
UDX form is /ip4/<ip>/udp/<port>/udx (or /ip6/...) /udx has no value; protocol code 0x0300 in both Dart and Go
The address is not 0.0.0.0 or :: those are listen addresses, not dial targets
The listener is not bound to 127.0.0.1 when dialled from another host a loopback listener is only used for loopback dials
IPv4 target from an IPv4-capable socket, IPv6 from IPv6 check the family of what each side bound; zone ids (%eth0) are stripped on parse
A Go program that parses /udx imports github.com/stephanfeb/go-libp2p-udx-transport the package registers /udx in init(); without the import, parsing fails

Dart specifics:

  • host.addrs passes through the addrsFactory. The default (defaultAddrsFactory in lib/p2p/host/basic/basic_host.dart) drops loopback and unspecified addresses, so on one machine host.addrs can be empty while the host listens fine. Read host.network.listenAddresses for what is bound.
  • A host behind NAT advertises its public address only after enough peers report the same observed address (Libp2p.observedAddrActivationThreshold, default 4), or when you supply it through an addrsFactory. A relay or a server on a public IP should announce its address explicitly (go-ricochet: --external-addrs/ExternalAddresses in config).
  • Before dart_libp2p 4.x, DNS multiaddrs (dns4, dns6, dnsaddr) and ws/wss/tls did not round-trip with go-multiaddr; upgrade if identify fails with a RangeError against a Go peer.

Step 2: the transport

Section titled “Step 2: the transport”

Both peers must share a transport: tcp, udx, /webrtc or /ble. A go-libp2p peer built with libp2p.NoTransports and only the UDX transport will never answer a TCP address, and the reverse.

For UDX, check the wire version first. All three implementations speak v3 and silently drop packets of another version, so a mismatch looks like an unreachable peer: the dial times out (15 s in dart_libp2p) and the other side logs nothing. js-udx answers with a version negotiation packet, so a js-udx dialer gives up at once.

Peer Works with UDX v3 peers from Notes
dart_udx 3.1.0 3.0.0 merged a second peer-opened stream into the first and numbered frame types differently
dart_udx 4.0.0 crossing dials (hole punch) stay two connections, as go-udx and js-udx do; 3.x folds them and the punch fails
dart_udx 4.2.1 / 4.3.0 1200-byte datagrams, then PMTUD; earlier releases sent up to 1454 bytes, which mobile carrier tunnels drop
go-udx v0.1.3 earlier v3 releases did not acknowledge PINGs
go-libp2p-udx-transport v0.1.4
dart_libp2p 3.0.0 earlier releases miss js-libp2p’s FIN on a Yamux window update

(From js-udx/doc/interop.md and the dart-udx CHANGELOG.) Use go-udx v0.3.0 and dart_udx 4.3.0 or later where you can.

Over UDX, libp2p is UDX stream 0 carrying Noise and then Yamux, in both dart_libp2p and go-libp2p-udx-transport (stream_conn.go). So a UDX handshake that completes followed by a failure is a Noise or Yamux problem, not a UDX one.

For WebRTC, an upgrade that throws after the signaling timeout (30 s) is ICE failing: see dart-libp2p-webrtc, and run dart run tool/nat_check.dart in dart-libp2p-webrtc to see the NAT type. Dart peers support TURN over UDP only.

Step 3: security (Noise)

Section titled “Step 3: security (Noise)”
Symptom Cause and fix
PeerIdMismatchException The peer at that address is not the one in /p2p/. Fix the address book or the copied id.
NoiseProtocolException Handshake failed. Check both ends use Noise (/noise); dart_libp2p adds Noise by default when no security is given. RSA and ECDSA peers need dart_libp2p 4.x (key parsing and peer id hashing changed).
Dial succeeds, then closes at once against go-libp2p with no Dart error Turn on NoiseProtocol, SecuredConnection and BasicUpgrader loggers; on the Go side GOLOG_LOG_LEVEL=debug

Step 4: the muxer (Yamux)

Section titled “Step 4: the muxer (Yamux)”

dart_libp2p uses /yamux/1.0.0 (YamuxSession.protocolId).

  • GO_AWAY received: the session closes without a reply. The Dart log names go-libp2p’s codes, for example garbage collected (0x1005) when the Go connection manager trimmed the connection. Raise the Go side’s connection limits or keep the connection in use.
  • Bad state: Maximum streams reached: streams that never finished, in older releases; in current ones, streams you never close or reset. Close every stream; a handler that reads one message must still close.
  • Handlers that read to EOF never finish: the peer half-closes. Make sure both sides call closeWrite()/Close() and that dart_libp2p is 3.0.0+ (FIN on WINDOW_UPDATE).
  • New streams hang on a dead path: dart_libp2p 4.x closes a dead UDX connection and redials; on older releases every new stream to that peer hung until keepalive failed.

Loggers: YamuxSession, YamuxStream, YamuxExceptions.

Step 5: protocol negotiation (multistream-select)

Section titled “Step 5: protocol negotiation (multistream-select)”
Symptom Cause and fix
Protocols not supported: [/my/proto/1.0.0] (ProtocolNotSupportedException) The remote has no handler for that exact string. Protocol ids are compared byte for byte: version, leading slash and case all count. Register the handler before the peer opens the stream.
UnrecognizedResponseException, IncorrectVersionException The other side is not speaking multistream on this stream: a raw stream used before negotiation, or a version mismatch of a custom framing.
MessageTooLargeException A multistream message over the limit: usually a framing bug that reads payload as a header.
Negotiation hangs, then times out The multistream read timeout (MultistreamConfig.readTimeout, 30 s) or the inbound deadline (10 s). The remote handler may be blocked before it reads, or the stream is on a connection that is dying.

Logger: multistream. For your own protocol’s framing, see dart-libp2p-custom-protocol.

Step 6: streams that open and then stall

Section titled “Step 6: streams that open and then stall”

Work through these in order.

  1. The reader is not reading. Since dart_udx 3.0, an unread UDXStream.data (no listener, or a paused subscription) stops granting credit, and the sender blocks. This is intended back-pressure. Make sure something consumes every inbound stream.
  2. A relay cut it. go-libp2p’s relay defaults limit a relayed connection to 2 minutes and 128 KiB; the circuit resets after that. A relayed connection with limits is “limited”: conn.stat.stats.limited in Dart, and a go-libp2p peer opens streams over it only with network.WithAllowLimitedConn(ctx, "<reason>") (Dart: Context().withAllowLimitedConn('<reason>')). go-ricochet’s relay is unlimited unless relay_limits.connection_duration_sec or connection_data is set; check /ops/limits on the server.
  3. An MTU black hole. A write of one packet works, a larger one stalls, typically on a phone on a carrier network. dart_udx before 4.2.1 sent 1454-byte datagrams that 464XLAT/GTP tunnels drop. Upgrade.
  4. The UDP socket died. Android closes an app’s sockets when it cuts its network in the background. dart_udx 4.2.0+ closes the connections and completes UDXMultiplexer.done; UDXMultiplexer.lost is true when the socket stopped on its own. dart_libp2p 4.x rebinds a lost listen socket. On older versions every dial through the dead socket timed out.
  5. Loss recovery. Before dart_udx 4.3.0, a retransmission leaked send window and a dozen losses froze the sender; a go-udx sender got no loss signal from a dart_udx receiver. Upgrade both ends.
  6. Path change. A peer that moved (Wi-Fi to cellular, NAT rebinding) keeps working over UDX path migration; dart_libp2p 4.x updates remoteMultiaddr. If not, redial.

Turn on UdxLogging (below) to see whether packets and ACKs still flow.

Step 7: relay, NAT and hole punching

Section titled “Step 7: relay, NAT and hole punching”

Decision order for two peers behind NAT:

  1. Does each peer have a relay reservation? Look for the reservation in RelayFinder/CircuitV2Client logs and a /p2p-circuit address in host.addrs. No reservation: check that the relay is reachable (UDP and TCP), runs the relay service, and that AutoRelay or Libp2p.relayServers([...]) points at it.
  2. Does the relayed connection work? Open a stream over it (allowing limited connections). If this fails, the problem is at steps 3 to 6.
  3. Does DCUtR start? Turn on p2p-holepunch. DCUtR exchanges only public addresses (MultiAddr.isPublic()); a peer with only RFC 1918 addresses has nothing to punch to. go-libp2p also treats 100.64.0.0/10 (carrier-grade NAT) as private.
  4. Does the punch fail? A symmetric NAT on either side usually means no direct path: the relay keeps carrying traffic. BasicHost.udpNatDeviceType shows what the host learned; Libp2p.stunNatDetection(true, servers: [...]) asks STUN servers. With a symmetric NAT dart_libp2p makes one punch attempt, not three. A failed punch leaves NAT entries that make a retry within about 30 s fail too.
  5. Is AutoNAT confused? AutoNAT v2 with a relay as the only server probes nothing useful. Turn off the server role on phones; see dart-libp2p-nat-relay.

Known resolved failure modes worth checking the version for: duplicate circuits through one relay causing RESOURCE_LIMIT_EXCEEDED on the relay (HAPPY_EYEBALLS_CIRCUIT_RELAY_FIX.md: dial deduplication by relay route), go-libp2p rejecting a Dart relay’s voucher with MALFORMED_MESSAGE, DCUtR messages without a length prefix, and two relayed connections to one peer closing each other. All are fixed in dart_libp2p 4.x.

Step 8: resource limits

Section titled “Step 8: resource limits”

dart_libp2p 4.x hosts made with Libp2p.new_ have go-libp2p’s default limits (ConfigurableLimiter with LimiterConfig.defaults()). The one that bites first is 8 connections per peer. A limit hit raises ResourceLimitExceededException and closes the connection or stream.

await host.network.resourceManager.viewSystem((scope) async {
  final s = scope.stat;
  print('conns: ${s.numConnsInbound + s.numConnsOutbound}, '
      'streams: ${s.numStreamsInbound + s.numStreamsOutbound}');
});

To rule the resource manager out while diagnosing, build the host with Libp2p.resourceLimiter(FixedLimiter()) (no limits) and see if the failure goes away; then set real limits with LimiterConfig(...) or LimiterConfig.scaled(memory:, fds:) instead of leaving it unlimited. Logger: ResourceManagerImpl, StreamScopeImpl. On go-ricochet, performance.max_concurrent_connections and max_connections_per_ip cap connections; /ops/limits on the ops listener shows the effective values.

Turning on logs

Section titled “Turning on logs”

Dart (package:logging)

Section titled “Dart (package:logging)”

Per-logger levels need hierarchicalLoggingEnabled = true; without it, only Logger.root.level counts.

import 'package:logging/logging.dart';


hierarchicalLoggingEnabled = true;
Logger.root.level = Level.WARNING;
for (final name in ['p2p-holepunch', 'UDXTransport', 'multistream']) {
  Logger(name).level = Level.ALL;
}
Logger.root.onRecord.listen((r) {
  print('${r.time.toIso8601String()} ${r.level.name} ${r.loggerName}: ${r.message}');
  if (r.error != null) print('  error: ${r.error}');
});

dart_libp2p logger names, by layer:

Layer Loggers
Transports UDXTransport, UDXStreamAdapter, UDXExceptions, TCPConnection, TCPListener
Upgrade BasicUpgrader, NoiseProtocol, SecuredConnection, YamuxSession, YamuxStream, YamuxExceptions
Swarm and dialing Swarm, SwarmConn, SwarmStream, HappyEyeballsDialer, AddrDialer, ConnectionHealthMonitor, BasicConnGater
Host and protocols basichost, routedhost, multistream, identify, identify.obsaddr, ping
Relay and NAT CircuitV2Client, RelayFinder, AutoRelay, RelayManager, Relay, RelayedConn, Reservation, p2p-holepunch, p2p-holepunch-tracer, natmgr, stun_nat_type, autonatv2, autonatv2.client, autonatv2.server, ambient_autonat_v2
Limits ResourceManagerImpl, StreamScopeImpl
WebRTC (dart_libp2p_webrtc) WebRtcTransport, WebRtcConn, WebRtcStream, WebRtcSignaling

The internet_chat demo exposes the same mechanism on its command line: --log INFO for everything, --debug p2p-holepunch (repeatable) for one logger. The Docker peers in dart-libp2p’s hole-punch harness take DEBUG_LOGGERS=p2p-holepunch,UDXTransport.

dart_udx (UdxLogging)

Section titled “dart_udx (UdxLogging)”

dart_udx does not use package:logging. It has a static switch:

import 'package:dart_udx/dart_udx.dart';


UdxLogging.setDefaultLogger();   // or: UdxLogging.logger = (level, msg) => ...;
UdxLogging.info = true;          // [DIAG-UDX-RECV], [DIAG-UDX-SEND], [DIAG-UDX-ACK-OUT] lines
UdxLogging.verbose = true;       // debug lines

Warnings and errors are logged whenever a logger is set. The info lines are per packet: turn them on for a short reproduction only. For numbers rather than lines, implement UdxMetricsObserver (handshake, RTT samples, congestion window, retransmits, flow-control blocks, stream limits) and pass it as UDXMultiplexer(socket, metricsObserver: ...). dart_libp2p has YamuxMetricsObserver and RelayMetricsObserver (Libp2p.relayMetricsObserver(...)) too.

Go

Section titled “Go”
  • go-libp2p uses go-log: GOLOG_LOG_LEVEL=debug for everything, or per subsystem in code, as dart-libp2p/interop/go-peer/main.go does: golog.SetLogLevel("holepunch", "debug"), golog.SetLogLevel("net/identify", "debug"). go-ricochet’s --debug-dht raises dht, dht/RtRefreshManager, relay and relaysvc to debug and basichost to info.
  • go-ricochet itself logs with slog at INFO (one msg=request line per request), DEBUG with --development. A logging: block in its config is an unknown key and stops the server at startup.
  • go-udx logs almost nothing; the one warning to look for is [UDX] WARNING: incoming channel full, connection dropped: the application is not accepting connections fast enough.
  • go-libp2p-udx-transport has no logging of its own.

JavaScript

Section titled “JavaScript”

@go-ricochet/libp2p-udx logs through js-libp2p’s component logger under libp2p:udx, libp2p:udx:listener and libp2p:udx:connection, which you enable as you enable any js-libp2p component log. @go-ricochet/udx has no log output.

Tools

Section titled “Tools”
  • A known-good peer. dart-libp2p’s Go peer: cd dart-libp2p/interop/go-peer && go build -o go-peer . && ./go-peer --mode=ping --transport=udx --target=<multiaddr>/p2p/<id> checks a Dart node from go-libp2p; --mode=server gives a Go node to dial. Run dart run bin/interop_echo_client.dart <multiaddr>/p2p/<id> in dart-libp2p against a Go node, and dart run bin/interop_netem_test.dart <multiaddr>/p2p/<id> for connect, identify, echo, a 64 KB echo, keepalive and multi-stream phases. If the known-good peer connects, the bug is in your node; if not, it is the network or the server.
  • Docker reproductions with real NAT and netem: see the table in cryptopeer-interop-testing. run_dcutr_scenario.sh <out-dir> writes container logs, the NAT gateways’ UDP packet captures and the control API responses; the peer containers’ GET /conns lists connections with a relayed flag.
  • NAT type: dart run tool/nat_check.dart in dart-libp2p-webrtc asks public STUN servers from one socket, for IPv4 and IPv6.
  • Packet capture: tcpdump -n -i any udp port <port> on either end tells you whether datagrams arrive at all, which separates a firewall or NAT problem from everything above it.
  • go-ricochet ops listener (127.0.0.1:9090 by default): /healthz, /readyz, /ops/limits, /metrics (with metrics on).

Gotchas

Section titled “Gotchas”
  • A timeout is a symptom, not a cause. A dial timeout with no error on the other side usually means the packets never arrived or were dropped: wrong address, firewall, UDX version mismatch, or a dead socket.
  • The first error is the one that matters. Later errors (Unexpected end of stream, stream resets, Session closed while opening stream) are the teardown after it.
  • Failed hole punches are normal behind NAT and are logged at FINE in dart_libp2p 4.x; the relayed connection stays in use. Do not treat them as the bug unless the relayed path also fails.
  • Loopback hides NAT and MTU problems. Reproduce with the Docker setups before you conclude a fix works.
  • One version table, both ends. Upgrading only one side of a UDX or hole-punch pair can make it worse (crossing dials, wire version).
  • go-libp2p’s connection manager trims idle peers (GO_AWAY garbage collected). A test or app that holds a connection without streams can lose it.

Red lines

Section titled “Red lines”
  • Do not turn off Noise (Libp2p.noSecurity()) to “see if security is the problem” on anything but loopback, and never ship it.
  • Do not leave UdxLogging.info, GOLOG_LOG_LEVEL=debug, pprof or a FixedLimiter on in production: they cost CPU, fill disks or remove DoS protection.
  • Do not expose go-ricochet’s ops listener beyond loopback without authentication; /ops/* names peers and their usage.
  • Do not paste private keys, RICOCHET_PG_PASSWORD or TURN secrets into logs or issue reports; peer ids and multiaddrs are fine.
  • Do not work around a protocol mismatch by changing the wire format on one side.

Reference

Section titled “Reference”

Built by Stephan Februarygithub.com/stephanfebLinkedIn