---
name: cryptopeer-connectivity-debugging
description: Use when cryptopeer peers do not connect or streams stall: dart_libp2p, go-libp2p with the UDX transport, js-libp2p, UDX, relays and hole punching. A decision procedure from multiaddrs through UDX version, Noise, Yamux, multistream, relay limits, NAT and the resource manager, with the loggers, env vars and tools to enable in Dart, Go and JS.
triggers: peers don't connect, dial timeout, stream stalls, PeerIdMismatchException, ProtocolNotSupportedException, ResourceLimitExceededException, LimitedConnException, GOLOG_LOG_LEVEL, UdxLogging, hierarchicalLoggingEnabled, p2p-holepunch, DCUtR fails, relay reservation, /udx multiaddr
---

# Debugging connectivity in the cryptopeer stack

A failed connection has one cause at one layer. Work up the layers in
order, address, transport, security, muxer, protocol, then relay/NAT and
limits, and turn on the log for the layer you suspect before you change
code. This skill is the procedure; the per-package skills (`udx-dart`,
`udx-go`, `udx-js`, `go-libp2p-udx`, `dart-libp2p-host`,
`dart-libp2p-nat-relay`, `dart-libp2p-webrtc`, `dart-libp2p-ble`,
`ricochet-server`) hold the API detail. To reproduce a failure in a test,
load `cryptopeer-interop-testing`.

## Step 0: collect the facts

Before anything else, write down for **both** peers:

1. Implementation and version: `dart_libp2p` and `dart_udx` from
   `pubspec.lock`; go-udx, go-libp2p-udx-transport and go-libp2p from
   `go.mod`; `@go-ricochet/udx` / `@go-ricochet/libp2p-udx` from
   `package-lock.json`.
2. The exact multiaddr being dialled, with `/p2p/<peer id>`.
3. The listen addresses the other side actually bound (not what it
   advertises).
4. Which side dials, whether a relay is involved, and the network between
   them (same host, LAN, NAT, carrier network, Docker).
5. The first error on each side, with logs turned on (see "Turning on
   logs").

Most reports are answered by item 1 or 2.

## Step 1: the address

| Check | Why |
|---|---|
| The multiaddr ends in `/p2p/<id>` and the id is the peer you mean | dart_libp2p 4.x fails the dial with `PeerIdMismatchException: dialed X, but the remote peer is Y` when someone else answers |
| UDX form is `/ip4/<ip>/udp/<port>/udx` (or `/ip6/...`) | `/udx` has no value; protocol code `0x0300` in both Dart and Go |
| The address is not `0.0.0.0` or `::` | those are listen addresses, not dial targets |
| The listener is not bound to `127.0.0.1` when dialled from another host | a loopback listener is only used for loopback dials |
| IPv4 target from an IPv4-capable socket, IPv6 from IPv6 | check the family of what each side bound; zone ids (`%eth0`) are stripped on parse |
| A Go program that parses `/udx` imports `github.com/stephanfeb/go-libp2p-udx-transport` | the package registers `/udx` in `init()`; without the import, parsing fails |

Dart specifics:

- `host.addrs` passes through the `addrsFactory`. The default
  (`defaultAddrsFactory` in `lib/p2p/host/basic/basic_host.dart`) drops
  loopback and unspecified addresses, so on one machine `host.addrs` can be
  empty while the host listens fine. Read `host.network.listenAddresses` for
  what is bound.
- A host behind NAT advertises its public address only after enough peers
  report the same observed address (`Libp2p.observedAddrActivationThreshold`,
  default 4), or when you supply it through an `addrsFactory`. A relay or a
  server on a public IP should announce its address explicitly (go-ricochet:
  `--external-addrs`/`ExternalAddresses` in config).
- Before dart_libp2p 4.x, DNS multiaddrs (`dns4`, `dns6`, `dnsaddr`) and
  `ws`/`wss`/`tls` did not round-trip with go-multiaddr; upgrade if identify
  fails with a `RangeError` against a Go peer.

## Step 2: the transport

Both peers must share a transport: `tcp`, `udx`, `/webrtc` or `/ble`. A
go-libp2p peer built with `libp2p.NoTransports` and only the UDX transport
will never answer a TCP address, and the reverse.

For UDX, check the wire version first. All three implementations speak v3
and **silently drop packets of another version**, so a mismatch looks like
an unreachable peer: the dial times out (15 s in dart_libp2p) and the other
side logs nothing. js-udx answers with a version negotiation packet, so a
js-udx dialer gives up at once.

| Peer | Works with UDX v3 peers from | Notes |
|---|---|---|
| dart_udx | 3.1.0 | 3.0.0 merged a second peer-opened stream into the first and numbered frame types differently |
| dart_udx | 4.0.0 | crossing dials (hole punch) stay two connections, as go-udx and js-udx do; 3.x folds them and the punch fails |
| dart_udx | 4.2.1 / 4.3.0 | 1200-byte datagrams, then PMTUD; earlier releases sent up to 1454 bytes, which mobile carrier tunnels drop |
| go-udx | v0.1.3 | earlier v3 releases did not acknowledge PINGs |
| go-libp2p-udx-transport | v0.1.4 | |
| dart_libp2p | 3.0.0 | earlier releases miss js-libp2p's FIN on a Yamux window update |

(From `js-udx/doc/interop.md` and the dart-udx CHANGELOG.) Use go-udx
v0.3.0 and dart_udx 4.3.0 or later where you can.

Over UDX, libp2p is UDX stream 0 carrying Noise and then Yamux, in both
dart_libp2p and go-libp2p-udx-transport (`stream_conn.go`). So a UDX
handshake that completes followed by a failure is a Noise or Yamux
problem, not a UDX one.

For WebRTC, an `upgrade` that throws after the signaling timeout (30 s) is
ICE failing: see `dart-libp2p-webrtc`, and run
`dart run tool/nat_check.dart` in dart-libp2p-webrtc to see the NAT type.
Dart peers support TURN over UDP only.

## Step 3: security (Noise)

| Symptom | Cause and fix |
|---|---|
| `PeerIdMismatchException` | The peer at that address is not the one in `/p2p/`. Fix the address book or the copied id. |
| `NoiseProtocolException` | Handshake failed. Check both ends use Noise (`/noise`); dart_libp2p adds Noise by default when no security is given. RSA and ECDSA peers need dart_libp2p 4.x (key parsing and peer id hashing changed). |
| Dial succeeds, then closes at once against go-libp2p with no Dart error | Turn on `NoiseProtocol`, `SecuredConnection` and `BasicUpgrader` loggers; on the Go side `GOLOG_LOG_LEVEL=debug` |

## Step 4: the muxer (Yamux)

dart_libp2p uses `/yamux/1.0.0` (`YamuxSession.protocolId`).

- **GO_AWAY received**: the session closes without a reply. The Dart log
  names go-libp2p's codes, for example `garbage collected` (0x1005) when
  the Go connection manager trimmed the connection. Raise the Go side's
  connection limits or keep the connection in use.
- **`Bad state: Maximum streams reached`**: streams that never finished, in
  older releases; in current ones, streams you never close or reset.
  Close every stream; a handler that reads one message must still close.
- **Handlers that read to EOF never finish**: the peer half-closes. Make
  sure both sides call `closeWrite()`/`Close()` and that dart_libp2p is
  3.0.0+ (FIN on WINDOW_UPDATE).
- **New streams hang on a dead path**: dart_libp2p 4.x closes a dead UDX
  connection and redials; on older releases every new stream to that peer
  hung until keepalive failed.

Loggers: `YamuxSession`, `YamuxStream`, `YamuxExceptions`.

## Step 5: protocol negotiation (multistream-select)

| Symptom | Cause and fix |
|---|---|
| `Protocols not supported: [/my/proto/1.0.0]` (`ProtocolNotSupportedException`) | The remote has no handler for that exact string. Protocol ids are compared byte for byte: version, leading slash and case all count. Register the handler before the peer opens the stream. |
| `UnrecognizedResponseException`, `IncorrectVersionException` | The other side is not speaking multistream on this stream: a raw stream used before negotiation, or a version mismatch of a custom framing. |
| `MessageTooLargeException` | A multistream message over the limit: usually a framing bug that reads payload as a header. |
| Negotiation hangs, then times out | The multistream read timeout (`MultistreamConfig.readTimeout`, 30 s) or the inbound deadline (10 s). The remote handler may be blocked before it reads, or the stream is on a connection that is dying. |

Logger: `multistream`. For your own protocol's framing, see
`dart-libp2p-custom-protocol`.

## Step 6: streams that open and then stall

Work through these in order.

1. **The reader is not reading.** Since dart_udx 3.0, an unread
   `UDXStream.data` (no listener, or a paused subscription) stops granting
   credit, and the sender blocks. This is intended back-pressure. Make sure
   something consumes every inbound stream.
2. **A relay cut it.** go-libp2p's relay defaults limit a relayed
   connection to **2 minutes and 128 KiB**; the circuit resets after that.
   A relayed connection with limits is "limited":
   `conn.stat.stats.limited` in Dart, and a go-libp2p peer opens streams
   over it only with `network.WithAllowLimitedConn(ctx, "<reason>")`
   (Dart: `Context().withAllowLimitedConn('<reason>')`). go-ricochet's relay
   is unlimited unless `relay_limits.connection_duration_sec` or
   `connection_data` is set; check `/ops/limits` on the server.
3. **An MTU black hole.** A write of one packet works, a larger one stalls,
   typically on a phone on a carrier network. dart_udx before 4.2.1 sent
   1454-byte datagrams that 464XLAT/GTP tunnels drop. Upgrade.
4. **The UDP socket died.** Android closes an app's sockets when it cuts its
   network in the background. dart_udx 4.2.0+ closes the connections and
   completes `UDXMultiplexer.done`; `UDXMultiplexer.lost` is true when the
   socket stopped on its own. dart_libp2p 4.x rebinds a lost listen socket.
   On older versions every dial through the dead socket timed out.
5. **Loss recovery.** Before dart_udx 4.3.0, a retransmission leaked send
   window and a dozen losses froze the sender; a go-udx sender got no loss
   signal from a dart_udx receiver. Upgrade both ends.
6. **Path change.** A peer that moved (Wi-Fi to cellular, NAT rebinding)
   keeps working over UDX path migration; dart_libp2p 4.x updates
   `remoteMultiaddr`. If not, redial.

Turn on `UdxLogging` (below) to see whether packets and ACKs still flow.

## Step 7: relay, NAT and hole punching

Decision order for two peers behind NAT:

1. **Does each peer have a relay reservation?** Look for the reservation in
   `RelayFinder`/`CircuitV2Client` logs and a `/p2p-circuit` address in
   `host.addrs`. No reservation: check that the relay is reachable
   (UDP and TCP), runs the relay service, and that AutoRelay or
   `Libp2p.relayServers([...])` points at it.
2. **Does the relayed connection work?** Open a stream over it (allowing
   limited connections). If this fails, the problem is at steps 3 to 6.
3. **Does DCUtR start?** Turn on `p2p-holepunch`. DCUtR exchanges only
   public addresses (`MultiAddr.isPublic()`); a peer with only RFC 1918
   addresses has nothing to punch to. go-libp2p also treats
   `100.64.0.0/10` (carrier-grade NAT) as private.
4. **Does the punch fail?** A symmetric NAT on either side usually means no
   direct path: the relay keeps carrying traffic. `BasicHost.udpNatDeviceType`
   shows what the host learned; `Libp2p.stunNatDetection(true, servers:
   [...])` asks STUN servers. With a symmetric NAT dart_libp2p makes one
   punch attempt, not three. A failed punch leaves NAT entries that make a
   retry within about 30 s fail too.
5. **Is AutoNAT confused?** AutoNAT v2 with a relay as the only server
   probes nothing useful. Turn off the server role on phones; see
   `dart-libp2p-nat-relay`.

Known resolved failure modes worth checking the version for: duplicate
circuits through one relay causing `RESOURCE_LIMIT_EXCEEDED` on the relay
(`HAPPY_EYEBALLS_CIRCUIT_RELAY_FIX.md`: dial deduplication by relay route),
go-libp2p rejecting a Dart relay's voucher with `MALFORMED_MESSAGE`,
DCUtR messages without a length prefix, and two relayed connections to one
peer closing each other. All are fixed in dart_libp2p 4.x.

## Step 8: resource limits

dart_libp2p 4.x hosts made with `Libp2p.new_` have go-libp2p's default
limits (`ConfigurableLimiter` with `LimiterConfig.defaults()`). The one
that bites first is **8 connections per peer**. A limit hit raises
`ResourceLimitExceededException` and closes the connection or stream.

```dart
await host.network.resourceManager.viewSystem((scope) async {
  final s = scope.stat;
  print('conns: ${s.numConnsInbound + s.numConnsOutbound}, '
      'streams: ${s.numStreamsInbound + s.numStreamsOutbound}');
});
```

To rule the resource manager out while diagnosing, build the host with
`Libp2p.resourceLimiter(FixedLimiter())` (no limits) and see if the failure
goes away; then set real limits with `LimiterConfig(...)` or
`LimiterConfig.scaled(memory:, fds:)` instead of leaving it unlimited.
Logger: `ResourceManagerImpl`, `StreamScopeImpl`. On go-ricochet,
`performance.max_concurrent_connections` and `max_connections_per_ip` cap
connections; `/ops/limits` on the ops listener shows the effective values.

## Turning on logs

### Dart (`package:logging`)

Per-logger levels need `hierarchicalLoggingEnabled = true`; without it,
only `Logger.root.level` counts.

```dart
import 'package:logging/logging.dart';

hierarchicalLoggingEnabled = true;
Logger.root.level = Level.WARNING;
for (final name in ['p2p-holepunch', 'UDXTransport', 'multistream']) {
  Logger(name).level = Level.ALL;
}
Logger.root.onRecord.listen((r) {
  print('${r.time.toIso8601String()} ${r.level.name} ${r.loggerName}: ${r.message}');
  if (r.error != null) print('  error: ${r.error}');
});
```

dart_libp2p logger names, by layer:

| Layer | Loggers |
|---|---|
| Transports | `UDXTransport`, `UDXStreamAdapter`, `UDXExceptions`, `TCPConnection`, `TCPListener` |
| Upgrade | `BasicUpgrader`, `NoiseProtocol`, `SecuredConnection`, `YamuxSession`, `YamuxStream`, `YamuxExceptions` |
| Swarm and dialing | `Swarm`, `SwarmConn`, `SwarmStream`, `HappyEyeballsDialer`, `AddrDialer`, `ConnectionHealthMonitor`, `BasicConnGater` |
| Host and protocols | `basichost`, `routedhost`, `multistream`, `identify`, `identify.obsaddr`, `ping` |
| Relay and NAT | `CircuitV2Client`, `RelayFinder`, `AutoRelay`, `RelayManager`, `Relay`, `RelayedConn`, `Reservation`, `p2p-holepunch`, `p2p-holepunch-tracer`, `natmgr`, `stun_nat_type`, `autonatv2`, `autonatv2.client`, `autonatv2.server`, `ambient_autonat_v2` |
| Limits | `ResourceManagerImpl`, `StreamScopeImpl` |
| WebRTC (dart_libp2p_webrtc) | `WebRtcTransport`, `WebRtcConn`, `WebRtcStream`, `WebRtcSignaling` |

The internet_chat demo exposes the same mechanism on its command line:
`--log INFO` for everything, `--debug p2p-holepunch` (repeatable) for one
logger. The Docker peers in dart-libp2p's hole-punch harness take
`DEBUG_LOGGERS=p2p-holepunch,UDXTransport`.

### dart_udx (`UdxLogging`)

dart_udx does not use `package:logging`. It has a static switch:

```dart
import 'package:dart_udx/dart_udx.dart';

UdxLogging.setDefaultLogger();   // or: UdxLogging.logger = (level, msg) => ...;
UdxLogging.info = true;          // [DIAG-UDX-RECV], [DIAG-UDX-SEND], [DIAG-UDX-ACK-OUT] lines
UdxLogging.verbose = true;       // debug lines
```

Warnings and errors are logged whenever a logger is set. The info lines are
per packet: turn them on for a short reproduction only. For numbers rather
than lines, implement `UdxMetricsObserver` (handshake, RTT samples,
congestion window, retransmits, flow-control blocks, stream limits) and pass
it as `UDXMultiplexer(socket, metricsObserver: ...)`. dart_libp2p has
`YamuxMetricsObserver` and `RelayMetricsObserver`
(`Libp2p.relayMetricsObserver(...)`) too.

### Go

- go-libp2p uses go-log: `GOLOG_LOG_LEVEL=debug` for everything, or
  per subsystem in code, as `dart-libp2p/interop/go-peer/main.go` does:
  `golog.SetLogLevel("holepunch", "debug")`,
  `golog.SetLogLevel("net/identify", "debug")`. go-ricochet's `--debug-dht`
  raises `dht`, `dht/RtRefreshManager`, `relay` and `relaysvc` to debug
  and `basichost` to info.
- go-ricochet itself logs with `slog` at INFO (one `msg=request` line per
  request), DEBUG with `--development`. A `logging:` block in its config is
  an unknown key and stops the server at startup.
- go-udx logs almost nothing; the one warning to look for is
  `[UDX] WARNING: incoming channel full, connection dropped`: the
  application is not accepting connections fast enough.
- go-libp2p-udx-transport has no logging of its own.

### JavaScript

`@go-ricochet/libp2p-udx` logs through js-libp2p's component logger under
`libp2p:udx`, `libp2p:udx:listener` and `libp2p:udx:connection`, which you
enable as you enable any js-libp2p component log.
`@go-ricochet/udx` has no log output.

## Tools

- **A known-good peer.** dart-libp2p's Go peer:
  `cd dart-libp2p/interop/go-peer && go build -o go-peer . && ./go-peer
  --mode=ping --transport=udx --target=<multiaddr>/p2p/<id>` checks a Dart
  node from go-libp2p; `--mode=server` gives a Go node to dial. Run
  `dart run bin/interop_echo_client.dart <multiaddr>/p2p/<id>` in
  dart-libp2p against a Go node, and `dart run bin/interop_netem_test.dart
  <multiaddr>/p2p/<id>` for connect, identify, echo, a 64 KB echo,
  keepalive and multi-stream phases. If the known-good peer connects, the
  bug is in your node; if not, it is the network or the server.
- **Docker reproductions** with real NAT and netem: see the table in
  `cryptopeer-interop-testing`. `run_dcutr_scenario.sh <out-dir>` writes
  container logs, the NAT gateways' UDP packet captures and the control API
  responses; the peer containers' `GET /conns` lists connections with a
  `relayed` flag.
- **NAT type**: `dart run tool/nat_check.dart` in dart-libp2p-webrtc asks
  public STUN servers from one socket, for IPv4 and IPv6.
- **Packet capture**: `tcpdump -n -i any udp port <port>` on either end
  tells you whether datagrams arrive at all, which separates a firewall or
  NAT problem from everything above it.
- **go-ricochet ops listener** (127.0.0.1:9090 by default): `/healthz`,
  `/readyz`, `/ops/limits`, `/metrics` (with metrics on).

## Gotchas

- **A timeout is a symptom, not a cause.** A dial timeout with no error on
  the other side usually means the packets never arrived or were dropped:
  wrong address, firewall, UDX version mismatch, or a dead socket.
- **The first error is the one that matters.** Later errors (`Unexpected
  end of stream`, stream resets, `Session closed while opening stream`) are
  the teardown after it.
- **Failed hole punches are normal** behind NAT and are logged at FINE in
  dart_libp2p 4.x; the relayed connection stays in use. Do not treat them
  as the bug unless the relayed path also fails.
- **Loopback hides NAT and MTU problems.** Reproduce with the Docker setups
  before you conclude a fix works.
- **One version table, both ends.** Upgrading only one side of a UDX or
  hole-punch pair can make it worse (crossing dials, wire version).
- **go-libp2p's connection manager trims idle peers** (GO_AWAY
  `garbage collected`). A test or app that holds a connection without
  streams can lose it.

## Red lines

- Do not turn off Noise (`Libp2p.noSecurity()`) to "see if security is the
  problem" on anything but loopback, and never ship it.
- Do not leave `UdxLogging.info`, `GOLOG_LOG_LEVEL=debug`, pprof or a
  `FixedLimiter` on in production: they cost CPU, fill disks or remove DoS
  protection.
- Do not expose go-ricochet's ops listener beyond loopback without
  authentication; `/ops/*` names peers and their usage.
- Do not paste private keys, `RICOCHET_PG_PASSWORD` or TURN secrets into
  logs or issue reports; peer ids and multiaddrs are fine.
- Do not work around a protocol mismatch by changing the wire format on one
  side.

## Reference

- Transports: https://cryptopeer.net/libp2p/core/transports/
- Resource manager: https://cryptopeer.net/libp2p/core/resource-manager/
- Relay setup: https://cryptopeer.net/libp2p/core/relay-setup/
- UDX interop versions: https://cryptopeer.net/udx/js/interop/
- WebRTC networks and TURN: https://cryptopeer.net/libp2p/webrtc/networks/
- Internet chat demo, troubleshooting table: https://cryptopeer.net/demos/internet-chat/
- In the repos: `dart-libp2p/CHANGELOG.md`, `dart-udx/CHANGELOG.md`,
  `dart-udx/SEQUENCE_NUMBER_DESYNC_FIX.md`,
  `dart-libp2p/HAPPY_EYEBALLS_CIRCUIT_RELAY_FIX.md`,
  `dart-libp2p/IPV6_FIX_SUMMARY.md`,
  `dart-libp2p/test/integration/holepunch_network/README.md`,
  `go-ricochet/config.example.yaml`.
- Related skills: `cryptopeer-interop-testing`, `udx-dart`, `udx-go`,
  `udx-js`, `go-libp2p-udx`, `dart-libp2p-host`,
  `dart-libp2p-custom-protocol`, `dart-libp2p-nat-relay`,
  `dart-libp2p-webrtc`, `dart-libp2p-ble`, `ricochet-server`,
  `go-p2p-forge`.