--- name: dart-libp2p-kad-dht description: Use when a dart_libp2p app needs the Kademlia DHT (package:dart_libp2p_kad_dht, IpfsDHTv2): bootstrapping, findPeer, provide / findProvidersAsync content routing, putValue / getValue signed records with namespace validators, advertise / findPeers rendezvous, client / server / auto modes, go-libp2p interop. triggers: dart_libp2p_kad_dht, IpfsDHTv2, IpfsDHT, DHTOptions, DHTMode, MemoryProviderStore, findProvidersAsync, NamespacedValidator, namespaceToCid, /ipfs/kad/1.0.0, kademlia dht --- # dart_libp2p_kad_dht ## What it is A Dart port of go-libp2p-kad-dht on top of dart_libp2p. It speaks `/ipfs/kad/1.0.0` with the protobuf wire format, so Dart and go-libp2p nodes share one DHT. It gives peer routing (`findPeer`), content routing (`provide`, `findProvidersAsync`), a validated key/value store (`putValue`, `getValue`) and namespace rendezvous (`advertise`, `findPeers`). Use `IpfsDHTv2` for new code. The legacy `IpfsDHT` still exists and follows the same record rules, but v2 is the one the demos and tests drive. Use something else when: - Two peers already know each other's address: just `host.connect`. - Peers are on the same LAN: mDNS is simpler. - You want to broadcast messages: GossipSub (`dart-libp2p-gossipsub` skill); the DHT only helps peers find each other. - Host, transports, relays, AutoNAT: see the `dart-libp2p-host` skill. ## Install ```yaml dependencies: dart_libp2p_kad_dht: ^1.6.0 dart_libp2p: ^4.7.0 # this package accepts >=1.0.0 <5.0.0 dcid: ^1.0.0 # CID ``` ```dart import 'dart:convert'; // utf8 import 'dart:typed_data'; // Uint8List import 'package:dart_libp2p/dart_libp2p.dart'; // Host, PeerId, MultiAddr, AddrInfo, RoutingOptions import 'package:dart_libp2p_kad_dht/dart_libp2p_kad_dht.dart'; // IpfsDHTv2, DHTOptions, DHTMode, MemoryProviderStore, namespaceToCid import 'package:dcid/dcid.dart'; // CID // Not in the barrel; import from src when you need them: import 'package:dart_libp2p_kad_dht/src/dht/v2/errors/dht_errors.dart'; // DHTException and subclasses import 'package:dart_libp2p_kad_dht/src/record/validator.dart'; // Validator import 'package:dart_libp2p_kad_dht/src/record/namespace_validator.dart'; // NamespacedValidator import 'package:dart_libp2p_kad_dht/src/record/public_key_validator.dart'; // PublicKeyValidator import 'package:dart_libp2p_kad_dht/src/record/ipns_validator.dart'; // IpnsValidator ``` ## Core workflow ### Minimal node (start the DHT before the host) Build the host as in the `dart-libp2p-host` skill, but do not call `host.start()` yet. ```dart final dht = IpfsDHTv2( host: host, providerStore: MemoryProviderStore(), options: DHTOptions( mode: DHTMode.client, // behind NAT; DHTMode.server on a public node bootstrapPeers: [ // must end in /p2p/ MultiAddr('/ip4/203.0.113.7/udp/4001/udx/p2p/12D3KooW...'), ], ), ); await dht.start(); // registers /ipfs/kad/1.0.0 (server modes) first await host.start(); // now Identify advertises the DHT protocol try { await dht.bootstrap(); // dials bootstrapPeers, fills the routing table } on DHTBootstrapException catch (e) { // keep running; retry later } // ... use it ... await dht.close(); // does not close the host await host.close(); ``` ### Bootstrap / relay node (public, serves the DHT) From `internet_chat/lib/src/relay_node.dart`. The host is created with `Libp2p.forceReachability(Reachability.public)`. ```dart final dht = IpfsDHTv2( host: host, providerStore: MemoryProviderStore(), options: DHTOptions(mode: DHTMode.server), ); await dht.start(); await host.start(); // Others use '$addr/p2p/${host.id.toBase58()}' as a bootstrap peer. ``` ### Find a peer ```dart final AddrInfo? info = await dht.findPeer(PeerId.fromString('12D3KooW...')); if (info != null) await host.connect(info); final closest = await dht.getClosestPeers(somePeerId); // List ``` ### Content routing (provide / find providers) ```dart final cid = CID.fromData(CID.V1, 'raw', utf8.encode('my-app/room/lobby')); await dht.provide(cid, true); // true: announce to the network; false: local only await for (final p in dht.findProvidersAsync(cid, 20)) { // 0 = no limit if (p.id == host.id) continue; // you are in your own results if (p.addrs.isNotEmpty) await host.connect(p); } ``` Provider records expire (`provideValidity`, 48 h). Re-provide on a timer; the chat demo re-provides every minute and searches every 10 s. ### Rendezvous by namespace ```dart final ttl = await dht.advertise('my-app/chat'); // Duration (provideValidity) final peers = await dht.findPeers('my-app/chat'); // Future> await for (final p in peers) { /* ... */ } ``` The key is `namespaceToCid(ns)`: CIDv1, raw, sha2-256 of the UTF-8 namespace, the same as `nsToCid` in go-libp2p routing discovery. ### Value records with your own namespace ```dart class GreetingValidator implements Validator { @override Future validate(String key, Uint8List value) async { if (value.length > 1024) throw Exception('value too large'); // To stop others overwriting it, tie the key to its owner here // (e.g. require a signature by the peer ID in the key). } @override Future select(String key, List values) async => 0; // index of best } final dht = IpfsDHTv2( host: host, providerStore: MemoryProviderStore(), validator: NamespacedValidator() ..['pk'] = PublicKeyValidator() ..['ipns'] = IpnsValidator(host.peerStore) ..['greeting'] = GreetingValidator(), ); await dht.start(); await dht.putValue('/greeting/alice', Uint8List.fromList(utf8.encode('hello'))); final v = await dht.getValue('/greeting/alice'); // asks the network final local = await dht.getValue('/greeting/alice', RoutingOptions()..offline = true); await for (final better in dht.searchValue('/greeting/alice', null)) { /* ... */ } ``` ### Errors ```dart try { await dht.findPeer(id); } on DHTTimeoutException catch (e) { // ... } on DHTNetworkException catch (e) { // ... } on DHTException catch (e) { print(e.message); } ``` ## Key concepts / API map - `IpfsDHTv2({required host, required providerStore, DHTOptions? options, NamespacedValidator? validator})`: the DHT; implements `IpfsDHT` (`Routing`, `Discovery`). - `start()` / `bootstrap({periodicRefreshInterval, quickConnectOnly})` / `close()`: lifecycle; `bootstrap` starts the DHT if needed. - `started`, `isServer`: state; `isServer` is whether `/ipfs/kad/1.0.0` is registered now. - `findPeer(PeerId, {RoutingOptions? options})` -> `Future`. - `getClosestPeers(PeerId, {networkQueryEnabled})` -> `Future>`. - `provide(CID, bool announce)`, `findProvidersAsync(CID, int count)` -> `Stream`. - `addProvider(Uint8List key, AddrInfo)`, `getLocalProviders(Uint8List key)`: local provider store, key is a multihash (`providerKeyOf(cid)`). - `putValue(String key, Uint8List value, {RoutingOptions? options})`, `getValue(key, [RoutingOptions?])`, `searchValue(key, RoutingOptions?)` -> `Stream`. - `advertise(String ns)`, `findPeers(String ns)`: discovery on `namespaceToCid(ns)`. - `metrics` -> `DHTMetrics` (`totalQueries`, `querySuccessRate`, `routingTableSize`, `networkSuccessRate`, ...). - `routingTable`, `getRoutingTableSize()`, `getRoutingTablePeers()`, `refreshRoutingTable()`. - `DHTOptions`: `mode` (default `auto`), `bucketSize` 20, `concurrency` 10, `resiliency` 3, `provideValidity` 48 h, `providerAddrTTL` 24 h, `autoRefresh` true, `bootstrapPeers` (`List`), `maxRetryAttempts` 3, `retryInitialBackoff`, `retryMaxBackoff`, `retryBackoffFactor`, `filterLocalhostInResponses` true, `maxRecordAge` 36 h, `allowUnvalidatedRecords` false. - `DHTMode`: `client`, `server`, `auto`, `autoServer`. - `MemoryProviderStore([ProviderManagerOptions options, PeerId? localPeerId])`; `ProviderManagerOptions` has `maxProvidersPerKey` 100, `maxKeysPerProvider` 1000, `maxProviderRecords` 100000, `cleanupInterval`, `provideValidity`. - `ProviderStore`: interface (`addProvider(CID, AddrInfo)`, `getProviders(CID)`, `close()`) for your own store. - `providerKeyOf(cid)`, `providerKeyFromWire`, `cidForProviderKey`, `namespaceToCid(ns)`: key helpers. - `Validator` (`validate`, `select`), `NamespacedValidator` (`[ns] = v`), `PublicKeyValidator`, `IpnsValidator(peerStore)`. - `AminoConstants`: `protocolID` (`/ipfs/kad/1.0.0`), defaults, `inboundStreamIdleTimeout` (1 min). - Errors (`src/dht/v2/errors/dht_errors.dart`): `DHTException`, `DHTNotStartedException`, `DHTClosedException`, `DHTBootstrapException`, `DHTNetworkException`, `DHTRoutingException`, `DHTQueryException`, `DHTProtocolException`, `DHTTimeoutException`, `DHTMaxRetriesException`, `DHTConfigurationException`. ## Gotchas - **Start the DHT before the host.** `host.start()` makes AutoRelay connect and run Identify at once. If `/ipfs/kad/1.0.0` is not registered yet, Go peers mark the node "peer stopped dht" and stop opening DHT streams to it on that connection. Order: create host, `dht.start()`, `host.start()`, `dht.bootstrap()`. The same applies to every protocol that Identify must advertise. - **`DHTMode.auto` (the default) starts as a client.** It serves only after the host emits `EvtLocalReachabilityChanged` with `Reachability.public` (AutoNAT, or `forceReachability`), and stops on private or unknown. On a loopback test network that event never comes, so nodes in `auto` never serve and lookups find nothing. Use `DHTMode.server` for bootstrap nodes and tests, `DHTMode.client` for NATed or mobile nodes. `autoServer` serves unless reachability is private. - **Client mode does not serve and does not enter routing tables.** Only peers whose Identify lists `/ipfs/kad/1.0.0` are added (since 1.4.1, as go). Two client-only nodes cannot find each other through each other; at least one reachable `server` node is needed, as the relay in the chat demo is. - **Bootstrap addresses must end in `/p2p/`.** `bootstrapPeers` takes `MultiAddr`, not `AddrInfo`. An address without a peer ID is skipped with a warning. Unreachable bootstrap peers are logged, not thrown; `bootstrap()` has a 30 s timeout and throws `DHTBootstrapException` only for other failures. Check `await dht.getRoutingTableSize()` afterwards instead of assuming success. - **Calling a query before `start()` throws `DHTNotStartedException`; after `close()` it throws `DHTClosedException`.** A closed DHT cannot be restarted; build a new one. - **Your own node is in `findProvidersAsync` results.** `provide` adds the local peer (with no addresses) to the local store first. Skip `p.id == host.id`, and skip providers with empty `addrs` or dial them through a relay (`/p2p-circuit/p2p/`), as the demo does. - **Provider keys are the CID's multihash (since 1.5.1).** CIDv0 and CIDv1 of the same multihash, any codec, share providers; this is what go and the spec do. Dart nodes older than 1.5.1 used full CID bytes: an updated node looking up providers on an old Dart server does not find them. A custom `ProviderStore` must key by `providerKeyOf(cid)`. - **`findProvidersAsync(cid, 0)` means no limit** (fixed in 1.6.0; before it returned local providers only). - **Namespace keys changed in 1.6.0.** `advertise`/`findPeers` now use `namespaceToCid(ns)` and accept any string (they threw for non-CID strings). Advertisements made by older versions are not found. - **Provider records expire.** `provideValidity` is 48 h on the storing node and nothing re-provides for you. `provide(cid, true)` announces to only `resiliency` (default 3) of the closest peers found. Re-provide on a timer, well inside the validity window. - **A value key needs a namespace validator.** `putValue('my-key', ...)` throws `DHTProtocolException`; keys look like `//`. The default validator has `pk` and `ipns` only (the old `v` namespace is gone). Passing `validator:` to `IpfsDHTv2` replaces the default, so add `pk` and `ipns` back if you need them. - **`select()` decides which record wins, not timestamps.** Since 1.5.0 the writer-chosen `timeReceived` no longer decides. If `validate` does not tie the key to an owner, anyone can replace the value with one your `select` prefers. - **`allowUnvalidatedRecords: true` is weak.** It accepts signed records in namespaces without a validator, and the first author to store a key on a node keeps it there until expiry. Trusted or test networks only. - **Values expire after `maxRecordAge` (36 h) and are not republished.** Put them again to keep them alive. - **`getValue` asks the network** (since 1.5.0) even with a local copy and returns the best valid record; pass `RoutingOptions()..offline = true` for local only. `/pk/` keys hold raw multihash bytes after the prefix: `'/pk/${String.fromCharCodes(peerId.toBytes())}'`. - **`DHTConfigBuilder` loses settings.** `IpfsDHTv2` takes `DHTOptions`, and `DHTConfigV2.toOptions()` keeps only the `DHTOptions` fields: `networkTimeout`, `queryTimeout`, `enableMetrics`, `optimisticProvide` set on the builder do not reach the DHT. Pass `DHTOptions` directly. - **`autoRefresh` defaults to `true`** and starts a refresh manager (5-minute interval) that runs a refresh during `start()`. Tests set `autoRefresh: false` to keep them quiet and fast. - **Upgrade if you see `RangeError ... 2..4096` against Go peers.** Before 1.6.0 a large `FIND_NODE`/`GET_PROVIDERS` answer split over several yamux frames failed to decode. Before 1.5.0, Go's second request on a reused stream got no answer. - **The library logs through `package:logging`** (no `print` since 1.4.1). Set `Logger.root.level` and add a listener to see bootstrap or query problems; per-peer failures are at `FINE`. - **go-libp2p interop is tested** (`test/interop/go_dht_interop_test.dart`): FIND_NODE via a Go bootstrap, `/pk/` records both ways, providers both ways via Go and Dart v2 servers. ## Testing - Put several real hosts on loopback UDX in one process. The repo's `test/dht/v2/real_net_stack.dart` has `createLibp2pNode(udxInstance: UDX(), resourceManager: NullResourceManager(), connManager: ConnectionManager(), hostEventBus: BasicBus())` with an addrs factory that keeps loopback addresses; copy that pattern or build hosts as in the `dart-libp2p-host` skill. - Run every node as `DHTMode.server` with `autoRefresh: false`; point each node's `bootstrapPeers` at the first node's `'${udxAddr}/p2p/${host.id.toBase58()}'`, then `await dht.bootstrap()`. - Test auto mode by emitting reachability yourself: ```dart final emitter = await host.eventBus.emitter(EvtLocalReachabilityChanged); await emitter.emit(EvtLocalReachabilityChanged(reachability: Reachability.public)); await emitter.close(); // poll until dht.isServer is true ``` - Seed a node's datastore without the network with `RecordSigner.createSignedRecord(key:, value:, privateKey:, peerId:)` (from `src/record/record_signer.dart`) and `dht.putRecordToDatastore(record)`. - Close DHTs, then hosts, in `tearDown` with a timeout (`.timeout(Duration(seconds: 5)).catchError((_) {})`). Give network tests a 30 to 60 s `Timeout`. - Lookups are asynchronous: poll for a condition instead of a fixed sleep. - Go interop tests need Go and a dart_libp2p checkout (`GO_PEER_DIR`, or `../dart-libp2p/interop/go-peer`). ## Red lines - Never start the host before the DHT on a node that should serve or be found by Go peers. - Never run `DHTMode.server` on a node that is not publicly reachable (mobile, behind NAT): peers will route queries to it that fail. - Never use `allowUnvalidatedRecords: true` on a public network, and never register a validator whose `validate` accepts anything for data that must not be overwritten. - Never key a custom `ProviderStore` by full CID bytes; use `providerKeyOf(cid)`. - Never change the protocol ID, wire format or key derivation (`namespaceToCid`, multihash provider keys): it splits you from go-libp2p and from other Dart nodes. - Always `close()` the DHT (it holds timers, stream handlers and an event bus subscription), then the host. ## Reference - Docs: https://cryptopeer.net/libp2p/dht/ - Repository: https://github.com/stephanfeb/dart_libp2p_kad_dht (`README.md`, `CHANGELOG.md`, `DEVELOPER_GUIDE.md`, `example/`, `test/dht/v2/`, `test/interop/`) - Real usage: `dart-libp2p-demos/internet_chat` (`lib/src/peer_node.dart`, `lib/src/relay_node.dart`): DHT client peers, a DHT server relay, rendezvous by provider records. - Spec: https://github.com/libp2p/specs/blob/master/kad-dht/README.md - Host setup: the `dart-libp2p-host` skill; messaging: the `dart-libp2p-gossipsub` skill.