Skip to content
cryptopeercryptopeer.net

dart-libp2p-kad-dht

Skill dart-libp2p-kad-dht. Use when a dart_libp2p app needs the Kademlia DHT (package:dart_libp2p_kad_dht, IpfsDHTv2): bootstrapping, findPeer, provide / findProvidersAsync content routing, putValue / getValue signed records with namespace validators, advertise / findPeers rendezvous, client / server / auto modes, go-libp2p interop.

SKILL.md (Claude Code, Codex, Cursor) · dart-libp2p-kad-dht.md (Boubou) · install: curl -fsSL https://cryptopeer.net/skills/install.sh | sh -s -- dart-libp2p-kad-dht

What it is

Section titled “What it is”

A Dart port of go-libp2p-kad-dht on top of dart_libp2p. It speaks /ipfs/kad/1.0.0 with the protobuf wire format, so Dart and go-libp2p nodes share one DHT. It gives peer routing (findPeer), content routing (provide, findProvidersAsync), a validated key/value store (putValue, getValue) and namespace rendezvous (advertise, findPeers).

Use IpfsDHTv2 for new code. The legacy IpfsDHT still exists and follows the same record rules, but v2 is the one the demos and tests drive.

Use something else when:

  • Two peers already know each other’s address: just host.connect.
  • Peers are on the same LAN: mDNS is simpler.
  • You want to broadcast messages: GossipSub (dart-libp2p-gossipsub skill); the DHT only helps peers find each other.
  • Host, transports, relays, AutoNAT: see the dart-libp2p-host skill.

Install

Section titled “Install”
dependencies:
  dart_libp2p_kad_dht: ^1.6.0
  dart_libp2p: ^4.7.0    # this package accepts >=1.0.0 <5.0.0
  dcid: ^1.0.0           # CID
import 'dart:convert';                                     // utf8
import 'dart:typed_data';                                  // Uint8List
import 'package:dart_libp2p/dart_libp2p.dart';            // Host, PeerId, MultiAddr, AddrInfo, RoutingOptions
import 'package:dart_libp2p_kad_dht/dart_libp2p_kad_dht.dart'; // IpfsDHTv2, DHTOptions, DHTMode, MemoryProviderStore, namespaceToCid
import 'package:dcid/dcid.dart';                          // CID


// Not in the barrel; import from src when you need them:
import 'package:dart_libp2p_kad_dht/src/dht/v2/errors/dht_errors.dart';   // DHTException and subclasses
import 'package:dart_libp2p_kad_dht/src/record/validator.dart';            // Validator
import 'package:dart_libp2p_kad_dht/src/record/namespace_validator.dart';  // NamespacedValidator
import 'package:dart_libp2p_kad_dht/src/record/public_key_validator.dart'; // PublicKeyValidator
import 'package:dart_libp2p_kad_dht/src/record/ipns_validator.dart';       // IpnsValidator

Core workflow

Section titled “Core workflow”

Minimal node (start the DHT before the host)

Section titled “Minimal node (start the DHT before the host)”

Build the host as in the dart-libp2p-host skill, but do not call host.start() yet.

final dht = IpfsDHTv2(
  host: host,
  providerStore: MemoryProviderStore(),
  options: DHTOptions(
    mode: DHTMode.client, // behind NAT; DHTMode.server on a public node
    bootstrapPeers: [
      // must end in /p2p/<peer ID>
      MultiAddr('/ip4/203.0.113.7/udp/4001/udx/p2p/12D3KooW...'),
    ],
  ),
);
await dht.start();   // registers /ipfs/kad/1.0.0 (server modes) first
await host.start();  // now Identify advertises the DHT protocol
try {
  await dht.bootstrap(); // dials bootstrapPeers, fills the routing table
} on DHTBootstrapException catch (e) {
  // keep running; retry later
}


// ... use it ...


await dht.close();   // does not close the host
await host.close();

Bootstrap / relay node (public, serves the DHT)

Section titled “Bootstrap / relay node (public, serves the DHT)”

From internet_chat/lib/src/relay_node.dart. The host is created with Libp2p.forceReachability(Reachability.public).

final dht = IpfsDHTv2(
  host: host,
  providerStore: MemoryProviderStore(),
  options: DHTOptions(mode: DHTMode.server),
);
await dht.start();
await host.start();
// Others use '$addr/p2p/${host.id.toBase58()}' as a bootstrap peer.

Find a peer

Section titled “Find a peer”
final AddrInfo? info = await dht.findPeer(PeerId.fromString('12D3KooW...'));
if (info != null) await host.connect(info);


final closest = await dht.getClosestPeers(somePeerId); // List<AddrInfo>

Content routing (provide / find providers)

Section titled “Content routing (provide / find providers)”
final cid = CID.fromData(CID.V1, 'raw', utf8.encode('my-app/room/lobby'));


await dht.provide(cid, true); // true: announce to the network; false: local only


await for (final p in dht.findProvidersAsync(cid, 20)) { // 0 = no limit
  if (p.id == host.id) continue;     // you are in your own results
  if (p.addrs.isNotEmpty) await host.connect(p);
}

Provider records expire (provideValidity, 48 h). Re-provide on a timer; the chat demo re-provides every minute and searches every 10 s.

Rendezvous by namespace

Section titled “Rendezvous by namespace”
final ttl = await dht.advertise('my-app/chat');          // Duration (provideValidity)
final peers = await dht.findPeers('my-app/chat');        // Future<Stream<AddrInfo>>
await for (final p in peers) { /* ... */ }

The key is namespaceToCid(ns): CIDv1, raw, sha2-256 of the UTF-8 namespace, the same as nsToCid in go-libp2p routing discovery.

Value records with your own namespace

Section titled “Value records with your own namespace”
class GreetingValidator implements Validator {
  @override
  Future<void> validate(String key, Uint8List value) async {
    if (value.length > 1024) throw Exception('value too large');
    // To stop others overwriting it, tie the key to its owner here
    // (e.g. require a signature by the peer ID in the key).
  }


  @override
  Future<int> select(String key, List<Uint8List> values) async => 0; // index of best
}


final dht = IpfsDHTv2(
  host: host,
  providerStore: MemoryProviderStore(),
  validator: NamespacedValidator()
    ..['pk'] = PublicKeyValidator()
    ..['ipns'] = IpnsValidator(host.peerStore)
    ..['greeting'] = GreetingValidator(),
);
await dht.start();


await dht.putValue('/greeting/alice', Uint8List.fromList(utf8.encode('hello')));
final v = await dht.getValue('/greeting/alice');                    // asks the network
final local = await dht.getValue('/greeting/alice', RoutingOptions()..offline = true);
await for (final better in dht.searchValue('/greeting/alice', null)) { /* ... */ }

Errors

Section titled “Errors”
try {
  await dht.findPeer(id);
} on DHTTimeoutException catch (e) {
  // ...
} on DHTNetworkException catch (e) {
  // ...
} on DHTException catch (e) {
  print(e.message);
}

Key concepts / API map

Section titled “Key concepts / API map”
  • IpfsDHTv2({required host, required providerStore, DHTOptions? options, NamespacedValidator? validator}): the DHT; implements IpfsDHT (Routing, Discovery).
  • start() / bootstrap({periodicRefreshInterval, quickConnectOnly}) / close(): lifecycle; bootstrap starts the DHT if needed.
  • started, isServer: state; isServer is whether /ipfs/kad/1.0.0 is registered now.
  • findPeer(PeerId, {RoutingOptions? options}) -> Future<AddrInfo?>.
  • getClosestPeers(PeerId, {networkQueryEnabled}) -> Future<List<AddrInfo>>.
  • provide(CID, bool announce), findProvidersAsync(CID, int count) -> Stream<AddrInfo>.
  • addProvider(Uint8List key, AddrInfo), getLocalProviders(Uint8List key): local provider store, key is a multihash (providerKeyOf(cid)).
  • putValue(String key, Uint8List value, {RoutingOptions? options}), getValue(key, [RoutingOptions?]), searchValue(key, RoutingOptions?) -> Stream<Uint8List>.
  • advertise(String ns), findPeers(String ns): discovery on namespaceToCid(ns).
  • metrics -> DHTMetrics (totalQueries, querySuccessRate, routingTableSize, networkSuccessRate, …).
  • routingTable, getRoutingTableSize(), getRoutingTablePeers(), refreshRoutingTable().
  • DHTOptions: mode (default auto), bucketSize 20, concurrency 10, resiliency 3, provideValidity 48 h, providerAddrTTL 24 h, autoRefresh true, bootstrapPeers (List<MultiAddr>), maxRetryAttempts 3, retryInitialBackoff, retryMaxBackoff, retryBackoffFactor, filterLocalhostInResponses true, maxRecordAge 36 h, allowUnvalidatedRecords false.
  • DHTMode: client, server, auto, autoServer.
  • MemoryProviderStore([ProviderManagerOptions options, PeerId? localPeerId]); ProviderManagerOptions has maxProvidersPerKey 100, maxKeysPerProvider 1000, maxProviderRecords 100000, cleanupInterval, provideValidity.
  • ProviderStore: interface (addProvider(CID, AddrInfo), getProviders(CID), close()) for your own store.
  • providerKeyOf(cid), providerKeyFromWire, cidForProviderKey, namespaceToCid(ns): key helpers.
  • Validator (validate, select), NamespacedValidator ([ns] = v), PublicKeyValidator, IpnsValidator(peerStore).
  • AminoConstants: protocolID (/ipfs/kad/1.0.0), defaults, inboundStreamIdleTimeout (1 min).
  • Errors (src/dht/v2/errors/dht_errors.dart): DHTException, DHTNotStartedException, DHTClosedException, DHTBootstrapException, DHTNetworkException, DHTRoutingException, DHTQueryException, DHTProtocolException, DHTTimeoutException, DHTMaxRetriesException, DHTConfigurationException.

Gotchas

Section titled “Gotchas”
  • Start the DHT before the host. host.start() makes AutoRelay connect and run Identify at once. If /ipfs/kad/1.0.0 is not registered yet, Go peers mark the node “peer stopped dht” and stop opening DHT streams to it on that connection. Order: create host, dht.start(), host.start(), dht.bootstrap(). The same applies to every protocol that Identify must advertise.
  • DHTMode.auto (the default) starts as a client. It serves only after the host emits EvtLocalReachabilityChanged with Reachability.public (AutoNAT, or forceReachability), and stops on private or unknown. On a loopback test network that event never comes, so nodes in auto never serve and lookups find nothing. Use DHTMode.server for bootstrap nodes and tests, DHTMode.client for NATed or mobile nodes. autoServer serves unless reachability is private.
  • Client mode does not serve and does not enter routing tables. Only peers whose Identify lists /ipfs/kad/1.0.0 are added (since 1.4.1, as go). Two client-only nodes cannot find each other through each other; at least one reachable server node is needed, as the relay in the chat demo is.
  • Bootstrap addresses must end in /p2p/<peer ID>. bootstrapPeers takes MultiAddr, not AddrInfo. An address without a peer ID is skipped with a warning. Unreachable bootstrap peers are logged, not thrown; bootstrap() has a 30 s timeout and throws DHTBootstrapException only for other failures. Check await dht.getRoutingTableSize() afterwards instead of assuming success.
  • Calling a query before start() throws DHTNotStartedException; after close() it throws DHTClosedException. A closed DHT cannot be restarted; build a new one.
  • Your own node is in findProvidersAsync results. provide adds the local peer (with no addresses) to the local store first. Skip p.id == host.id, and skip providers with empty addrs or dial them through a relay (<relay>/p2p-circuit/p2p/<id>), as the demo does.
  • Provider keys are the CID’s multihash (since 1.5.1). CIDv0 and CIDv1 of the same multihash, any codec, share providers; this is what go and the spec do. Dart nodes older than 1.5.1 used full CID bytes: an updated node looking up providers on an old Dart server does not find them. A custom ProviderStore must key by providerKeyOf(cid).
  • findProvidersAsync(cid, 0) means no limit (fixed in 1.6.0; before it returned local providers only).
  • Namespace keys changed in 1.6.0. advertise/findPeers now use namespaceToCid(ns) and accept any string (they threw for non-CID strings). Advertisements made by older versions are not found.
  • Provider records expire. provideValidity is 48 h on the storing node and nothing re-provides for you. provide(cid, true) announces to only resiliency (default 3) of the closest peers found. Re-provide on a timer, well inside the validity window.
  • A value key needs a namespace validator. putValue('my-key', ...) throws DHTProtocolException; keys look like /<ns>/<rest>. The default validator has pk and ipns only (the old v namespace is gone). Passing validator: to IpfsDHTv2 replaces the default, so add pk and ipns back if you need them.
  • select() decides which record wins, not timestamps. Since 1.5.0 the writer-chosen timeReceived no longer decides. If validate does not tie the key to an owner, anyone can replace the value with one your select prefers.
  • allowUnvalidatedRecords: true is weak. It accepts signed records in namespaces without a validator, and the first author to store a key on a node keeps it there until expiry. Trusted or test networks only.
  • Values expire after maxRecordAge (36 h) and are not republished. Put them again to keep them alive.
  • getValue asks the network (since 1.5.0) even with a local copy and returns the best valid record; pass RoutingOptions()..offline = true for local only. /pk/ keys hold raw multihash bytes after the prefix: '/pk/${String.fromCharCodes(peerId.toBytes())}'.
  • DHTConfigBuilder loses settings. IpfsDHTv2 takes DHTOptions, and DHTConfigV2.toOptions() keeps only the DHTOptions fields: networkTimeout, queryTimeout, enableMetrics, optimisticProvide set on the builder do not reach the DHT. Pass DHTOptions directly.
  • autoRefresh defaults to true and starts a refresh manager (5-minute interval) that runs a refresh during start(). Tests set autoRefresh: false to keep them quiet and fast.
  • Upgrade if you see RangeError ... 2..4096 against Go peers. Before 1.6.0 a large FIND_NODE/GET_PROVIDERS answer split over several yamux frames failed to decode. Before 1.5.0, Go’s second request on a reused stream got no answer.
  • The library logs through package:logging (no print since 1.4.1). Set Logger.root.level and add a listener to see bootstrap or query problems; per-peer failures are at FINE.
  • go-libp2p interop is tested (test/interop/go_dht_interop_test.dart): FIND_NODE via a Go bootstrap, /pk/ records both ways, providers both ways via Go and Dart v2 servers.

Testing

Section titled “Testing”
  • Put several real hosts on loopback UDX in one process. The repo’s test/dht/v2/real_net_stack.dart has createLibp2pNode(udxInstance: UDX(), resourceManager: NullResourceManager(), connManager: ConnectionManager(), hostEventBus: BasicBus()) with an addrs factory that keeps loopback addresses; copy that pattern or build hosts as in the dart-libp2p-host skill.
  • Run every node as DHTMode.server with autoRefresh: false; point each node’s bootstrapPeers at the first node’s '${udxAddr}/p2p/${host.id.toBase58()}', then await dht.bootstrap().
  • Test auto mode by emitting reachability yourself:
final emitter = await host.eventBus.emitter(EvtLocalReachabilityChanged);
await emitter.emit(EvtLocalReachabilityChanged(reachability: Reachability.public));
await emitter.close();
// poll until dht.isServer is true
  • Seed a node’s datastore without the network with RecordSigner.createSignedRecord(key:, value:, privateKey:, peerId:) (from src/record/record_signer.dart) and dht.putRecordToDatastore(record).
  • Close DHTs, then hosts, in tearDown with a timeout (.timeout(Duration(seconds: 5)).catchError((_) {})). Give network tests a 30 to 60 s Timeout.
  • Lookups are asynchronous: poll for a condition instead of a fixed sleep.
  • Go interop tests need Go and a dart_libp2p checkout (GO_PEER_DIR, or ../dart-libp2p/interop/go-peer).

Red lines

Section titled “Red lines”
  • Never start the host before the DHT on a node that should serve or be found by Go peers.
  • Never run DHTMode.server on a node that is not publicly reachable (mobile, behind NAT): peers will route queries to it that fail.
  • Never use allowUnvalidatedRecords: true on a public network, and never register a validator whose validate accepts anything for data that must not be overwritten.
  • Never key a custom ProviderStore by full CID bytes; use providerKeyOf(cid).
  • Never change the protocol ID, wire format or key derivation (namespaceToCid, multihash provider keys): it splits you from go-libp2p and from other Dart nodes.
  • Always close() the DHT (it holds timers, stream handlers and an event bus subscription), then the host.

Reference

Section titled “Reference”

Built by Stephan Februarygithub.com/stephanfebLinkedIn