Kademlia DHT

A comprehensive Dart implementation of the libp2p Kademlia Distributed Hash Table (DHT) for building decentralized peer-to-peer applications. This library provides the core infrastructure for peer discovery, content routing, and distributed key-value storage in P2P networks.

🚀 Featured Implementation: IpfsDHTv2

IpfsDHTv2 is our flagship implementation featuring a modular, production-ready architecture with enhanced performance, observability, and maintainability. It’s a drop-in replacement for the original IpfsDHT with significant improvements.

🌟 Key Improvements in v2

• 🔄 Modular Architecture: Clean separation of concerns with focused components
• 📊 Built-in Observability: Comprehensive metrics and monitoring out of the box
• 🛡️ Enhanced Error Handling: Structured exceptions with retry logic
• ⚡ Performance Optimized: Parallel operations and efficient routing
• 🧪 Testing-Friendly: Dependency injection for easy testing
• 🔧 Flexible Configuration: Builder pattern for complex setups

🌟 Features

Core DHT Capabilities

• Peer Discovery: Find peers by their ID across the network using Kademlia routing
• Content Routing: Discover who has specific content using content addressing (CID)
• Distributed Storage: Store and retrieve key-value pairs across the network
• Service Discovery: Advertise and find services in the P2P network
• Provider Records: Track and announce content availability across the network

Network Modes

• Client Mode: Lightweight mode for mobile and resource-constrained devices
• Server Mode: Full participant mode for infrastructure and bootstrap nodes
• Auto Mode: Automatically switches between client/server based on network conditions

Advanced Features

• Bootstrap Integration: Easy connection to existing libp2p networks with configurable bootstrap peers
• Routing Table Management: Kademlia-based peer routing with configurable bucket sizes
• Query Engine: Efficient parallel query execution with configurable concurrency
• Retry Logic: Configurable retry mechanisms with exponential backoff
• Network Size Estimation: Built-in network size estimation capabilities

Production-Ready Components

• Cryptographic Validation: Signed records with anti-replay protection
• 4-Phase Bootstrap: Comprehensive network connectivity with health monitoring
• Provider Operations: Complete network-integrated provider management
• Datastore Operations: Full local record storage with validation
• Metrics & Monitoring: Real-time performance tracking and health checks

🚀 Quick Start

Installation

Add the package to your pubspec.yaml:

dependencies:
  dart_libp2p_kad_dht: ^1.1.0
  dart_libp2p: ^0.5.2
  dcid: ^1.0.0

Basic Usage with IpfsDHTv2

import 'package:dart_libp2p_kad_dht/src/dht/v2/dht_v2.dart';
import 'package:dart_libp2p/dart_libp2p.dart';

Future<void> main() async {
  // Create a libp2p host
  final host = await createLibp2pHost();

  // Create a provider store for content routing
  final providerStore = MemoryProviderStore();

  // Create and start the DHT v2 (recommended)
  final dht = IpfsDHTv2(
    host: host,
    providerStore: providerStore,
    options: const DHTOptions(
      mode: DHTMode.auto,
      bucketSize: 20,
      concurrency: 10,
    ),
  );

  await dht.start();
  await dht.bootstrap(); // Connect to the network

  // Find a peer
  final peerInfo = await dht.findPeer(targetPeerId);

  // Store a value (cryptographically signed)
  await dht.putValue('my-key', utf8.encode('my-value'));

  // Retrieve a value (with validation)
  final value = await dht.getValue('my-key');

  // Announce content availability
  await dht.provide(CID.fromString('QmExample...'), true);

  // Find content providers
  await for (final provider in dht.findProvidersAsync(CID.fromString('QmExample...'), 10)) {
    print('Found provider: ${provider.id}');
  }

  // Check metrics
  final metrics = dht.metrics;
  print('Success rate: ${metrics.querySuccessRate * 100}%');

  // Cleanup
  await dht.close();
  await host.close();
}

Critical: Initialization Order

You must start the DHT before starting the host. When host.start() is called, AutoRelay immediately connects to relay servers and triggers an Identify exchange. If the DHT hasn’t registered its protocol handler yet, the first Identify response will be missing /ipfs/kad/1.0.0. Go-based peers that receive this will mark your node as “peer stopped dht” and refuse to open DHT streams to it — breaking peer discovery permanently for that connection.

// CORRECT — DHT starts first, then host
final host = await createLibp2pHost(); // Do NOT call host.start() yet
final dht = IpfsDHTv2(host: host, providerStore: store, options: options);
await dht.start();       // Registers /ipfs/kad/1.0.0 protocol handler
await host.start();      // Now AutoRelay's Identify will include DHT protocol
await dht.bootstrap();

// WRONG — host starts before DHT
final host = await createLibp2pHost();
await host.start();      // AutoRelay connects, Identify sent WITHOUT /ipfs/kad/1.0.0
final dht = IpfsDHTv2(host: host, providerStore: store, options: options);
await dht.start();       // Too late — Go already marked us "peer stopped dht"

This applies to any protocol handler that must be advertised via Identify — always register handlers before host.start().

Advanced Configuration with Builder Pattern

import 'package:dart_libp2p_kad_dht/src/dht/v2/config/dht_config.dart';

// Use builder pattern for complex configuration
final config = DHTConfigBuilder()
  .mode(DHTMode.server)
  .bucketSize(25)
  .concurrency(15)
  .filterLocalhost(false)
  .networkTimeout(Duration(seconds: 30))
  .queryTimeout(Duration(seconds: 60))
  .enableMetrics(true)
  .optimisticProvide(true)
  .build();

final dht = IpfsDHTv2(
  host: host,
  providerStore: providerStore,
  options: config.toOptions(),
);

📚 Examples

Interactive P2P Node (v2)

Run a fully interactive P2P node with all DHT operations:

dart run example/basic_p2p_node.dart

Available Commands:

• stats - Show network statistics and metrics
• store <key> <value> - Store a cryptographically signed key-value pair
• get <key> - Retrieve a validated value
• announce <content-id> - Announce content availability
• find-content <content-id> - Find content providers
• find-peer <peer-id> - Find a specific peer
• metrics - Show detailed performance metrics
• quit - Exit the demo

Mobile Optimized Node

For resource-constrained environments:

dart run example/mobile_p2p_node.dart

Server Node

High-performance server deployment:

dart run example/server_node.dart --port 4001

🔧 Configuration

DHT v2 Options

final dhtOptions = DHTOptions(
  mode: DHTMode.auto,        // client, server, or auto
  bucketSize: 20,            // K-bucket size
  concurrency: 10,           // Concurrent operations
  resiliency: 3,             // Query redundancy
  bootstrapPeers: [          // Network entry points
    AddrInfo(peerId1, [addr1]),
    AddrInfo(peerId2, [addr2]),
  ],
);

Advanced Configuration with Builder

final config = DHTConfigBuilder()
  .mode(DHTMode.server)
  .bucketSize(30)
  .concurrency(20)
  .resiliency(5)
  .networkTimeout(Duration(seconds: 30))
  .queryTimeout(Duration(seconds: 60))
  .enableMetrics(true)
  .optimisticProvide(true)
  .maxRetryAttempts(5)
  .retryInitialBackoff(Duration(milliseconds: 250))
  .build();

🏗️ Architecture

DHT v2 Modular Components

┌─────────────────────────────────────────────────────────────┐
│                        IpfsDHTv2                           │
│                  (Main Interface)                          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐         │
│  │  Network    │  │  Routing    │  │   Query     │         │
│  │  Manager    │  │  Manager    │  │  Manager    │         │
│  └─────────────┘  └─────────────┘  └─────────────┘         │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐                          │
│  │  Protocol   │  │  Metrics    │                          │
│  │  Manager    │  │  Manager    │                          │
│  └─────────────┘  └─────────────┘                          │
└─────────────────────────────────────────────────────────────┘

Component Responsibilities

• NetworkManager: Message handling, connection management, retry logic
• RoutingManager: Routing table management, peer discovery, bootstrap
• QueryManager: Query coordination, peer lookups, value operations
• ProtocolManager: Protocol message processing, request/response handling
• MetricsManager: Performance monitoring, error tracking, health checks

📊 Monitoring & Metrics

DHT v2 provides comprehensive monitoring out of the box:

final metrics = dht.metrics;

// Query metrics
print('Total queries: ${metrics.totalQueries}');
print('Success rate: ${metrics.querySuccessRate * 100}%');
print('Average latency: ${metrics.averageQueryLatency.inMilliseconds}ms');

// Network metrics
print('Network requests: ${metrics.totalNetworkRequests}');
print('Network success rate: ${metrics.networkSuccessRate * 100}%');

// Routing table metrics
print('Routing table size: ${metrics.routingTableSize}');
print('Peers added: ${metrics.peersAdded}');

// Performance metrics
print('Queries per second: ${metrics.queriesPerSecond}');
print('Network requests per second: ${metrics.networkRequestsPerSecond}');

🛡️ Error Handling

DHT v2 provides structured error handling:

try {
  final peer = await dht.findPeer(targetPeerId);
} on DHTNetworkException catch (e) {
  print('Network error: ${e.message}');
  // Handle network-specific errors
} on DHTTimeoutException catch (e) {
  print('Timeout error: ${e.message}');
  // Handle timeout-specific errors
} on DHTException catch (e) {
  print('DHT error: ${e.message}');
  // Handle general DHT errors
}

🧪 Testing

Run the comprehensive test suite:

dart test

The test suite includes:

• Unit tests for all components
• Integration tests with real network scenarios
• Performance benchmarks
• Mobile device simulation tests

📖 Documentation

• Developer Guide: Comprehensive guide for P2P application developers
• Examples: Detailed examples and use cases
• DHT v2 Documentation: Complete v2 architecture guide
• Integration Tests: Real-world usage patterns

🔄 Migration from Original DHT

Simple Migration

// Old
import 'package:dart_libp2p_kad_dht/src/dht/dht.dart';
final dht = IpfsDHT(host: host, providerStore: providerStore, options: options);

// New (recommended)
import 'package:dart_libp2p_kad_dht/src/dht/v2/dht_v2.dart';
final dht = IpfsDHTv2(host: host, providerStore: providerStore, options: options);

Benefits of Migration

• Better error handling: Structured exceptions and retry logic
• Improved observability: Built-in metrics and monitoring
• Better testability: Modular design and dependency injection
• Enhanced performance: Optimized query patterns and caching
• Future-proof: Easier to extend and maintain

🤝 Contributing

We welcome contributions! Please see our contributing guidelines:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

Development Setup

# Clone the repository
git clone https://github.com/stephanfeb/dart_libp2p_kad_dht.git
cd dart_libp2p_kad_dht

# Install dependencies
dart pub get

# Run tests
dart test

# Run examples
dart run example/basic_p2p_node.dart

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Based on the go-libp2p-kad-dht implementation
• Implements the Kademlia DHT algorithm
• Built for the libp2p networking stack

🔗 Related Projects

• dart_libp2p: Core libp2p networking library
• dcid: Content addressing utilities
• dart_udx: UDP-based transport layer

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: Developer Guide

---

Built with ❤️ for the decentralized web