Distributed Key-Value Database

This is a lightweight, distributed key-value database built with Go. It features hash-based sharding, read-only replicas, and HTTP-based communication for efficient data storage and retrieval across multiple nodes.

Features

• Distributed Architecture: Utilizes hash-based sharding for data distribution across multiple nodes.
• Read-Only Replicas: Enhances read scalability with dedicated read-only replica nodes.
• Eventual Consistency: Implements asynchronous replication for eventual consistency between primary shards and replicas.
• HTTP-Based Communication: Enables seamless inter-shard operations and data retrieval.
• BoltDB Backend: Leverages BoltDB for efficient local storage on each node.
• Concurrent Operations: Uses Go's goroutines for non-blocking replica updates.

Architecture

The distributed key-value store consists of multiple shards, each responsible for a subset of the key space. Each shard has a primary node for read and write operations, and a read-only replica for enhanced read performance.

         Client
           |
           v
         Server
      /    |    \
     v     v     v
Shard1  Shard2    Shard3
  |        |        |
  v        v        v
Replica1 Replica2 Replica3

Getting Started

Prerequisites

• Go 1.16 or higher
• BoltDB

Installation

1. Clone the repository:

   git clone /~https://github.com/ShubhKanodia/Distributed_KV_DB.git

2. Navigate to the project directory:

   cd Distributed_KV_DB

3. Build the project:

   go build

Running the Distributed Key-Value Database

Use the provided launch.sh script to start multiple shards and their replicas:

./launch.sh

This script will launch three primary shards and their corresponding replicas.

Usage

Setting a Key-Value Pair

curl "http://localhost:8080/set?key=exampleKey&value=exampleValue"

Getting a Value

curl "http://localhost:8080/get?key=exampleKey"

Demo

Normal Get and Set Operations

Shard Redirection

When querying any shard, the server automatically redirects the request to the appropriate shard:

On querying shard1 at localhost:8081 for the value at shard0

On querying shard2 at localhost:8082 for the value at shard0

querying shard0 for the value at shard2

Replication and Read-Only Replicas

Replicas support read operations but reject write attempts:

Setting value at primary(shard1) on localhost:8081

Querying its replica on locahost:8084 for read

Trying to write a value to its replica using 'set' operation(rejected)

Configuration

The database uses a TOML configuration file (sharding.toml) to define shard and replica settings:

[[shards]]
name = "shard1"
idx = 0
address = "localhost:8080"
replica = "localhost:8083"

[[shards]]
name = "shard2"
idx = 1
address = "localhost:8081"
replica = "localhost:8084"

[[shards]]
name = "shard3"
idx = 2
address = "localhost:8082"
replica = "localhost:8085"