Skip to content

igumus/blockstorage

Repository files navigation

Block Storage Example

A custom block storage example to manage (store/index etc.) block (aka file/document content) operations in a p2p manner.

Introduction

blockstorage is a p2p service that I wrote to understand the internal workings of block management. Important topics that I focusing on:

  1. Libp2p
    • Kademlia DHT
      • Content Routing usage
      • Peer Routing usage
      • P2P Protocol usage
    • Peer Discovery (Boostrap Peer, Rendezvous, etc.)
  2. IPLD
    • Concepts & Spec
    • Proof of Concept custom implementation (via protobuf)
  3. Formats

Libraries/Implementations

  • go-objectstore-lib : Contains base abstraction of storing encoded/serialized objects

  • go-objectstore-fs : Contains file system based implementation/functionality to store encoded/serialized objects

Layout

  • api/proto : Contains protobuf definitions
  • blockpb : Contains protobuf and grpc related objects according to store.proto
  • blockpb/store_aux.go : Contains auxiliary functions/definitions to extends proto objects
  • util/ctx.go : Contains context cheking helper function and error definitions
  • peer : Contains p2p functions and definitions.
  • errors.go : Contains blockstorage error definitions and error checking functions
  • grpc : Contains blockstorage GRPC endpoint definition and RPC function implementations
  • impl.go : Contains BlockStorage interface implementation and helper functions
  • options.go : Contains BlockStorage construction option definitions
  • peer.go : Contains p2p related protocol definition and functions
  • storage.go : Contains blockstorage construction and BlockStorage interface definition

Status

blockstorage is still in progress.

TODOs

  • Prepare Step
    • Project Structure
    • Makefile tasks
  • Grpc Step
    • proto file
    • generate objects
    • add makefile task
    • implement service endpoint
  • Peer Step
    • protocol definition
    • store definition
  • Future
    • add block indexing mechanism
    • add garbage collection trigger mechanism (to temporaryStore)
    • add long term storage trigger