Skip to content

Latest commit

 

History

History
185 lines (145 loc) · 49.9 KB

contribution-map.md

File metadata and controls

185 lines (145 loc) · 49.9 KB
Raw

A map that guides what and how contributors can contribute

Table of Contents

TiDB is an open-source distributed HTAP database compatible with the MySQL protocol

Contribution tutorial:

Module Description Code Directory Required Skills Learning Materials Source Reading
Parser A Yacc generated parser that is highly compatible with MySQL syntax. parser Golang, Yacc
  • Bison manual
  • Compilers: Principles, Techniques, and Tools
    		•  TiDB SQL Parser Implementation(CN)
    DDL This module is used to realize that when different TiDBs in a cluster transition to the new schema, all TiDBs can access and update all data online. ddl Golang
  • Online, Asynchronous Schema Change in F1
  • Deep analysis of Google F1's schema change algorithm(CN)
  • TiDB DDL architecture
    		•  TiDB DDL Implementation(CN)
    MySQL Protocol server
    Transaction A transaction symbolizes a unit of work performed within TiDB. The transaction module in TiDB plays the role of coordinator of distributed transactions.
  • session
  • kv
  • tikv/client-go
    		•  Golang, ACID, 2PC, Percolator transaction model
  • A Critique of ANSI SQL Isolation Levels
  • The Art of Database Transaction Processing: Transaction Management and Concurrency Control (Database Technology Series)(Chinese Edition)
  • Two-phase commit protocol
  • Large-scale Incremental Processing Using Distributed Transactions and Notifications
    		•  Two phase committer(CN)
    Planner planner Golang
    Executor
  • executor
  • expression
  • chunk
  • tikv/client-go
    		•  Golang, Relational Algebra, Parallel Programming CMU: Advanced DB System - Query Execution
  • Introduction to Chunk and Execution Framework(CN)
  • TiDB Hash Join(CN)
  • Index Lookup Join(CN)
  • Hash Aggregation(CN)
    		•

    TiKV, distributed transactional key-value database

    Contribution tutorial:

    Module Description Code Directory Required Skills Learning Materials Source Reading What I can Contribute
    Util Utilities like thread-pool, logger, encoding and decoding, etc.
  • Utilities
  • Pipeline batch system
    		•  Rust
  • Rust book
  • Practical networked applications in Rust
  • Protocol buffers
  • gRPC concepts
    		•  Service layer(CN)
    Network Network layer Server Rust, Protobuf, gRPC Ditto
    Raw KV API
  • API entrance
  • Storage struct
    		•  Rust Ditto
    Transaction KV API
  • API entrance
  • Transaction
  • GC worker
  • Pessimistic transaction
    		•  Rust, 2PC, Percolator transaction model
  • Two Phase Commit
  • Percolator paper
    		•
  • Storage Layer(CN)
  • Distributed transaction(CN)
  • MVCC read(CN)
    		•
    Multi-raft Raftstore Rust, Raft
  • Raft paper
  • Raft implementation in etcd
    		•
  • Raftstore overview(CN)
    		•
    Engine
  • Engine traits
  • Engine rocks
    		•  Rust, RocksDB Engine abstraction
    Coprocessor
  • TiDB query aggr
  • TiDB query codegen
  • TiDB query common
  • TiDB query executors
  • TiDB query expr
  • TiDB query datatype
    		•  Rust
  • Coprocessor Overview(CN)
  • Coprocessor Executor(CN)
    		•  Coprocessor Issues
    Backup Backup

    PD Placement driver for TiKV

    Module Description Code Directory Required Skills Learning Materials What I can Contribute
    API HTTP interfaces for interacting with PD server API Golang
  • Golang http lib
  • Golang Mux
    		•  OpenAPI Specification
    TSO Centralized timestamp allocation TSO Golang, etcd
  • Lamport timestamps
  • etcd clientv3
    		•  Optimize the performance
    Core Basic facilities Core Golang PD best practice(CN) Improve hotspot recognition; adaptive Scheduling; scheduling according to region histogram
    Statistics Statistics which the scheduling relies on Statistics Golang Ditto Ditto
    Scheduler Controllers of scheduling Schedulers Golang Ditto Ditto
    Schedule Components related to scheduling like selector, filter, etc. Schedule Golang Ditto Ditto

    TiKV Clients

    Module Required Skills Learning Materials What I can Contribute
    Rust Client Rust, TiKV concepts, Transaction model
  • TiKV documentation
  • Rust book
  • Practical networked applications in Rust
  • Percolator paper
    		•  Issues
    Go Client Golang, TiKV concepts, Transaction model Ditto Issues
    Java Client Java, TiKV concepts, Transaction model Ditto Issues
    Python Client Python, TiKV concepts, Transaction model Ditto Issues

    Libraries depended by TiKV

    Module Description Code Directory Required Skills Learning Materials Source Reading What I can Contribute
    grpc-rs The gRPC library for Rust built on C Core library and futures grpc c bindings, grpc rust wrapper C++, Rust
  • grpc introduction
  • Rust bindgen
  • Rust async book
    		•
  • gRPC server Initialization(CN)
  • grpc-rs Implementation(CN)
    		•  Load balance; reflection; custom metadata; migrate to async/await
    fail-rs A fail point implementation for Rust. Rust FreeBSD's failpoints Introduction for fail-rs(CN)
    raft-rs Raft distributed consensus algorithm implemented in Rust Rust, Raft
  • Raft paper
  • Raft implementation in etcd
    		•
    rust-prometheus Prometheus instrumentation library for Rust applications Rust
  • Prometheus doc
  • Prometheus Go client
    		•
  • rust-prometheus basic
  • rust-prometheus advance
    		•
    rust-rocksdb Rust wrapper for RocksDB Rust, C++, RocksDB
  • Rust ffi
  • Rust bindgen
  • RocksDB wiki
    		•  Migrate bindgen to C++
    Titan A RocksDB plugin for key-value separation C++, LSM-tree, RocksDB
  • Titan storage design and implementation
  • WiscKey: Separating Keys from Values in SSD-conscious Storage
    		•

    Ecosystem Tools: DM Data Migration Platform

    Module Description Code Directory Required Skills Learning Materials What I can Contribute Contributing Tutorials
    DM-master The component for managing and scheduling the operation of data replication task dm-master Golang, etcd, gRPC, Protobuf DM overview, source code reading: DM(CN) Issues
    DM-worker The component for executing specific data replication tasks dm-worker Golang, etcd, gRPC, Protobuf Ditto Ditto
    dmctl A command line tool used to control the DM cluster dm-ctl Golang, gRPC, Protobuf Ditto Ditto
    Dump unit The processing unit for exporting SQL data from MySQL mydumper Golang, MySQL Ditto Ditto
    Import/Load unit The processing unit for importing SQL data into TiDB loader Golang, MySQL Ditto Ditto
    Relay unit The processing unit for dumping binlog data from MySQL relay Golang, MySQL Ditto Ditto
    Binlog replication/sync unit The processing unit for replicating binlog data into TiDB syncer Golang, MySQL Ditto Ditto
    Checker unit The processing unit for checking restrictions of the replication checker Golang, MySQL Ditto Ditto
    DM-portal A web-based application for managing data migration task configuration portal Golang, MySQL, Frontend Ditto Ditto
    DM-ansible The ansible playbook for DM dm-ansible DM, Ansible Ditto Ditto
    DM-tracer A DM-specific tracer framework dm-tracer Golang, gRPC, Protobuf Ditto Ditto

    Ecosystem Tools - Binlog : A tool used to collect and merge tidb's binlog for real-time data backup and synchronization

    Module Description Code Directory Required Skills Learning Materials What I can Contribute Contributing Tutorials
    Pump Client Send Binlog to the appropriate Pump on TiDB pump_client, binloginfo Golang, gRPC, etcd TiDB Binlog overview, TiDB source code reading: Binlog(CN) Help Wanted Issues
    Pump Receive and store Binlogs, sorted by commitTS pump Golang, LevelDB, etcd, gRPC Ditto Ditto
    Drainer Pull Binlog from Pump, do merge and sort, and then sync to downstream drainer Golang, Mysql, Kafka, gRPC Ditto Ditto

    Ecosystem Tools - Lightning: A high-speed data import tool for TiDB

    Module Description Code Directory Required Skills Learning Materials What I can Contribute Contributing Tutorials
    Web Interface web
    CSV and SQL Parser mydump
    Delivery backend (output as SQL or as KV pairs) backend
    Checkpoints checkpoints
    Configuration config
    Logging log
    Metrics metric
    Workers (concurrency control) worker
    Restore Driver restore
    Utilities common
    Importer import

    Ecosystem Tools - BR: A command-line tool for distributed backup and restoration of the TiDB cluster data

    Module Description Code Directory Required Skills Learning Materials What I can Contribute Contributing Tutorials
    Backup Backup data in cluster which has 3 mode: backup full cluster, backup specified DB and backup specified Table br/backup, tikv/backup gRPC, golang, rust backup-principle
    Restore Restore data to new cluster after backup, relatively can restore full cluster, restore specified DB and restore specified Table. br/restore, tikv/sst_importer gRPC, golang, rust restoration-principle

    TiDB on K8S/Docker : Creates and manages TiDB clusters running in Kubernetes

    Module Description Code Directory Required Skills Learning Materials What I can Contribute Contributing Tutorials
    TiDB Operator TiDB Operator manages TiDB clusters on Kubernetes and automates tasks related to operating a TiDB cluster. It makes TiDB a truly cloud-native database. tidb-operator Golang, Kubernetes, TiDB how to use tidb-operator, Kubernetes Concepts, Kubernetes client-go SDK, Kubernetes Operator Pattern, TiDB Architecture, Helm, Terraform Help Wanted Issues Contributing Guide
    Advanced StatefulSet This is an Advanced StatefulSet CRD implementation based on official StatefulSet in Kubernetes. In addition to the official StatefulSet, it can scale the set in an arbitrary way. advanced-statefulset Golang, Kubernetes Kubernetes Concepts, Kubernetes StatefulSet, Kubernetes StatefulSet Implementations, PingCAP StatefulSet Implementations
    TiDB Docker Compose Docker compose files for TiDB. With it, you can quickly deploy a TiDB testing cluster with a single command. tidb-docker-compose Docker, Docker Compose how to start tidb with tidb-docker-compose, docker get started, docker compose

    Deployment Tools - tidb-ansible: A TiDB cluster deployment tool based on Ansible playbook.

    Module Description Code Directory Required Skills Learning Materials What I can Contribute Contributing Tutorials
    tidb ansible TiDB deployment operation and maintenance tools Ansible, Python, Jinja2, Shell How to deploy TiDB cluster through tidb-ansible, Ansible docs(en) and docs(cn)

    Chaos-Mesh: A Chaos Engineering Platform for Kubernetes

    Module Description Code Directory Required Skills Learning Materials What I can Contribute Contributing Tutorials
    chaos-mesh go, kubenetes

    Documentation

    Project Required Skills Learning Materials What I can Contribute Contributing Tutorials
    Documentation(EN) Sharp eyes, Good English writing, Technical writing or communication, Translation from English to Chinese, Knowledge about TiDB Commit Message and Pull Request Style, Code Comment Style Fix typos or format (punctuation, space, indentation, code block, etc.); Fix or update inappropriate or outdated descriptions; Add missing content (sentence, paragraph, or a new document); Translate docs changes from English to Chinese; Submit, reply to, and resolve issues; (Advanced) Review Pull Requests docs readme, docs contribution guide
    Documentation(CN) Sharp eyes, Good English writing, Technical writing or communication, Translation from English to Chinese, Knowledge about TiDB Chinese documentation style guide and template, Commit Message and Pull Request Style, Code Comment Style Ibid, except for translating document changes from Chinese to English docs-cn readme, docs-cn contribution guide

    TiDB User Group(CN)

    Required Skills Learning Materials What I can Contribute Contributing Tutorials
    Anyone of ansible, linux, DM, Binlog, lightning, pd, tidb, tikv, grafana, prometheus. Any experience in using RDBMS like MySQL or other RDBMS. TiDB DBA courses Help other TiDB users by answering questions in AskTUG; Publish practice articles of using TiDB or other databases; Giving presentations at TUG activities to share experience with TiDB; Involving in the organization/management of TUG (TiDB User Group) to expand the user base of TiDB Join TUG

    PingCAP University(CN)

    Required Skills Learning Materials What I can Contribute Contributing Tutorials
    TiDB DBA Course Design; TiDB Contributor Course Design; Sharing notes on TiDB; Become an administrator of the Wechat (online) group of learning; Joining the group to make subtitles for the video courses or translate them; Helping answer questions on TiDB; Notes, subtitles, translation, FAQ Join PU

    TiDB Developer Group

    Channel Description
    TiDB Internals Forum This is the hub for discussing everything related to the implementation and design of TiDB, TiKV and other TiDB ecosystem projects.
    TiDB Slack Channel The official Slack channel of the TiDB community.
    TiKV Slack Channel The official Slack channel of the TiKV community.
    Documentation Slack Channel The official Slack channel of the TiDB Documentation community.