Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Indexer/TiDB][chore] Fix schema indentation and capitalization #17687

Merged
merged 2 commits into from
May 14, 2024
Merged

[Indexer/TiDB][chore] Fix schema indentation and capitalization #17687

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
970f08d
[Indexer/TiDB][EASY] Add pre-requisites to README.
amnn May 12, 2024
c394727
[Indexer/TiDB][chore] Fix schema indentation and capitalization
amnn May 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 34 additions & 4 deletions crates/sui-indexer/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,26 +59,56 @@ Run this command under `sui/crates/sui-indexer`, which will wipe DB; In case of
```sh
diesel database reset --database-url="<DATABASE_URL>"
```
### Local Development(TiDB)

## Steps to run locally (TiDB)

### Prerequisites

1. Install TiDB

``` sh
curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh
```

2. Install a compatible version of MySQL (At the time of writing, this is MySQL 8.0 -- note that 8.3 is incompatible).

``` sh
brew install [email protected]
```

3. Install a version of `diesel_cli` that supports MySQL (and probably also Postgres). This version of the CLI needs to be built against the version of MySQL that was installed in the previous step (compatible with the local installation of TiDB, 8.0.37 at time of writing).

``` sh
MYSQLCLIENT_LIB_DIR=/opt/homebrew/Cellar/[email protected]/8.0.37/lib/ cargo install diesel_cli --no-default-features --features postgres --features mysql --force
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

```

### Run the indexer

1.Run TiDB

```sh
curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh
tiup playground
```

2.Verify tidb is running by connecting to it using the mysql client, create database `test`

```sh
mysql --comments --host 127.0.0.1 --port 4000 -u root
create database test;
```
```

3.DB setup, under `sui/crates/sui-indexer` run:

```sh
# an example DATABASE_URL is "mysql://root:[email protected]:4000/test"
diesel setup --database-url="<DATABASE_URL> --migration-dir='migrations/mysql'"
diesel database reset --database-url="<DATABASE_URL> --migration-dir='migrations/mysql'"
```

Note that you'll need an existing database for the above to work. Replace `test` with the name of the database created.
4. run indexer as a writer, which pulls data from fullnode and writes data to DB

```sh
# Change the RPC_CLIENT_URL to http://0.0.0.0:9000 to run indexer against local validator & fullnode
cargo run --bin sui-indexer --features mysql-feature --no-default-features -- --db-url "<DATABASE_URL>" --rpc-client-url "https://fullnode.devnet.sui.io:443" --fullnode-sync-worker --reset-db
```
```
22 changes: 11 additions & 11 deletions crates/sui-indexer/migrations/mysql/2024-03-22-052019_events/up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,27 @@ CREATE TABLE events
(
tx_sequence_number BIGINT NOT NULL,
event_sequence_number BIGINT NOT NULL,
transaction_digest blob NOT NULL,
checkpoint_sequence_number bigint NOT NULL,
transaction_digest BLOB NOT NULL,
checkpoint_sequence_number BIGINT NOT NULL,
-- array of SuiAddress in bytes. All signers of the transaction.
senders json NOT NULL,
senders JSON NOT NULL,
-- bytes of the entry package ID. Notice that the package and module here
-- are the package and module of the function that emitted the event, diffrent
-- from the package and module of the event type.
package blob NOT NULL,
package BLOB NOT NULL,
-- entry module name
module text NOT NULL,
module TEXT NOT NULL,
-- StructTag in Display format, fully qualified including type parameters
event_type text NOT NULL,
event_type TEXT NOT NULL,
-- Components of the StructTag of the event type: package, module,
-- name (name of the struct, without type parameters)
event_type_package blob NOT NULL,
event_type_module text NOT NULL,
event_type_name text NOT NULL,
event_type_package BLOB NOT NULL,
event_type_module TEXT NOT NULL,
event_type_name TEXT NOT NULL,
-- timestamp of the checkpoint when the event was emitted
timestamp_ms BIGINT NOT NULL,
-- bcs of the Event contents (Event.contents)
bcs MEDIUMBLOB NOT NULL,
bcs MEDIUMBLOB NOT NULL,
PRIMARY KEY(tx_sequence_number, event_sequence_number, checkpoint_sequence_number)
) PARTITION BY RANGE (checkpoint_sequence_number) (
PARTITION events_partition_0 VALUES LESS THAN MAXVALUE
Expand All @@ -31,4 +31,4 @@ CREATE INDEX events_package ON events (package(255), tx_sequence_number, event_s
CREATE INDEX events_package_module ON events (package(255), module(255), tx_sequence_number, event_sequence_number);
CREATE INDEX events_event_type ON events (event_type(255), tx_sequence_number, event_sequence_number);
CREATE INDEX events_type_package_module_name ON events (event_type_package(128), event_type_module(128), event_type_name(128), tx_sequence_number, event_sequence_number);
CREATE INDEX events_checkpoint_sequence_number ON events (checkpoint_sequence_number);
CREATE INDEX events_checkpoint_sequence_number ON events (checkpoint_sequence_number);
113 changes: 56 additions & 57 deletions crates/sui-indexer/migrations/mysql/2024-03-25-203621_objects/up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,38 +1,38 @@
-- Your SQL goes here
CREATE TABLE objects (
object_id blob NOT NULL,
object_version bigint NOT NULL,
object_digest blob NOT NULL,
checkpoint_sequence_number bigint NOT NULL,
object_id BLOB NOT NULL,
object_version BIGINT NOT NULL,
object_digest BLOB NOT NULL,
checkpoint_sequence_number BIGINT NOT NULL,
-- Immutable/Address/Object/Shared, see types.rs
owner_type smallint NOT NULL,
owner_type SMALLINT NOT NULL,
-- bytes of SuiAddress/ObjectID of the owner ID.
-- Non-null for objects with an owner: Addresso or Objects
owner_id blob,
owner_id BLOB,
-- Object type
object_type text,
object_type TEXT,
-- Components of the StructTag: package, module, name (name of the struct, without type parameters)
object_type_package blob,
object_type_module text,
object_type_name text,
object_type_package BLOB,
object_type_module TEXT,
object_type_name TEXT,
-- bcs serialized Object
serialized_object mediumblob NOT NULL,
serialized_object MEDIUMBLOB NOT NULL,
-- Non-null when the object is a coin.
-- e.g. `0x2::sui::SUI`
coin_type text,
coin_type TEXT,
-- Non-null when the object is a coin.
coin_balance bigint,
coin_balance BIGINT,
-- DynamicField/DynamicObject, see types.rs
-- Non-null when the object is a dynamic field
df_kind smallint,
df_kind SMALLINT,
-- bcs serialized DynamicFieldName
-- Non-null when the object is a dynamic field
df_name blob,
df_name BLOB,
-- object_type in DynamicFieldInfo.
df_object_type text,
df_object_type TEXT,
-- object_id in DynamicFieldInfo.
df_object_id blob,
CONSTRAINT objects_pk PRIMARY KEY (object_id(128))
df_object_id BLOB,
CONSTRAINT objects_pk PRIMARY KEY (object_id(128))
);

-- OwnerType: 1: Address, 2: Object, see types.rs
Expand All @@ -47,26 +47,26 @@ CREATE INDEX objects_owner_package_module_name_full_type ON objects (owner_id(12
-- 2. allow null values in some columns for deleted / wrapped objects
-- 3. object_status to mark the status of the object, which is either Active or WrappedOrDeleted
CREATE TABLE objects_history (
object_id blob NOT NULL,
object_version bigint NOT NULL,
object_status smallint NOT NULL,
object_digest blob,
checkpoint_sequence_number bigint NOT NULL,
owner_type smallint,
owner_id blob,
object_type text,
object_id BLOB NOT NULL,
object_version BIGINT NOT NULL,
object_status SMALLINT NOT NULL,
object_digest BLOB,
checkpoint_sequence_number BIGINT NOT NULL,
owner_type SMALLINT,
owner_id BLOB,
object_type TEXT,
-- Components of the StructTag: package, module, name (name of the struct, without type parameters)
object_type_package blob,
object_type_module text,
object_type_name text,
serialized_object mediumblob,
coin_type text,
coin_balance bigint,
df_kind smallint,
df_name blob,
df_object_type text,
df_object_id blob,
CONSTRAINT objects_history_pk PRIMARY KEY (checkpoint_sequence_number, object_id(128), object_version)
object_type_package BLOB,
object_type_module TEXT,
object_type_name TEXT,
serialized_object MEDIUMBLOB,
coin_type TEXT,
coin_balance BIGINT,
df_kind SMALLINT,
df_name BLOB,
df_object_type TEXT,
df_object_id BLOB,
CONSTRAINT objects_history_pk PRIMARY KEY (checkpoint_sequence_number, object_id(128), object_version)
) PARTITION BY RANGE (checkpoint_sequence_number) (
PARTITION objects_history_partition_0 VALUES LESS THAN MAXVALUE
);
Expand All @@ -81,29 +81,28 @@ CREATE INDEX objects_history_owner_package_module_name_full_type ON objects_hist
-- effectively the snapshot of objects at the same checkpoint,
-- except that it also includes deleted or wrapped objects with the corresponding object_status.
CREATE TABLE objects_snapshot (
object_id BLOB NOT NULL,
object_version bigint NOT NULL,
object_status smallint NOT NULL,
object_digest BLOB,
checkpoint_sequence_number bigint NOT NULL,
owner_type smallint,
owner_id BLOB,
object_type text,
object_type_package blob,
object_type_module text,
object_type_name text,
serialized_object mediumblob,
coin_type text,
coin_balance bigint,
df_kind smallint,
df_name BLOB,
df_object_type text,
df_object_id BLOB,
CONSTRAINT objects_snapshot_pk PRIMARY KEY (object_id(128))
object_id BLOB NOT NULL,
object_version BIGINT NOT NULL,
object_status SMALLINT NOT NULL,
object_digest BLOB,
checkpoint_sequence_number BIGINT NOT NULL,
owner_type SMALLINT,
owner_id BLOB,
object_type TEXT,
object_type_package BLOB,
object_type_module TEXT,
object_type_name TEXT,
serialized_object MEDIUMBLOB,
coin_type TEXT,
coin_balance BIGINT,
df_kind SMALLINT,
df_name BLOB,
df_object_type TEXT,
df_object_id BLOB,
CONSTRAINT objects_snapshot_pk PRIMARY KEY (object_id(128))
);
CREATE INDEX objects_snapshot_checkpoint_sequence_number ON objects_snapshot (checkpoint_sequence_number);
CREATE INDEX objects_snapshot_owner ON objects_snapshot (owner_type, owner_id(128), object_id(128));
CREATE INDEX objects_snapshot_coin ON objects_snapshot (owner_id(128), coin_type(128), object_id(128));
CREATE INDEX objects_snapshot_package_module_name_full_type ON objects_snapshot (object_type_package(128), object_type_module(128), object_type_name(128), object_type(128));
CREATE INDEX objects_snapshot_owner_package_module_name_full_type ON objects_snapshot (owner_id(128), object_type_package(128), object_type_module(128), object_type_name(128), object_type(128));

24 changes: 12 additions & 12 deletions crates/sui-indexer/migrations/mysql/2024-03-26-004025_transactions/up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
CREATE TABLE transactions (
tx_sequence_number BIGINT NOT NULL,
transaction_digest BLOB NOT NULL,
tx_sequence_number BIGINT NOT NULL,
transaction_digest BLOB NOT NULL,
-- bcs serialized SenderSignedData bytes
raw_transaction MEDIUMBLOB NOT NULL,
raw_transaction MEDIUMBLOB NOT NULL,
-- bcs serialized TransactionEffects bytes
raw_effects MEDIUMBLOB NOT NULL,
checkpoint_sequence_number BIGINT NOT NULL,
timestamp_ms BIGINT NOT NULL,
raw_effects MEDIUMBLOB NOT NULL,
checkpoint_sequence_number BIGINT NOT NULL,
timestamp_ms BIGINT NOT NULL,
-- array of bcs serialized IndexedObjectChange bytes
object_changes JSON NOT NULL,
object_changes JSON NOT NULL,
-- array of bcs serialized BalanceChange bytes
balance_changes JSON NOT NULL,
balance_changes JSON NOT NULL,
-- array of bcs serialized StoredEvent bytes
events JSON NOT NULL,
events JSON NOT NULL,
-- SystemTransaction/ProgrammableTransaction. See types.rs
transaction_kind smallint NOT NULL,
transaction_kind SMALLINT NOT NULL,
-- number of successful commands in this transaction, bound by number of command
-- in a programmaable transaction.
success_command_count smallint NOT NULL,
CONSTRAINT transactions_pkey PRIMARY KEY (tx_sequence_number, checkpoint_sequence_number)
success_command_count SMALLINT NOT NULL,
CONSTRAINT transactions_pkey PRIMARY KEY (tx_sequence_number, checkpoint_sequence_number)
) PARTITION BY RANGE (checkpoint_sequence_number) (
PARTITION transactions_partition_0 VALUES LESS THAN MAXVALUE
);
Expand Down
16 changes: 8 additions & 8 deletions crates/sui-indexer/migrations/mysql/2024-04-24-180008_checkpoints/up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
CREATE TABLE checkpoints
(
sequence_number bigint PRIMARY KEY,
checkpoint_digest blob NOT NULL,
epoch bigint NOT NULL,
checkpoint_digest BLOB NOT NULL,
epoch BIGINT NOT NULL,
-- total transactions in the network at the end of this checkpoint (including itself)
network_total_transactions bigint NOT NULL,
previous_checkpoint_digest blob,
network_total_transactions BIGINT NOT NULL,
previous_checkpoint_digest BLOB,
-- if this checkpoitn is the last checkpoint of an epoch
end_of_epoch boolean NOT NULL,
end_of_epoch BOOLEAN NOT NULL,
-- array of TranscationDigest in bytes included in this checkpoint
tx_digests JSON NOT NULL,
timestamp_ms BIGINT NOT NULL,
Expand All @@ -17,11 +17,11 @@ CREATE TABLE checkpoints
storage_rebate BIGINT NOT NULL,
non_refundable_storage_fee BIGINT NOT NULL,
-- bcs serialized Vec<CheckpointCommitment> bytes
checkpoint_commitments MEDIUMBLOB NOT NULL,
checkpoint_commitments MEDIUMBLOB NOT NULL,
-- bcs serialized AggregateAuthoritySignature bytes
validator_signature blob NOT NULL,
validator_signature BLOB NOT NULL,
-- bcs serialzied EndOfEpochData bytes, if the checkpoint marks end of an epoch
end_of_epoch_data blob
end_of_epoch_data BLOB
);

CREATE INDEX checkpoints_epoch ON checkpoints (epoch, sequence_number);
Expand Down
2 changes: 1 addition & 1 deletion crates/sui-indexer/migrations/mysql/2024-04-24-180207_epochs/up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ CREATE TABLE epochs
protocol_version BIGINT NOT NULL,
total_stake BIGINT NOT NULL,
storage_fund_balance BIGINT NOT NULL,
system_state MEDIUMBLOB NOT NULL,
system_state MEDIUMBLOB NOT NULL,
-- The following fields are nullable because they are filled in
-- only at the end of an epoch.
epoch_total_transactions BIGINT,
Expand Down
Loading
Loading