/rəˈflek(t)/ - to embody or represent in a faithful or appropriate way
Reflekt helps teams define, govern, and model event data for warehouse-first product analytics. It integrates with Customer Data Platforms, Schema Registries, Data Warehouses, and dbt. Events are defined using JSON schema, version controlled, and updated by pull requests (PRs), enabling:
- Branches for parallel development and testing.
- Reviews and discussion amongst teams and stakeholders.
- CI/CD suites to:
reflekt lintschemas against naming and metadata rules.reflekt pushschemas to deploy them to a schema registry for event data validation.
Stop writing SQL & YAML. reflekt build a dbt package to model, document, and test events in the warehouse.
To learn more about reflekt, checkout:
Reflekt is available on PyPI. Install it with pip or package manager, preferably in a virtual environment:
❯ source /path/to/venv/bin/activate # Activate virtual environment
❯ pip install reflekt # Install Reflekt
❯ reflekt --version # Confirm installation
Reflekt CLI Version: 0.6.0To create a new Reflekt project, make a directory, initialize a Git repo, and run reflekt init.
❯ mkdir ~/Repos/my-reflekt-project # Create a new directory for the project
❯ cd ~/Repos/my-reflekt-project # Navigate to the project directory
❯ git init # Initialize a new Git repo (REQUIRED)
❯ reflekt init # Initialize a new Reflekt project in the current directory
# Follow the prompts to configure the projectYou now have a Reflekt project with the structure:
my-reflekt-project
├── .logs/ # CLI command logs
├── .reflekt_cache/ # Local cache used by reflekt CLI
├── artifacts/ # Data artifacts created by `reflekt build`
├── schemas/ # Event schema definitions
├── .gitignore
├── README.md
└── reflekt_project.yml # Project configurationReflekt uses 3 files to define and configure a Reflekt project.
Defines project settings, event and metadata conventions, data artifact generation, and optional registry config (Avo only).
Example: reflekt_project.yml (click to expand)
# Example reflekt_project.yml
# GENERAL CONFIG ----------------------------------------------------------------------
version: 1.0
name: reflekt_demo # Project name
vendor: com.company_name # Default vendor for schemas in reflekt project
default_profile: dev_reflekt # Default profile to use from reflekt_profiles.yml
profiles_path: ~/.reflekt/reflekt_profiles.yml # Path to reflekt_profiles.yml
# SCHEMAS CONFIG ----------------------------------------------------------------------
schemas: # Define schema conventions
conventions:
event:
casing: title # title | snake | camel | pascal | any
numbers: false # Allow numbers in event names
reserved: [] # Reserved event names
property:
casing: snake # title | snake | camel | pascal | any
numbers: false # Allow numbers in property names
reserved: [] # Reserved property names
data_types: [ # Allowed data types
string, integer, number, boolean, object, array, any, 'null'
]
# REGISTRY CONFIG ---------------------------------------------------------------------
registry: # Additional config for schema registry if needed
avo: # Avo specific config
branches: # Provide ID for Avo branches for `reflekt pull` to work
staging: AbC12dEfG # Safe to version control (See Avo docs to find branch ID: https://bit.ly/avo-docs-branch-id)
main: main # 'main' always refers to the main branch
# ARTIFACTS CONFIG -----------------------------------------------------------------------
artifacts: # Configure how data artifacts are built
dbt: # dbt package config
sources:
prefix: __src_ # Source files will start with this prefix
models:
prefix: stg_ # Model files will start with this prefix
docs:
prefix: _stg_ # Docs files will start with this prefix
in_folder: false # Docs files in separate folder?
tests: # dbt tests to add based on column name (can be empty dict {})
id: [unique, not_null]This file defines connections to schema registries and data warehouse connections.
Example: reflekt_profiles.yml (click to expand)
# Example reflekt_profiles.yml
version: 1.0
dev_reflekt: # Profile name (multiple allowed)
registry: # Schema registry connection details (multiple allowed)
- type: segment
api_token: segment_api_token # https://docs.segmentapis.com/tag/Getting-Started#section/Get-an-API-token
- type: avo
workspace_id: avo_workspace_id # https://www.avo.app/docs/public-api/export-tracking-plan#endpoint
service_account_name: avo_service_account_name # https://www.avo.app/docs/public-api/authentication#creating-service-accounts
service_account_secret: avo_service_account_secret
source: # Data warehouse connection details (multiple allowed)
- id: snowflake # ID must be unique per profile
type: snowflake # Specify details where raw event data is stored
account: abc12345
database: raw
warehouse: transforming
role: transformer
user: reflekt_user # Create reflekt_user with access to raw data (permissions: USAGE, SELECT)
password: reflekt_user_password
- id: redshift # ID must be unique per profile
type: redshift # Specify details where raw event data is stored
host: example-redshift-cluster-1.abc123.us-west-1.redshift.amazonaws.com
database: analytics
port: 5439
user: reflekt_user # Create reflekt_user with access to raw data (permissions: USAGE, SELECT)
password: reflekt_user_password
- id: bigquery # ID must be unique per profile
type: bigquery # Specify details where raw event data is stored
project: raw-data
dataset: jaffle_shop_segment
keyfile_json: # Create a reflekt-user service account with access to BigQuery project where raw event data lands (permissions: BigQuery Data Viewer, BigQuery Job User). Include JSON keyfile fields below.
type: "service_account"
project_id: "foo-bar-123456"
private_key_id: "abc123def456ghi789"
private_key: "-----BEGIN PRIVATE KEY-----\nmy-very-long-private-keyF\n\n-----END PRIVATE KEY-----\n"
client_email: "[email protected]"
client_id: "123456789101112131415161718"
auth_uri: "https://accounts.google.com/o/oauth2/auth"
token_uri: "https://oauth2.googleapis.com/token"
auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/reflekt-user%40foo-bar-123456.iam.gserviceaccount.com"A meta-schema used by reflekt lint to ensure all events in schemas/ follow the Reflekt format. Can also be used to define gloablly required metadata for all event schemas.
Example: schemas/.reflekt/meta/1-0.json (click to expand)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": ".reflekt/meta/1-0.json",
"description": "Meta-schema for all Reflekt events",
"self": {
"vendor": "reflekt",
"name": "meta",
"format": "jsonschema",
"version": "1-0"
},
"type": "object",
"allOf": [
{
"$ref": "http://json-schema.org/draft-07/schema#"
},
{
"properties": {
"self": {
"type": "object",
"properties": {
"vendor": {
"type": "string",
"description": "The company, application, team, or system that authored the schema (e.g., com.company, com.company.android, com.company.marketing)"
},
"name": {
"type": "string",
"description": "The schema name. Describes what the schema is meant to capture (e.g., pageViewed, clickedLink)"
},
"format": {
"type": "string",
"description": "The format of the schema",
"const": "jsonschema"
},
"version": {
"type": "string",
"description": "The schema version, in MODEL-ADDITION format (e.g., 1-0, 1-1, 2-3, etc.)",
"pattern": "^[1-9][0-9]*-(0|[1-9][0-9]*)$"
},
"metadata": { // EXAMPLE: Defining required metadata (code_owner, product_owner)
"type": "object",
"description": "Required metadata for all event schemas",
"properties": {
"code_owner": {"type": "string"},
"product_owner": {"type": "string"}
},
"required": ["code_owner", "product_owner"],
"additionalProperties": false
},
},
"required": ["vendor", "name", "format", "version"],
"additionalProperties": false
},
"properties": {},
"tests": {},
},
"required": ["self", "metadata", "properties"]
}
]
}
Events in a Reflekt project are defined using the JSON schema specification and are stored in the schemas/ directory of the project. Click to expand the Order Completed example below.
Example: my-reflekt-project/schemas/jaffle_shop/Order_Completed/1-0.json (click to expand)
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "jaffle_shop/Order_Completed/1-0.json", // Unique ID for schema (relative to `schemas/` dir)
"description": "User completed an order.", // Event description (REQUIRED)
"self": {
"vendor": "com.thejaffleshop", // Company, application, or system that authored the schema
"name": "Order Completed", // Name of the event
"format": "jsonschema", // Format of the schema
"version": "1-0", // Version of the schema
"metadata": { // Metadata for the event
"code_owner": "@the-jaffle-shop/frontend-guild",
"product_owner": "[email protected]",
}
},
"type": "object",
"properties": { // Event properties (REQUIRED, but can be empty)
"coupon": {
"description": "Coupon code used for the order.", // Property description (REQUIRED)
"type": [
"string",
"null" // Allow null values
]
},
"currency": {
"description": "Currency for the order.",
"type": "string"
},
"discount": {
"description": "Total discount for the order.",
"type": "number"
},
"order_id": {
"description": "Unique identifier for the order.",
"type": "string"
},
"products": {
"description": "List of products in the cart.",
"type": "array", // Array type
"items": { // Items in the array
"type": "object",
"properties": { // Properties of the items
"category": {
"description": "Category of the product.",
"type": "string"
},
"name": {
"description": "Name of the product.",
"type": "string"
},
"price": {
"description": "Price of the product.",
"type": "number"
},
"product_id": {
"description": "Unique identifier for the product.",
"type": "string"
},
"quantity": {
"description": "Quantity of the product in the cart.",
"type": "integer"
},
"sku": {
"description": "Stock keeping unit for the product.",
"type": "string"
}
},
"required": [ // Required properties for the items
"product_id",
"sku",
"category",
"name",
"price",
"quantity"
],
"additionalProperties": false, // Are additional item properties allowed?
}
},
"revenue": {
"description": "Total revenue for the order.",
"type": "number"
},
"session_id": {
"description": "Unique identifier for the session.",
"type": "string"
},
"shipping": {
"description": "Shipping cost for the order.",
"type": "number"
},
"subtotal": {
"description": "Subtotal for the order (revenue - discount).",
"type": "number"
},
"tax": {
"description": "Tax for the order.",
"type": "number"
},
"total": {
"description": "Total cost for the order (revenue - discount + shipping + tax = subtotal + shipping + tax).",
"type": "number"
}
},
"required": [ // Required properties (can be empty)
"session_id",
"order_id",
"revenue",
"coupon",
"discount",
"subtotal",
"shipping",
"tax",
"total",
"currency",
"products"
],
"additionalProperties": false // Are additional properties allowed?
}Schemas in a Reflekt project are identified and --selected by their $id, which is their path relative to the schemas/ directory. For example:
| File Path to Schema | Schema $id |
|---|---|
~/repos/my-reflekt-project/schemas/jaffle_shop/Cart_Viewed/1-0.json |
jaffle_shop/Cart_Viewed/1-0.json |
~/repos/my-reflekt-project/schemas/jaffle_shop/Cart_Viewed/2-1.json |
jaffle_shop/Cart_Viewed/2-1.json |
Each schema has a version (e.g., 1-0, 2-1), used to indicate changes to data collection requirements. New event schemas start at 1-0 and follow a MAJOR-MINOR version spec, as shown in the table below.
| Type | Description | Example | Use Case |
|---|---|---|---|
| MAJOR | Breaking change incompatible with previous data. | 1-0, 2-0(ends in `-0) |
- Add/remove/rename a required property - Change a property from optional to required - Change a property's type |
| MINOR | Non-breaking change compatible with previous data. | 1-1, 2-3 |
- Add/remove/rename an optional property - Change a property from required to optional |
Note
For MINOR schema versions (non-breaking changes), you can either:
- Update the existing schema and increment the MINOR version number.
- Create a new
.jsonfile with the updated schema and increment the MINOR version number.
For MAJOR schema versions (breaking changes), you MUST:
- Create a new
.jsonfile with the updated schema. This way, an application/product/feature can begin using the new schema while others continue to use the old schema (migrating later).
Schemas can be linted to test if they follow the naming conventions in your [reflekt_project.yml] and metadata conventions in schemas/.reflekt/meta/1-0.json.
❯ reflekt lint --select schemas/jaffle_shop
[18:31:12] INFO Running with reflekt=0.5.0
[18:31:12] INFO Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop
[18:31:12] INFO Found 9 schema(s) to lint
[18:31:12] INFO 1 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json
[18:31:19] INFO 2 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json
[18:31:20] INFO 3 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json
[18:31:24] INFO 4 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json
[18:31:26] INFO 5 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json
[18:31:32] INFO 6 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json
[18:31:37] INFO 7 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json
[18:31:40] INFO 8 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json
[18:31:44] INFO 9 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json
[18:31:51] INFO Completed successfullyIn order to validate events as they flow from Application -> Registry -> Customer Data Platform (CDP) -> Data Warehouse, we need to send a copy of our event schemas to a schema registry (see supported registries). This is done with the reflekt push command.
❯ reflekt push --registry segment --select schemas/jaffle_shop
[18:41:05] INFO Running with reflekt=0.5.0
[18:41:06] INFO Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop
[18:41:06] INFO Found 9 schemas to push
[18:41:06] INFO 1 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json
[18:41:06] INFO 2 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json
[18:41:06] INFO 3 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json
[18:41:06] INFO 4 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json
[18:41:06] INFO 5 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json
[18:41:06] INFO 6 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json
[18:41:06] INFO 7 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json
[18:41:06] INFO 8 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json
[18:41:06] INFO 9 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json
[18:41:08] INFO Completed successfullyModeling event data in dbt is a lot of work. Everyone wants staging models that are clean, documented, and tested. But who wants to write and maintain SQL and YAML for hundreds of events?
You don't have to choose. Put reflekt build to work for you - staging models, documentation, even tests - all in a dbt package ready for you to use in your dbt project.
❯ reflekt build --artifact dbt --select schemas/jaffle_shop --source snowflake.raw.jaffle_shop_segment --sdk segment
[18:56:25] INFO Running with reflekt=0.5.0
[18:56:26] INFO Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop
[18:56:26] INFO Found 9 schemas to build
[18:56:27] INFO Building dbt package:
name: jaffle_shop
dir: /Users/gclunies/Repos/reflekt/artifacts/dbt/jaffle_shop
--select: jaffle_shop
--sdk_arg: segment
--source: snowflake.raw.jaffle_shop_segment
[18:56:27] INFO Building dbt source 'jaffle_shop_segment'
[18:56:27] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json
[18:56:28] INFO Building dbt table 'order_completed' in source 'jaffle_shop_segment'
[18:56:28] INFO Building staging model 'stg_jaffle_shop_segment__order_completed.sql'
[18:56:28] INFO Building dbt documentation '_stg_jaffle_shop_segment__order_completed.yml'
[18:56:28] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json
[18:56:29] INFO Building dbt table 'identifies' in source 'jaffle_shop_segment'
[18:56:29] INFO Building staging model 'stg_jaffle_shop_segment__identifies.sql'
[18:56:29] INFO Building dbt documentation '_stg_jaffle_shop_segment__identifies.yml'
[18:56:29] INFO Building dbt artifacts for schema: Segment 'users' table
[18:56:29] INFO Building dbt table 'users' in source 'jaffle_shop_segment'
[18:56:29] INFO Building staging model 'stg_jaffle_shop_segment__users.sql'
[18:56:29] INFO Building dbt documentation '_stg_jaffle_shop_segment__users.yml'
[18:56:29] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json
[18:56:29] INFO Building dbt table 'product_clicked' in source 'jaffle_shop_segment'
[18:56:29] INFO Building staging model 'stg_jaffle_shop_segment__product_clicked.sql'
[18:56:29] INFO Building dbt documentation '_stg_jaffle_shop_segment__product_clicked.yml'
[18:56:29] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json
[18:56:29] INFO Building dbt table 'cart_viewed' in source 'jaffle_shop_segment'
[18:56:29] INFO Building staging model 'stg_jaffle_shop_segment__cart_viewed.sql'
[18:56:29] INFO Building dbt documentation '_stg_jaffle_shop_segment__cart_viewed.yml'
[18:56:29] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json
[18:56:30] INFO Building dbt table 'product_removed' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__product_removed.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__product_removed.yml'
[18:56:30] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json
[18:56:30] INFO Building dbt table 'product_added' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__product_added.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__product_added.yml'
[18:56:30] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json
[18:56:30] INFO Building dbt table 'checkout_step_viewed' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__checkout_step_viewed.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__checkout_step_viewed.yml'
[18:56:30] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json
[18:56:30] INFO Building dbt table 'checkout_step_completed' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__checkout_step_completed.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__checkout_step_completed.yml'
[18:56:30] INFO Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json
[18:56:30] INFO Building dbt table 'pages' in source 'jaffle_shop_segment'
[18:56:30] INFO Building staging model 'stg_jaffle_shop_segment__pages.sql'
[18:56:30] INFO Building dbt documentation '_stg_jaffle_shop_segment__pages.yml'
[18:56:30] INFO Building dbt artifacts for schema: Segment 'tracks' table
[18:56:31] INFO Building dbt table 'tracks' in source 'jaffle_shop_segment'
[18:56:31] INFO Building staging model 'stg_jaffle_shop_segment__tracks.sql'
[18:56:31] INFO Building dbt documentation '_stg_jaffle_shop_segment__tracks.yml'
[18:56:31] INFO Copying dbt package from temporary path /Users/gclunies/Repos/reflekt/.reflekt_cache/artifacts/dbt/jaffle_shop to /Users/gclunies/Repos/reflekt/artifacts/dbt/jaffle_shop
[18:56:31] INFO Successfully built dbt packageA description of commands can be seen by running reflekt --help. The help page for each CLI command is shown below.
❯ reflekt init --help
[11:17:16] INFO Running with reflekt=0.6.0
Usage: reflekt init [OPTIONS]
Initialize a Reflekt project.
╭─ Options ──────────────────────────────────────────────────────╮
│ --dir TEXT [default: .] │
│ --verbose --no-verbose [default: no-verbose] │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────╯❯ reflekt debug --help
[11:18:07] INFO Running with reflekt=0.6.0
Usage: reflekt debug [OPTIONS]
Check Reflekt project configuration.
╭─ Options ───────────────────────────────────╮
│ --help Show this message and exit. │
╰─────────────────────────────────────────────╯❯ reflekt lint --help
[11:20:29] INFO Running with reflekt=0.6.0
Usage: reflekt lint [OPTIONS]
Lint schema(s) to test for naming and metadata conventions.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * --select -s TEXT Schema(s) to lint. Starting with 'schemas/' is optional. [default: None] [required] │
│ --verbose -v Verbose logging. │
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯❯ reflekt push --help
[11:22:37] INFO Running with reflekt=0.6.0
Usage: reflekt push [OPTIONS]
Push schema(s) to a schema registry.
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * --registry -r [avo|segment] Schema registry to push to. [default: None] [required] │
│ * --select -s TEXT Schema(s) to push to schema registry. Starting with 'schemas/' is optional. [default: None] │
│ --delete -D Delete schema(s) from schema registry. Prompts for confirmation │
│ --force -F Force command to run without confirmation. │
│ --profile -p TEXT Profile in reflekt_profiles.yml to use for schema registry connection. │
│ --verbose -v Verbose logging. │
│ --help Show this message and exit. │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯❯ reflekt pull --help
[11:25:18] INFO Running with reflekt=0.6.0
Usage: reflekt pull [OPTIONS]
Pull schema(s) from a schema registry.
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * --registry -r [avo|segment] Schema registry to pull from. [default: None] [required] │
│ * --select -s TEXT Schema(s) to pull from schema registry. If registry uses tracking plans, starting with the plan name. [default: None] [required] │
│ --profile -p TEXT Profile in reflekt_profiles.yml to use for schema registry connection. │
│ --verbose -v Verbose logging. │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯❯ reflekt build --help
[11:31:36] INFO Running with reflekt=0.6.0
Usage: reflekt build [OPTIONS]
Build data artifacts based on schemas.
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * --artifact -a [dbt] Type of data artifact to build. [default: None] [required] │
│ * --select -s TEXT Schema(s) to build data artifacts for. Starting with 'schemas/' is optional. [default: None] [required] │
│ * --sdk [segment] The SDK used to collect the event data. [default: None] [required] │
│ * --source TEXT The <source_id>.<database>.<schema> storing raw event data. <source_id> must be a data warehouse source defined in reflekt_profiles.yml [default: None] [required] │
│ --profile -p TEXT Profile in reflekt_profiles.yml to look for the data source specified by the --source option. Defaults to default_profile in reflekt_project.yml │
│ --verbose -v Verbose logging. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯❯ reflekt report --help
[08:20:09] INFO Running with reflekt=0.6.0
Usage: reflekt report [OPTIONS]
Generate Markdown report(s) for schema(s).
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────╮
│ * --select -s TEXT Schema(s) to generate Markdown report(s) for. Starting with │
│ 'schemas/' is optional. │
│ [default: None] │
│ [required] │
│ --to-file -f Write report(s) to file instead of terminal. │
│ --verbose -v Verbose logging. │
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯Reflekt understands how Customer Data Platforms (CDPs) collect event data and load them into data warehouses, allowing it to:
- Parse the schemas in a Reflekt project.
- Find matching tables for events and columns for properties in the data warehouse.
- Build a
dbtpackage with sources, staging models, and documentation for event data.
| CDP | Supported |
|---|---|
| Segment | ✅ |
| Rudderstack | 🚧 Coming Soon 🚧 |
| Amplitude | 🚧 Coming Soon 🚧 |
Schema registries store and serve schemas. When a schema is registered in a regsitry, it can be used to validate events as they flow through your data collection infrastructure. Reflekt works with schema registries from CDPs, SaaS vendors, and open-source projects, letting teams to decide between managed and self-hosted solutions.
| Registry | Open Source | Schema Versions | Recommended Workflow |
|---|---|---|---|
| Segment Protocols | ❌ | MAJOR only |
Manage schemas in Reflekt.reflekt push to Protocols for event validation.reflekt build --artifact dbt to build dbt package. |
| Avo | ❌ | MAJOR only |
Manage schemas in Avo.reflekt pull to get schemas.reflekt build --artifact dbt to build dbt package. |
| reflekt-registry 🚧 Coming Soon 🚧 |
✅ | MAJOR & MINOR |
Manage schemas in Reflekt.reflekt push to reflekt-registry.reflekt build --artifact dbt to build dbt package. |
In order to build dbt packages, Reflekt needs to connect to a cloud data warehouse where raw event data is stored.
| Data Warehouse | Supported |
|---|---|
| Snowflake | ✅ |
| Redshift | ✅ |
| BigQuery | ✅ |
Reflekt NEVER copies, moves, or modifies events in the data warehouse. It ONLY reads table and column names for templating.
dbt enables anyone that knows SQL to transform data in a cloud data warehouse. When modeling in dbt, it is best practice to:
- Define sources pointing to the raw data.
- Write staging models that rename, recast, or usefully reconsider columns into a consistent format.
- Document and test the staging models.
But as the number of events intsrumented in your product grow or change as the product evolves, mainting this best practice can be burdensome and boring. This is where reflekt build steps in.
- Source Code: github.com/GClunies/reflekt
- Issue Tracker: github.com/GClunies/reflekt/issues
- Pull Requests: github.com/GClunies/reflekt/pulls
This project is licensed under the Apache-2.0 License.