doc(schema): Export canonical event schema as JSON Schema (#558)

This introduces schemars as dependency and adds a new subcommand to relay that generates the schema, which we then use to publish the schema as release artifact.

Alex Hofsteede attempted to document the event schema using JSON schema too, but sort of got stuck halfway through. I don't know if it was lack of time or something else. I believe the difference is that we only attempt to document the canonical version after store normalization while Alex tried to validate every legacy format as well. The only exception we make there is that tags/headers/cookies are also allowed as objects even though the canonical form is array of arrays.


This schema file's target audience is mostly SDK developers, but maybe Snuba ppl can profit from it too.

Note: We cannot profit from this within the Python SDK as the only tooling for using JSON schema in type annotations is https://pypi.org/project/jsonschema-typed which users would have to install in their devenv to see any effect. What I actually wanted was code generation from JSON schema, not a runtime plugin for the type checker. I think for Snuba it would be fine but it's still inconvenient to set up and the error messages will be very poor for sure.

I am fairly certain that this will be a good data source for auto-generated documentation at the very least. We should merge this before it gets out of sync with master even more, later we can make adjustments to make the autogenerated docs better.

.gitignore

@@ -16,3 +16,7 @@ target
 # Editors
 /.idea/
 .DS_Store
+
+# NPM utilities for building docs
+/node_modules/
+package-lock.json

Cargo.toml

@@ -26,6 +26,7 @@ exclude = [
 default = ["ssl"]
 ssl = ["relay-server/ssl"]
 processing = ["relay-server/processing"]
+jsonschema = ["relay-general/jsonschema"]
 
 [profile.release]
 debug = true
@@ -46,6 +47,7 @@ relay-config = { path = "relay-config" }
 relay-server = { path = "relay-server" }
 relay-general = { path = "relay-general" }
 sentry = { version = "0.18.0", features = ["with_debug_meta"] }
+schemars = { version = "0.7.1", features = ["uuid", "chrono"] }
 serde = "1.0.114"
 serde_json = "1.0.55"
 hostname = "0.3.1"

Makefile

@@ -91,22 +91,35 @@ api-docs: setup-git
 	@cargo doc
 .PHONY: api-docs
 
-prose-docs: .venv/bin/python extract-doc
+prose-docs: .venv/bin/python extract-metric-docs extract-jsonschema-docs
 	.venv/bin/mkdocs build
 	touch site/.nojekyll
 .PHONY: prose-docs
 
-extract-doc: .venv/bin/python
+extract-metric-docs: .venv/bin/python
 	.venv/bin/pip install -U -r requirements-doc.txt
 	cd scripts && ../.venv/bin/python extract_metric_docs.py
 
+extract-jsonschema-docs: install-jsonschema-docs
+	rm -rf docs/event-schema/event.schema.*
+	set -e && cargo run --features jsonschema -- event-json-schema \
+		> docs/event-schema/event.schema.json
+	set -e && ./node_modules/.bin/quicktype-markdown \
+		Event docs/event-schema/event.schema.json \
+		> docs/event-schema/event.schema.md
+
+install-jsonschema-docs:
+	npm install git+https://github.com/untitaker/quicktype-markdown
+
 docserver: prose-docs
 	.venv/bin/mkdocs serve
 .PHONY: docserver
 
 travis-upload-prose-docs: prose-docs
 	cd site && zip -r gh-pages .
-	zeus upload -t "application/zip+docs" site/gh-pages.zip \
+	set -e && zeus upload -t "application/zip+docs" site/gh-pages.zip \
+		|| [[ ! "$(TRAVIS_BRANCH)" =~ ^release/ ]]
+	set -e && zeus upload -t "application/octet-stream" -n event.schema.json docs/event-schema/event.schema.json \
 		|| [[ ! "$(TRAVIS_BRANCH)" =~ ^release/ ]]
 .PHONY: travis-upload-docs
 

docs/.gitignore

@@ -1 +1,2 @@
+event-schema/event.schema.*
 configuration/metrics.md

docs/event-schema/index.md

@@ -0,0 +1,14 @@
+# Event schema
+
+This page is intended to eventually replace [our current event schema documentation]. As opposed to the current one, this one is automatically generated from source code and therefore more likely to be up-to-date and exhaustive. The plan is to eventually make this document the source of truth, i.e. move it into `develop.sentry.dev`.
+
+**It is still a work-in-progress.** Right now we recommend using our existing docs as linked above and only fall back to this doc to resolve ambiguities.
+
+In addition to documentation the event schema is documented in machine-readable form:
+
+- [Download JSON schema](event.schema.json) (which is what this document is generated from)
+
+{% include "event-schema/event.schema.md" %}
+
+
+[our current event schema documentation]: https://develop.sentry.dev/sdk/event-payloads/

mkdocs.yml

@@ -25,6 +25,8 @@ nav:
       - architecture/ingest-event-path.md
       - architecture/project-configuration.md
 
+  - "": event-schema/index.md
+
 markdown_extensions:
   - admonition: {}
   - tables: {}
@@ -37,6 +39,15 @@ markdown_extensions:
           class: mermaid
           format: !!python/name:pymdownx.superfences.fence_div_format
 
+plugins:
+  - search
+  # Used in event schema docs
+  - macros
+
+  - exclude:
+      glob:
+        - event-schema/event.schema.md
+
 extra_css:
   # Config for diagrams
   - https://unpkg.com/[email protected]/dist/mermaid.css