Streamline app schema creation process

[deontologician]

Right now you need to run set-schema before running hz serve without --dev, or else you need to run hz serve --auto-create-collection and query each table you want to set up the database. This isn't great, and makes non-dev mode really annoying.

I was talking with @Tryneus and there are a couple options we thought of:

Proposal A

Have hz init take a connection option, and set up the database with the default users groups and rules that it does now at the time it sets up the app directory

Proposal B

• Have hz init create a schema.toml file somewhere in the directory (maybe inside .hz/) with some simple default groups (and the admin group carte blanche rule).
• Add an option to hz serve to read and load this schema when it starts up, before starting to serve (this option would probably be part of --dev)

ping @mlucy @danielmewes @coffeemug @dalanmiller @encryptio

[mlucy]

This would be really nice actually; currently when you deploy with Horizon cloud it pushes .hz/schema.toml as your schema. If the Horizon server also read that when starting, we would avoid a lot of nasty cases where people are developing with one schema, forget to export it, and end up pushing an old schema and breaking their app.

[mglukhovsky]

This needs a different name than "app schema", but this seems really useful for easy and predictable deployment.

[dalanmiller]

Add an option to hz serve to read and load this schema when it starts up, before starting to serve (this option would probably be part of --dev)

👍

Now that I finally understand the consequences of this design decision, I think that one should be able to provide a schema.toml to hz serve which will then setup the proper schemas, indexes, and permissions, before starting to serve the app. I'm not sure I can foresee any alternative when using something like Docker Compose to deploy where the database instance may not exist until you actually start your app.

Would it be feasible (as you've suggested) to create .hz/schema.toml on hz init and then also update that file as the schema changes during --dev prototyping? So it's basically seamless when a developed would go to deploy via hz deploy?

[deontologician]

Would it be feasible [...] to [...] update that file as the schema changes during --dev prototyping?

It's not a super great idea since we can't just modify a toml file and write it back without clobbering comments. If we do #543 this would be an option

[mlucy]

So, Chris and I just ran through the "embedding Horizon" flow on the website. It's extremely non-obvious what to do. What happened was Horizon connected to RethinkDB and started spamming"error: Connection to RethinkDB terminated: Error: The databases horizon and horizon_internal do not exist.Run hz set-schema to initialize the database, then start the Horizon server." over and over. I ran hz set-schema and it complained about not receiving enough arguments. Then I ran echo | hz set-schema -, which worked but did not actually set the schema because of #547 . I had to run echo | hz set-schema -n horizon - (which I only knew to do because I could decode the "horizon and horizon_internal do not exist" error messages to "the project was named 'horizon'").

This flow is really bad. I'm not sure what the best way to fix it is, but it would be nice if Horizon set an empty schema automatically, or if not having a schema was a warning or something instead.

[dalanmiller]

There's a couple items out of this I propose:

• .hz/schema.toml needs to be a default file from hz init
• .hz/schema.toml should contain the bare minimum for horizon to operate (at the moment horizon and horizon_internal and the project name from .hz/schema.toml)
  • Else conditions to this from @mlucy in Proposal: change behavior of set-schema and get-schema to be more intuitive #547:
  • Otherwise, if there is only one Horizon project in the database (which will be true 99% of the time), use that project.
  • Otherwise, error out and tell people to specify which project to use. (I think looking at the CWD to get the project name makes a lot of sense for hz init ., but not much sense at all for set-schema and get-schema.)
• hz serve needs a flag to accept a schema.toml to load and perform the migrations prior to starting Horizon. This is imperative for any Docker setups.
• hz get-schema
  • will be renamed to hz schema save
  • will automatically save to and overwrite .hz/schema.toml but is configurable with a positional argument
• hz set-schema
  • will be renamed to hz schema load
  • will automatically retrieve from .hz/schema.toml but is configurable with a positional argument

Renaming is important as get- and set- I think are a bit ambiguous, as well, this opens up the CLI down the road for hz schema remote or hz schema cloud.

[mlucy]

+1 for the change to hz schema save and hz schema load defaulting to .hz/schema.toml.

I also think that by default hz serve should look for .hz/schema.toml if there's no schema set, rather than the user having to specify that they want to load .hz/schema.toml the first time they start the server.

[deontologician]

this looks good, I think we all intuitively want the same thing

[dalanmiller]

I also think that by default hz serve should look for .hz/schema.toml if there's no schema set, rather than the user having to specify that they want to load .hz/schema.toml the first time they start the server.

What would be the correct way for hz serve to determine that no schema is set versus schema is just out of data with .hz/schema.toml? And should this always happen or only in --dev mode?

[deontologician]

I think only in dev mode.

It seems like we should either

1. Only set-schema if we detect there is no horizon_internal database (so on first hz serve)
2. set-schema every time with --update.