Make SQL Migrations accessible as a method #57
Labels
enhancement
New feature or request
Comments
|
I had a feeling this would come up quickly 😅 Is your thought that we could export the migrations as SQL string constants? The issue I'm seeing with that is that Goose Go migrations appear to only work with individual SQL statements, whereas River's migrations are comprised of multiple statements. Or maybe I'm misreading that? |
|
Another option is to expose this as a Go API which takes a pgx pool or driver wrapper and which you can then call something like |
|
You can use sql files with use goose and separate the up/down migrations. -- +goose Up
-- +goose StatementBegin
STATEMENT_1;
STATEMENT_2;
STATEMENT_N;
-- +goose StatementEnd
-- +goose Down
-- +goose StatementBegin
DOWN_STATEMENT_1;
DOWN_STATEMENT_2;
DOWN_STATEMENT_N;
-- +goose StatementEndEmbed the whole directory with all sql migration files: //go:embed sql/migrations/*.sql
var EmbedMigrations embed.FSRun the migrations: var sqlDB *sql.DB
goose.SetBaseFS(EmbedMigrations)
goose.SetDialect("postgres")
goose.Up(db, "sql/migrations")But to answer your initial question: I think you could pass whole SQL files with multiples statements to |
This is a great idea. Maybe is would also be possible to just run a method like |
|
I was looking into This project looks AWESOME! Really well done. Pretty much found myself here for the same reason, where I wouldn't be using the CLI and instead prefer to call a method or function at application startup. I copied over the sql statements verbatim from the second migration, and let goose handle the river migrations, but it'd be ideal if this project exposed said functionality so it could be invoked in the same place as my other migrations. Also, if you do add a Migration 2 (SQL)-- +goose Up
CREATE TYPE river_job_state AS ENUM(
'available',
'cancelled',
'completed',
'discarded',
'retryable',
'running',
'scheduled'
);
CREATE TABLE river_job(
-- 8 bytes
id bigserial PRIMARY KEY,
-- 8 bytes (4 bytes + 2 bytes + 2 bytes)
--
-- `state` is kept near the top of the table for operator convenience -- when
-- looking at jobs with `SELECT *` it'll appear first after ID. The other two
-- fields aren't as important but are kept adjacent to `state` for alignment
-- to get an 8-byte block.
state river_job_state NOT NULL DEFAULT 'available' ::river_job_state,
attempt smallint NOT NULL DEFAULT 0,
max_attempts smallint NOT NULL,
-- 8 bytes each (no alignment needed)
attempted_at timestamptz,
created_at timestamptz NOT NULL DEFAULT NOW(),
finalized_at timestamptz,
scheduled_at timestamptz NOT NULL DEFAULT NOW(),
-- 2 bytes (some wasted padding probably)
priority smallint NOT NULL DEFAULT 1,
-- types stored out-of-band
args jsonb,
attempted_by text[],
errors jsonb[],
kind text NOT NULL,
metadata jsonb NOT NULL DEFAULT '{}' ::jsonb,
queue text NOT NULL DEFAULT 'default' ::text,
tags varchar(255)[],
CONSTRAINT finalized_or_finalized_at_null CHECK ((state IN ('cancelled', 'completed', 'discarded') AND finalized_at IS NOT NULL) OR finalized_at IS NULL),
CONSTRAINT max_attempts_is_positive CHECK (max_attempts > 0),
CONSTRAINT priority_in_range CHECK (priority >= 1 AND priority <= 4),
CONSTRAINT queue_length CHECK (char_length(queue) > 0 AND char_length(queue) < 128),
CONSTRAINT kind_length CHECK (char_length(kind) > 0 AND char_length(kind) < 128)
);
-- We may want to consider adding another property here after `kind` if it seems
-- like it'd be useful for something.
CREATE INDEX river_job_kind ON river_job USING btree(kind);
CREATE INDEX river_job_state_and_finalized_at_index ON river_job USING btree(state, finalized_at) WHERE finalized_at IS NOT NULL;
CREATE INDEX river_job_prioritized_fetching_index ON river_job USING btree(state, queue, priority, scheduled_at, id);
CREATE INDEX river_job_args_index ON river_job USING GIN(args);
CREATE INDEX river_job_metadata_index ON river_job USING GIN(metadata);
-- +goose StatementBegin
CREATE OR REPLACE FUNCTION river_job_notify()
RETURNS TRIGGER
AS $$
DECLARE
payload json;
BEGIN
IF NEW.state = 'available' THEN
-- Notify will coalesce duplicate notificiations within a transaction, so
-- keep these payloads generalized:
payload = json_build_object('queue', NEW.queue);
PERFORM
pg_notify('river_insert', payload::text);
END IF;
RETURN NULL;
END;
$$
LANGUAGE plpgsql;
-- +goose StatementEnd
CREATE TRIGGER river_notify
AFTER INSERT ON river_job
FOR EACH ROW
EXECUTE PROCEDURE river_job_notify();
CREATE UNLOGGED TABLE river_leader(
-- 8 bytes each (no alignment needed)
elected_at timestamptz NOT NULL,
expires_at timestamptz NOT NULL,
-- types stored out-of-band
leader_id text NOT NULL,
name text PRIMARY KEY,
CONSTRAINT name_length CHECK (char_length(name) > 0 AND char_length(name) < 128),
CONSTRAINT leader_id_length CHECK (char_length(leader_id) > 0 AND char_length(leader_id) < 128)
);
-- +goose Down
DROP TABLE river_job;
DROP FUNCTION river_job_notify;
DROP TYPE river_job_state;
DROP TABLE river_leader; |
|
Hey, did you try ariga/atlas for versioned migration? https://atlasgo.io/versioned/intro |
Hah, thanks Michael! And nice to see you here. I opened #67 which exposes a new That won't be quite enough to get Goose working though because Goose exposes an
Yep, or at least insofar as its safe to run be modifying database DDL concurrently. #67 should already have that covered, and exposes a |
Nice! Thanks for putting this together.
It's a nice property that
Nvm, I see ya'll already covered this in the Limited driver support documentation. |
|
No worries. I'm not actually against putting in a driver for core's |
brandur
added a commit
that referenced
this issue
Dec 3, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 3, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 3, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 3, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
|
We released River 0.0.12 that adds a new Goose is still not supported because of the |
brandur
added a commit
that referenced
this issue
Dec 7, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 7, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 7, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 9, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 9, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 9, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 9, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 10, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 13, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 13, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 13, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. Notably, this development branch has to use some `go.mod` `replace` directives to demonstrate that it works correctly. If we go this direction, we'll need to break it into chunks to release it without them: 1. Break out changes to `river/riverdriver`. Tag and release it. 2. Break out changes to `riverdriver/river*` drivers. Have them target the release in (1), comment out `replace`s, then tag and release them. 3. Target the remaining River changes to the releases in (1) and (2), comment out `replace`s, then tag and release the top-level driver. Unfortunately future deep incisions to drivers will require similar gymnastics, but I don't think there's a way around it (we already have this process except it's currently two steps instead of three). The hope is that these will change relatively rarely, so it won't be too painful. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 13, 2023
… River Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
brandur
added a commit
that referenced
this issue
Dec 13, 2023
… River (#98) Here, add a new minimal driver called `riverdriver/riversql` that supports Go's built-in `database/sql` package, but only for purposes of migrations. The idea here is to fully complete #57 by providing a way of making `rivermigrate` interoperable with Go migration frameworks that support Go-based migrations like Goose, which provides hooks for `*sql.Tx` [1] rather than pgx. `riverdriver/riversql` is not a full driver and is only meant to be used with `rivermigrate`. We document this clearly in a number of places. To make a multi-driver world possible with River, we have to start the work of building a platform that does more than `riverpgxv5`'s "cheat" workaround. This works by having each driver implement specific database operations like `MigrationGetAll`, which target their wrapped database package of choice. This is accomplished by having each driver bundle in its own sqlc that targets its package. So `riverpgxv5` has an `sqlc.yaml` that targets `pgx/v5`, while `riversql` has one that targets `database/sql`. There's some `sqlc.yaml` duplication involved here, but luckily both drivers can share a `river_migration.sql` file that contains all queries involved, so you only need to change one place. `river_migration.sql` also migrates entirely out of the main `./internal/dbsqlc`. The idea here is that eventually `./internal/dbsqlc` will disappear completely, usurped entirely by driver-specific versions. As this is done, all references to `pgx` will disappear from the top-level package. There are some complications here to figure out like `LISTEN`/`NOTIFY` though, and I'm not clear whether `database/sql` could ever become a fully functional driver as it might be missing some needed functionality (e.g. subtransactions are still not supported after talking about them for ten f*ing years [2]. However, even if it's not, the system would let us support other fully functional packages or future major versions of pgx (or even past ones like `pgx/v4` if there's demand). `river/riverdriver` becomes a package as it now has types in it that need to be referenced by driver implementations, and this would otherwise not be possible without introducing a circular dependency. [1] https://github.com/pressly/goose#go-migrations [2] golang/go#7898
|
Alright, we just added a https://riverqueue.com/docs/migrations#migrating-river-with-goose Going to close this issue out as done, but let me know if there are any other limitations that people are finding. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thank you for publishing such a great library. The library fits exactly my needs for private project I would to like to open source once its reaches stability.
Currently I am doing database migrations with https://github.com/pressly/goose on the startup of the app. This makes it easy to deploy on hosters like heroku or fly.io.
Would it be possible to make the functionality of 'river migrate-up' accessible as method? I know this could be a conflict for the leader election if every instance tries to do the migrations on startup but it should be no problem on a single instance.
If you are open to it, I could try to create a pull request for this changes.
The text was updated successfully, but these errors were encountered: