AutoMigrate for Postgres dialect #926
Conversation
|
I had to update a lot of to the existing unit tests (~5-7 files) because I noticed that many of them weren't properly cleaning up the database, causing side-effects in the auto-migrate tests. UPD: opened #927 for this specific change. Should make it easier to review the current one once the other changes are merged. |
bevzzz
changed the title
bevzzz
force-pushed
the
feature/automigrate-rename-table
branch
5 times, most recently
from
3ef6553
to
ce75416
Compare
bevzzz
force-pushed
the
feature/automigrate-rename-table
branch
3 times, most recently
from
eeadd41
to
9faa55b
Compare
bevzzz
force-pushed
the
feature/automigrate-rename-table
branch
from
85d0877
to
e7e7c4f
Compare
…ce#1028) * feat(schema): add support type for net/netip.Addr and net/netip.Prefix * fix(schema): net.IPNet(not ptr) is not implement fmt.Stringer Edit: updated commit message to comply with commitlint [subject-case] rule. Original subject: "Add support type..."
- RawQuery - CreateTableQuery
Additionally: - allow custom FK names (limited applicability rn) - implement GetReverse for AddFK and DropFK - detect renamed foreign keys - EqualSignature handles empty models (no columns)
This is a WIP commit.
- Do not implement AppendQuery on Operation level. This is a leaky abstraction as queries are dialect-specific and migrate package should not be concerned with how they are constructed. - AlterTableQuery also is an unnecessary abstraction. Now pgdialect will just build a simple string-query for each Operation. - Moved operations.go to migrate/ package and deleted alt/ package. - Minor clean-ups and documentation. testChangeColumnType is commented out because the implementation is missing.
Bufixes and improvements: - pgdialect.Inspector canonicalizes all default expressions (lowercase) to make sure they are always comparable with the model definition. - sqlschema.SchemaInspector canonicalizes all default expressions (lowercase) - pgdialect and sqlschema now support type-equivalence, which prevents unnecessary migrations like CHAR -> CHARACTER from being created. Changing PRIMARY KEY and UNIQUE-ness are outside of this commit's scope, because those constraints can span multiple columns.
- Add -Op suffix to all operations - Replace Apply() method with AppendSQL() in Migrator interface - Add snapshot tests for SQL generation - Suspended support for renaming constraints Included bugfixes: - changeColumnType uses "type equivalence" to update varchar length if needed - addColumn supports DEFAULT clause
- CreateSQLMigrations() writes up- and down- SQL to migration files in the migrations directory - Migrate() generates migration files and applies them, creating a corresponding entry in the database - Run() is deprecated. It was decided that AutoMigrator will not support in-place migrations for now - no-op operations produce a comment in the down files if the an operation is not reversible
|
Looks like we're now creating those up and down files! 
|
This commit is aimed at making sqlschema models more extensible
and the migration-planning logic more readable. Key changes are:
- TableDefinition and Column definition both
have Additional interface{} field to allow adding inspector-specific
data which remains opaque to the general bulk of the code.
For example, SchemaInspector passes (*Model)(nil) zero interface
which is used to create a table.
- TableDefinitions is structured as a map, similarly to ColumnDefinitions.
Map lookups are convenient and are more readable than the custom tableSet
struct we used before. The latter is now retired.
- sqlschema exposes data structures for describing the databse schema and its
interface should not be cluttered with utilities only used by Detector.
Which is why both Signature and RefMap are moved to migrate package.
- Simplified refMap only has the essential methods. It allows detector to
avoid re-creating FKs for tables/columns that were renamed.
- Refactored a whole bunch of DependsOn() methods for various operations to
leverage the new unitility methods on sqlschema.ForeignKey and Columns.
- Deleted tests for refMap, as they are now an implementation detail and not
part of the API.
- TableDefinition (base implementation) now uses FQN as a key for every table to be able to store tables from different schemas with the same name
This feature will be added in the follow-up pull requests.
|
@bevzzz few things that I've noticed:
So we will have: type Table interface {
GetTables() *orderedmap.Map[string, Table]
}
// BaseTable is a base table definition.
//
// Dialects and only dialects can use it to implement the Table interface.
// Other packages must use the Table interface.WDYT? |
No, I don't think moving schemas between tables is possible. Rather, I wanted to make sure that the tables in different schemas aren't shadowing each other names, i.e. I want to be able to rename Most databases I've worked with had several schemas, so I thought this may make automigrations more resilient to name clashes. UPD: I realize now that we can achieve the same by using the name from the
Sure, I would even argue that Other than the different order in which Operations may be applied, is there any reason a lack of order in |
I assume that query generation won't be stable if 2 columns/tables are changed at once. I also see that the tests rely on having a |
|
@bevzzz I am going to introduce Regarding schemas, I think that we might need to accept a schema name via configuration and then only work with that schema. But that can be added later. |
|
@bevzzz я все :) Тесты я поправил но билд почему-то не проходит c непонятной ошибкой. Предлагаю пока это игнорировать. |
AutoMigrator only supports 1 schema at a time. Use WithSchemaName() option to configure it. Defaults to the default schema in the dialect.
Now that AutoMigrator only works with one schema at a time, there's no need to keep the code which was used ot differentiate tables between schemas
|
Short update: we've decided to drop multi-schema support, so the user would instead configure AutoMigrator to work with a particular schema by passing |
This PR is my second attempt at implementing #456 :)
This time around I decided not to meddle with
ALTER TABLEquery and focus on the functionality we want to bring -- AutoMigrator. I took into account our discussions in #726 and left it up to the dialects to implement "alter table" operations, keeping it internal. My thinking was that, if auto migration is done well, the users wouldn't need to work with ALTER TABLE all that much.AutoMigrator exposes a simple and minimal API:
CreateSQLMigrations(ctx)andMigrate(ctx, options). These are similar to Migrator except that they do not require writing any SQL/Go migrations manually.What AutoMigrator can do:
DEFAULTvalueNOT NULLandUNIQUEconstraints.tx.up.sql/.tx.down.sql)What it doesn't do:
AutoMigrator performs schema migrations and not data migrations. For example, when changing a primary key on the table, related foreign keys will not be updated -- they will continue referencing the old PK; to "re-link" them would require updating the referencing values themselves, which is beyond AutoMigrator's scope.
Presently, only
pgdialectsupports database inspection and migration, for which it implements two additional interfaces. AutoMigrator will return an informative error if used with a dialect that doesn't support this feature.Other changes:
test.shforwards additional arguments togo test(for example, now possible to run 1 test with
./test.sh -run=TestAutoMigrator_Run)Schema stringfield to differentiate tables in different schemas.