-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use PostgresAdmin directly for DB backups #160
Commits on Apr 13, 2021
-
Configuration menu - View commit details
-
Copy full SHA for b57d6f4 - Browse repository at this point
Copy the full SHA b57d6f4View commit details -
[PostgresAdmin] Move .with_pg_connection method
Allows this to be shared with other classes.
Configuration menu - View commit details
-
Copy full SHA for aac2c37 - Browse repository at this point
Copy the full SHA aac2c37View commit details
Commits on Apr 27, 2021
-
[DatabaseAdmin] Use PostgresAdmin directly
Instead of calling a rake task in core, call `PostgresAdmin` which is now part of this repository. This takes many of the checks that exist in `lib/evm_database_ops.rb` (core) and puts them in this class, tweaking them slightly so they don't require things like `ActiveRecord` to function, or classes that only exist in core `(VmdbDatabaseConnection` for example) In addition, this also doesn't include any of the `.with_file_storage` calls as that will be phased out. `PostgresAdmin` just needs a local file to write to, and since `MiqFileStorage` used a FIFO to write data to external sources (s3, swift, FTP), there is no change required there, and this can just accept a local file and pass that to `PostgresAdmin`.
Configuration menu - View commit details
-
Copy full SHA for 3680658 - Browse repository at this point
Copy the full SHA 3680658View commit details
Commits on May 17, 2021
-
[PostgresAdmin] Move in prepare logic
The prepare logic should live in the PostgresAdmin class as it is something that could be used for not only the console, but also CLI. Also, fixed a few things that wouldn't have worked (integration testing of `restore` hasn't been done with this code yet): - Removed the MiqRegion.replication_type= call (not used, I don't think) - Remove connection_pool disconnect (not relevant, since we aren't using ActiveRecord with this code base) - Fixed .connection_count to handle things differently per backup type and dbname (this existed previously with VmdbDatabaseConnection, but was not properly ported over) Everything else should be the same logically speaking, just updated for working within the PostgresAdmin class itself.
Configuration menu - View commit details
-
Copy full SHA for baa30f5 - Browse repository at this point
Copy the full SHA baa30f5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 62d1c28 - Browse repository at this point
Copy the full SHA 62d1c28View commit details -
[PostgresAdmin] Use pg-logical_replication
Use PG::LogicalReplication::Client to remove subscriptions or publications on a db prior to doing a database reload. Doesn't drop the `MiqRegion`, as was previously done as part of PglogicalSubscription in core: https://github.com/ManageIQ/manageiq/blob/8ca93da126aada556685db5d05aaf5e9a0666900/app/models/pglogical_subscription.rb#L70 But since this is a database reload, and the Region will be removed from the DB anyway, this should not matter, and the necessary code required to perform that delete will be pointless to include in this gem.
Configuration menu - View commit details
-
Copy full SHA for dce893b - Browse repository at this point
Copy the full SHA dce893bView commit details -
[PostgresAdmin] Close connection only if exists
If an error occurs while connecting (for example, if you have the wrong user), then this will fail with just that error, instead of an error that exists in the ensure as well.
Configuration menu - View commit details
-
Copy full SHA for fa461c6 - Browse repository at this point
Copy the full SHA fa461c6View commit details -
[PostgresRunner] Stub .with_pg_connection for spec:dev
During a restore, when calling `PostgresAdmin.with_pg_connection`, use the one defined in `PostgresRunner` since it includes connection information for the secondary DB, instead of the one running default one running on the user's machine.
Configuration menu - View commit details
-
Copy full SHA for 3b49216 - Browse repository at this point
Copy the full SHA 3b49216View commit details -
[PostgresRunner] Add .hard_reset (local only)
Adds a helper method to PostgresRunner (.hard_reset), which is used to reset the database to a fresh state. Required for local tests only since the DB backup was taken off of an appliance, so when that is restored first, it changes the default DB roles to that `.initial_setup` create and is assumed by the class.
Configuration menu - View commit details
-
Copy full SHA for 9a26852 - Browse repository at this point
Copy the full SHA 9a26852View commit details -
[postgres_admin_spec.rb] Ensure env is setup
For the `.restore` specs, ensure the env is setup properly so the specs can run properly. With the new methods that check the existing database connections in `PostgresAdmin` directly, this allows said connection to work properly. For CI, we don't want to do this, since it doesn't actually setup the ENV (bad method name, my bad...), but runs the restore in a task all by itself.
Configuration menu - View commit details
-
Copy full SHA for 5c99876 - Browse repository at this point
Copy the full SHA 5c99876View commit details -
[postgres_admin_spec.rb] Disable pipe spec
Doesn't seem to be working currently (at least locally), and supporting this isn't terribly necessary now that we won't be streaming files in Ruby. Might delete later...
Configuration menu - View commit details
-
Copy full SHA for 98bf4fb - Browse repository at this point
Copy the full SHA 98bf4fbView commit details -
[PostgresRunner] Simplify .restore and specs
Previously, the logic to pre-determine the backup type required knowledge of the file storage type that PostgresAdmin did not have. In the case where the file storage data was being streamed (S3, FTP, etc.) and was only read from the source once, you had to call out to the `PostgresAdmin.pg_dump_file?` and `PostgresAdmin.base_backup_file?` calls manually to determine what type file was going to be restored so the proper code could be executed, and `PostgresAdmin` didn't have to worry if the file data was being streamed or was just read off of a disk/mount. Since .validate_backup_file_type (not show in diff) is now part of .restore, this extra case logic is no longer necessary, as `backup_type` will always be set if not passed in. As such, the code and spec for it has been updated to reflect that.
Configuration menu - View commit details
-
Copy full SHA for 23b2a49 - Browse repository at this point
Copy the full SHA 23b2a49View commit details -
[CiPostgresRunner] Stub ManageIQ:ApplianceConsole.logger
When not in a RSpec context (example: when we are doing database operations in a separate process in CI), ensure that we have configured a `ManageIQ::ApplianceConsole.logger` so things don't break in CI.
Configuration menu - View commit details
-
Copy full SHA for 662dcee - Browse repository at this point
Copy the full SHA 662dceeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 607d6c5 - Browse repository at this point
Copy the full SHA 607d6c5View commit details -
[CiPostgresRunner]
sudo sed
instead of>>
Using `>>` requires being at the proper elevated permission at the time the command is executed, and since that isn't possible with these scripts, it is simpler to use sed for this, and append a `sudo` to the front.
Configuration menu - View commit details
-
Copy full SHA for 7612d81 - Browse repository at this point
Copy the full SHA 7612d81View commit details -
[CiPostgresRunner] Don't copy pg_hba.conf
This seems to cause issues, and postgres on Travis is setup to use `trust` instead of `peer`. Since the tests generally work with the default, continue using that instead of using what is brought over from a `.restore`.
Configuration menu - View commit details
-
Copy full SHA for bcb52d7 - Browse repository at this point
Copy the full SHA bcb52d7View commit details