Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use PostgresAdmin directly for DB backups #160

Merged
merged 16 commits into from
May 18, 2021

Commits on Apr 13, 2021

  1. Configuration menu
    Copy the full SHA
    b57d6f4 View commit details
    Browse the repository at this point in the history
  2. [PostgresAdmin] Move .with_pg_connection method

    Allows this to be shared with other classes.
    NickLaMuro committed Apr 13, 2021
    Configuration menu
    Copy the full SHA
    aac2c37 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2021

  1. [DatabaseAdmin] Use PostgresAdmin directly

    Instead of calling a rake task in core, call `PostgresAdmin` which is
    now part of this repository.  This takes many of the checks that exist
    in `lib/evm_database_ops.rb` (core) and puts them in this class,
    tweaking them slightly so they don't require things like `ActiveRecord`
    to function, or classes that only exist in core
    `(VmdbDatabaseConnection` for example)
    
    In addition, this also doesn't include any of the `.with_file_storage`
    calls as that will be phased out.  `PostgresAdmin` just needs a local
    file to write to, and since `MiqFileStorage` used a FIFO to write data
    to external sources (s3, swift, FTP), there is no change required there,
    and this can just accept a local file and pass that to `PostgresAdmin`.
    NickLaMuro committed Apr 27, 2021
    Configuration menu
    Copy the full SHA
    3680658 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2021

  1. [PostgresAdmin] Move in prepare logic

    The prepare logic should live in the PostgresAdmin class as it is
    something that could be used for not only the console, but also CLI.
    
    Also, fixed a few things that wouldn't have worked (integration testing
    of `restore` hasn't been done with this code yet):
    
    - Removed the MiqRegion.replication_type= call (not used, I don't think)
    - Remove connection_pool disconnect (not relevant, since we aren't using
      ActiveRecord with this code base)
    - Fixed .connection_count to handle things differently per backup type
      and dbname (this existed previously with VmdbDatabaseConnection, but
      was not properly ported over)
    
    Everything else should be the same logically speaking, just updated for
    working within the PostgresAdmin class itself.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    baa30f5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    62d1c28 View commit details
    Browse the repository at this point in the history
  3. [PostgresAdmin] Use pg-logical_replication

    Use PG::LogicalReplication::Client to remove subscriptions or
    publications on a db prior to doing a database reload.
    
    Doesn't drop the `MiqRegion`, as was previously done as part of
    PglogicalSubscription in core:
    
      https://github.com/ManageIQ/manageiq/blob/8ca93da126aada556685db5d05aaf5e9a0666900/app/models/pglogical_subscription.rb#L70
    
    But since this is a database reload, and the Region will be removed from
    the DB anyway, this should not matter, and the necessary code required
    to perform that delete will be pointless to include in this gem.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    dce893b View commit details
    Browse the repository at this point in the history
  4. [PostgresAdmin] Close connection only if exists

    If an error occurs while connecting (for example, if you have the wrong
    user), then this will fail with just that error, instead of an error
    that exists in the ensure as well.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    fa461c6 View commit details
    Browse the repository at this point in the history
  5. [PostgresRunner] Stub .with_pg_connection for spec:dev

    During a restore, when calling `PostgresAdmin.with_pg_connection`, use
    the one defined in `PostgresRunner` since it includes connection
    information for the secondary DB, instead of the one running default one
    running on the user's machine.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    3b49216 View commit details
    Browse the repository at this point in the history
  6. [PostgresRunner] Add .hard_reset (local only)

    Adds a helper method to PostgresRunner (.hard_reset), which is used to
    reset the database to a fresh state.
    
    Required for local tests only since the DB backup was taken off of an
    appliance, so when that is restored first, it changes the default DB
    roles to that `.initial_setup` create and is assumed by the class.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    9a26852 View commit details
    Browse the repository at this point in the history
  7. [postgres_admin_spec.rb] Ensure env is setup

    For the `.restore` specs, ensure the env is setup properly so the specs
    can run properly.
    
    With the new methods that check the existing database connections in
    `PostgresAdmin` directly, this allows said connection to work properly.
    
    For CI, we don't want to do this, since it doesn't actually setup the
    ENV (bad method name, my bad...), but runs the restore in a task all by
    itself.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    5c99876 View commit details
    Browse the repository at this point in the history
  8. [postgres_admin_spec.rb] Disable pipe spec

    Doesn't seem to be working currently (at least locally), and supporting
    this isn't terribly necessary now that we won't be streaming files in
    Ruby.
    
    Might delete later...
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    98bf4fb View commit details
    Browse the repository at this point in the history
  9. [PostgresRunner] Simplify .restore and specs

    Previously, the logic to pre-determine the backup type required
    knowledge of the file storage type that PostgresAdmin did not have.  In
    the case where the file storage data was being streamed (S3, FTP, etc.)
    and was only read from the source once, you had to call out to the
    `PostgresAdmin.pg_dump_file?` and `PostgresAdmin.base_backup_file?`
    calls manually to determine what type file was going to be restored so
    the proper code could be executed, and `PostgresAdmin` didn't have to
    worry if the file data was being streamed or was just read off of a
    disk/mount.
    
    Since .validate_backup_file_type (not show in diff) is now part of
    .restore, this extra case logic is no longer necessary, as `backup_type`
    will always be set if not passed in.
    
    As such, the code and spec for it has been updated to reflect that.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    23b2a49 View commit details
    Browse the repository at this point in the history
  10. [CiPostgresRunner] Stub ManageIQ:ApplianceConsole.logger

    When not in a RSpec context (example: when we are doing database
    operations in a separate process in CI), ensure that we have configured
    a `ManageIQ::ApplianceConsole.logger` so things don't break in CI.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    662dcee View commit details
    Browse the repository at this point in the history
  11. [CiPostgresRunner] sudo cp

    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    607d6c5 View commit details
    Browse the repository at this point in the history
  12. [CiPostgresRunner] sudo sed instead of >>

    Using `>>` requires being at the proper elevated permission at the time
    the command is executed, and since that isn't possible with these
    scripts, it is simpler to use sed for this, and append a `sudo` to the
    front.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    7612d81 View commit details
    Browse the repository at this point in the history
  13. [CiPostgresRunner] Don't copy pg_hba.conf

    This seems to cause issues, and postgres on Travis is setup to use
    `trust` instead of `peer`.  Since the tests generally work with the
    default, continue using that instead of using what is brought over from
    a `.restore`.
    NickLaMuro committed May 17, 2021
    Configuration menu
    Copy the full SHA
    bcb52d7 View commit details
    Browse the repository at this point in the history