You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Aug 16, 2022. It is now read-only.
Running cloudquery fetch fails with many ERROR: relation "<some relation>" does not exist (SQLSTATE 42P01)
Who is affected?
Setups where the username used to connect to PostgreSQL is cloudquery . More specifically, You may encounter this problem if you are running an older version of the cloudquery binary (<= v0.28.0), and a newer version of the provider ( aws >= v0.12.17 , gcp >= v0.8.16, azure >= v0.11.13, k8s >= v0.5.12, terraform >= v0.4.11, okta >= v0.5.11, digitalocean >= v0.5.13).
What is the cause of the issue?
PostgreSQL tables belong to ‘schemas’. By default, when accessing (Creating/Inserting/Selecting) tables, the schema is decided by a search_path . The default search_path is "$user", public. This means that, when creating/inserting/selecting a table, PostgreSQL will first try the schema with the same name as your PosgreSQL username. If such a table is not found. For instance, When connecting to PostgreSQL with username someuser and running SELECT * FROM sometable, PostgreSQL will first look for someuser.sometable, and, if the table is not found, the table public.sometable.
This can cause unpredictable behavior that depends on the PostgreSQL username. For instance, when running with PostgreSQL username cloudquery, tables are created in the cloudquery schema (usually used for internal cloudquery metadata), instead of the public schema.
This change introduced compatibility issues when running a newer provider version and an older cloudquery executable. The older cloudquery executable will create tables in cloudquery schema, but the newer provider will look for the tables in the public schema.
What action can I take to fix this?
Option 1 - upgrade your cloudquery executable
Run cloudquery provider drop [your-provider] —-force. (Required, because your tables need to be recreated). Note that this will drop all your tables until you re-run fetch!
Upgrade your cloudquery executable to v0.28.0 or above
Note that if upgrading to v0.29.0 or above, you will encounter a different breaking change: the move from config.hcl file to cloudquery.yml . If choosing this route, you will need to either:
Convert your config.hcl to a cloudquery.yml file (re-run cloudquery init and edit the connection and providers sections. You can also take a look at examples on our docs and hub).
Add --config config.hcl to your commandlines (hcl is still supported for now, but requires this explicit commandline flag).
What is the issue
Running
cloudquery fetch
fails with manyERROR: relation "<some relation>" does not exist (SQLSTATE 42P01)
Who is affected?
Setups where the username used to connect to PostgreSQL is
cloudquery
. More specifically, You may encounter this problem if you are running an older version of thecloudquery
binary (<= v0.28.0
), and a newer version of the provider (aws >= v0.12.17
,gcp >= v0.8.16
,azure >= v0.11.13
,k8s >= v0.5.12
,terraform >= v0.4.11
,okta >= v0.5.11
,digitalocean >= v0.5.13
).What is the cause of the issue?
search_path
. The defaultsearch_path
is"$user", public
. This means that, when creating/inserting/selecting a table, PostgreSQL will first try the schema with the same name as your PosgreSQL username. If such a table is not found. For instance, When connecting to PostgreSQL with usernamesomeuser
and runningSELECT * FROM sometable
, PostgreSQL will first look forsomeuser.sometable
, and, if the table is not found, the tablepublic.sometable
.cloudquery
, tables are created in thecloudquery
schema (usually used for internal cloudquery metadata), instead of thepublic schema
.search_path=public
.cloudquery
executable. The oldercloudquery
executable will create tables incloudquery
schema, but the newer provider will look for the tables in thepublic
schema.What action can I take to fix this?
Option 1 - upgrade your
cloudquery
executableRun
cloudquery provider drop [your-provider] —-force
. (Required, because your tables need to be recreated). Note that this will drop all your tables until you re-run fetch!Upgrade your
cloudquery
executable tov0.28.0
or aboveNote that if upgrading to
v0.29.0
or above, you will encounter a different breaking change: the move fromconfig.hcl
file tocloudquery.yml
. If choosing this route, you will need to either:config.hcl
to acloudquery.yml
file (re-runcloudquery init
and edit theconnection
andproviders
sections. You can also take a look at examples on our docs and hub).--config config.hcl
to your commandlines (hcl is still supported for now, but requires this explicit commandline flag).Linux - latest version
Linux -
v0.28.0
:macOS (latest):
macOS (
v0.28.0
)re-run
cloudquery fetch
. You should no longer seetable not found
errors.Option 2 - pin your provider version to a version before this change, until you are ready to upgrade your
cloudquery
executable:config.hcl
file, change yourprovider
block to (choose the one you need:)Where can I get further assistance?
The text was updated successfully, but these errors were encountered: