Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update v61 migration to handle duplicate job names before unique constraint #2464

Merged
merged 2 commits into from
Mar 31, 2023
Merged

Update v61 migration to handle duplicate job names before unique constraint #2464

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9912536
Update v61 migration to handle duplicate job names before unique cons…
collado-mike Mar 30, 2023
0ef8e13
Merge branch 'main' into fix/61_db_migration_duplicate_jobs
wslulciuc Mar 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 39 additions & 4 deletions api/src/main/resources/marquez/db/migration/V61__unique_job_fqn_index.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,45 @@
WITH fqn AS (
SELECT f.uuid,
f.job_fqn AS name,
f.namespace_name,
j.name AS simple_name,
j.parent_job_uuid,
f.parent_job_name::text,
j.symlink_target_uuid
FROM jobs_fqn f,
jobs j
WHERE j.uuid = f.uuid
)
UPDATE jobs SET symlink_target_uuid=q.target_uuid
FROM (
SELECT j.uuid, j.namespace_name, j.name, j.simple_name, jv.uuid AS target_uuid, jv.simple_name
FROM jobs_view j INNER JOIN jobs_view jv ON j.namespace_name=jv.namespace_name AND j.name=jv.name AND j.simple_name != jv.simple_name
FROM fqn j
INNER JOIN fqn jv ON j.namespace_name=jv.namespace_name AND j.name=jv.name AND j.simple_name != jv.simple_name
WHERE j.symlink_target_uuid IS NULL
AND jv.symlink_target_uuid IS NULL
AND j.parent_job_uuid IS NULL) q
AND jv.symlink_target_uuid IS NULL
AND j.parent_job_uuid IS NULL) q
WHERE jobs.uuid=q.uuid;

WITH fqn AS (
SELECT f.uuid,
f.job_fqn AS name,
f.namespace_name,
j.name AS simple_name,
j.parent_job_uuid,
f.parent_job_name::text,
j.symlink_target_uuid,
j.created_at
FROM jobs_fqn f,
jobs j
WHERE j.uuid = f.uuid
)
UPDATE jobs SET name=(q.simple_name || '_' || q.row)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will set a job with a conflict to {job_name}_{row_num}?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but row_number() over (PARTITION BY j.namespace_name, j.name ORDER BY j.created_at) AS row means that all jobs that have the same name will have an incrementing counter suffixed so we avoid the conflict. E.g., if we had three jobs with the FQN a.b, the names would be

  1. a.b_1
  2. a.b_2
  3. a.b_3
    Thus, we can add the uniqueness constraint because they'll all end up with different names.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet, thanks for clarifying and the examples 👍

FROM (
SELECT j.uuid, j.namespace_name, j.name, j.simple_name, jv.uuid AS target_uuid,
row_number() over (PARTITION BY j.namespace_name, j.name ORDER BY j.created_at) AS row
FROM fqn j
INNER JOIN fqn jv ON j.namespace_name=jv.namespace_name AND j.name=jv.name AND j.symlink_target_uuid=jv.uuid
) q
WHERE jobs.uuid=q.uuid;

ALTER TABLE jobs RENAME COLUMN name TO simple_name;
Expand All @@ -30,4 +65,4 @@ WHERE jobs.uuid=f.uuid;
ALTER TABLE jobs ALTER COLUMN name SET NOT NULL;

ALTER TABLE jobs DROP CONSTRAINT unique_jobs_namespace_uuid_name_parent;
ALTER TABLE jobs ADD CONSTRAINT unique_jobs_namespace_uuid_name_parent UNIQUE (namespace_uuid, name);
ALTER TABLE jobs ADD CONSTRAINT unique_jobs_namespace_uuid_name_parent UNIQUE (namespace_uuid, name);