Temporal 1.24.0 <- 1.23.1 upgrade error #6050
Assignees
Labels
Comments
|
Bump. We experience the same bug after upgrade to 1.24.0 on postgresql. Any ideas hot to quickly fix this without waiting for new release? |
|
The release notes did not say that there is a schema update required for 1.24.0. Is there? |
|
@bllfr0g there is an schema update, but we applied it and we are still experiencing the same issue. ❯ git status
HEAD detached at v1.24.0
nothing to commit, working tree clean
❯ SQL_DATABASE=temporal ./temporal-sql-tool update -schema-dir schema/postgresql/v12/temporal/versioned
2024-06-03T09:31:30.000-0400 INFO UpdateSchemeTask started {"config": {"DBName":"","TargetVersion":"","SchemaDir":"schema/postgresql/v12/temporal/versioned","IsDryRun":false}, "logging-call-at": "updatetask.go:97"}
2024-06-03T09:31:30.151-0400 DEBUG Schema Dirs: [v1.12] {"logging-call-at": "updatetask.go:195"}
2024-06-03T09:31:30.152-0400 INFO Processing schema file: schema/postgresql/v12/temporal/versioned/v1.12/nexus_incoming_services.sql {"logging-call-at": "updatetask.go:241"}
2024-06-03T09:31:30.152-0400 DEBUG running 1 updates for current version 1.11 {"logging-call-at": "updatetask.go:131"}
2024-06-03T09:31:30.152-0400 DEBUG ---- Executing updates for version 1.12 ---- {"logging-call-at": "updatetask.go:150"}
2024-06-03T09:31:30.152-0400 DEBUG CREATE TABLE nexus_incoming_services ( service_id BYTEA NOT NULL, data BYTEA NOT NULL, data_encoding VARCHAR(16) NOT NULL, version BIGINT NOT NULL, PRIMARY KEY (service_id) ); {"logging-call-at": "updatetask.go:152"}
2024-06-03T09:31:30.475-0400 DEBUG CREATE TABLE nexus_incoming_services_partition_status ( id INT NOT NULL PRIMARY KEY DEFAULT 0, version BIGINT NOT NULL, CONSTRAINT only_one_row CHECK (id = 0) ); {"logging-call-at": "updatetask.go:152"}
2024-06-03T09:31:30.533-0400 DEBUG ---- Done ---- {"logging-call-at": "updatetask.go:169"}
2024-06-03T09:31:30.863-0400 DEBUG Schema updated from 1.11 to 1.12 {"logging-call-at": "updatetask.go:142"}
2024-06-03T09:31:30.863-0400 INFO UpdateSchemeTask done {"logging-call-at": "updatetask.go:120"}
❯ SQL_DATABASE=temporal_visibility ./temporal-sql-tool update -schema-dir schema/postgresql/v12/visibility/versioned
2024-06-03T09:31:39.527-0400 INFO UpdateSchemeTask started {"config": {"DBName":"","TargetVersion":"","SchemaDir":"schema/postgresql/v12/visibility/versioned","IsDryRun":false}, "logging-call-at": "updatetask.go:97"}
2024-06-03T09:31:39.685-0400 DEBUG Schema Dirs: [v1.5] {"logging-call-at": "updatetask.go:195"}
2024-06-03T09:31:39.688-0400 INFO Processing schema file: schema/postgresql/v12/visibility/versioned/v1.5/add_root_workflow_info.sql {"logging-call-at": "updatetask.go:241"}
2024-06-03T09:31:39.688-0400 DEBUG running 1 updates for current version 1.4 {"logging-call-at": "updatetask.go:131"}
2024-06-03T09:31:39.688-0400 DEBUG ---- Executing updates for version 1.5 ---- {"logging-call-at": "updatetask.go:150"}
2024-06-03T09:31:39.688-0400 DEBUG ALTER TABLE executions_visibility ADD COLUMN root_workflow_id VARCHAR(255) NULL; {"logging-call-at": "updatetask.go:152"}
2024-06-03T09:31:39.792-0400 DEBUG ALTER TABLE executions_visibility ADD COLUMN root_run_id VARCHAR(255) NULL; {"logging-call-at": "updatetask.go:152"}
2024-06-03T09:31:39.864-0400 DEBUG CREATE INDEX by_root_workflow_id ON executions_visibility (namespace_id, root_workflow_id, (COALESCE(close_time, '9999-12-31 23:59:59')) DESC, start_time DESC, run_id); {"logging-call-at": "updatetask.go:152"}
2024-06-03T09:31:43.395-0400 DEBUG CREATE INDEX by_root_run_id ON executions_visibility (namespace_id, root_run_id, (COALESCE(close_time, '9999-12-31 23:59:59')) DESC, start_time DESC, run_id); {"logging-call-at": "updatetask.go:152"}
2024-06-03T09:31:44.441-0400 DEBUG ---- Done ---- {"logging-call-at": "updatetask.go:169"}
2024-06-03T09:31:44.755-0400 DEBUG Schema updated from 1.4 to 1.5 {"logging-call-at": "updatetask.go:142"}
2024-06-03T09:31:44.755-0400 INFO UpdateSchemeTask done {"logging-call-at": "updatetask.go:120"} |
|
We also upgraded to the latest temporal-ui image we could find in docker hub (2.27.2). We still see the same error. We were previously using what the helm chart is using, 2.25.0. |
|
We noticed after enough new workflows, the error goes away. It seems the latest schema change introduced a new column You could clean up old entries in your namespace, or wait for them to expire or leave the first page of your queries when browsing. Still, hoping this gets fixed before upgrading our production environments. |
|
I am still shocked that there is a schema upgrade for 1.24.0 but it was not called out in the release notes. |
|
Thanks for reporting. Sorry for the miss. We will take this as high priority task. |
|
I updated release notes. Thanks for reporting. |
Merged
1 task
rodrigozhou
added a commit
that referenced
this issue
Jun 5, 2024
## What changed? <!-- Describe what has changed in this PR --> Fix SQL visibility root workflow fields in schema: they are supposed to be not null. ## Why? <!-- Tell your future self why have you made these changes --> The schema set the root workflow fields as nullable, but the code was assuming to be not null. GH Issue: #6050 ## How did you test it? <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> Start Temporal v1.23.1, create some workflows, update the schemas in v1.24.0, upgrade Temporal v1.24.0: - reproduced the error - applied this fix, and the error was gone. ## Potential risks <!-- Assuming the worst case, what can be broken when deploying this change to production? --> ## Documentation <!-- Have you made sure this change doesn't falsify anything currently stated in `docs/`? If significant new behavior is added, have you described that in `docs/`? --> ## Is hotfix candidate? <!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
stephanos
pushed a commit
that referenced
this issue
Jun 5, 2024
## What changed? <!-- Describe what has changed in this PR --> Fix SQL visibility root workflow fields in schema: they are supposed to be not null. ## Why? <!-- Tell your future self why have you made these changes --> The schema set the root workflow fields as nullable, but the code was assuming to be not null. GH Issue: #6050 ## How did you test it? <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> Start Temporal v1.23.1, create some workflows, update the schemas in v1.24.0, upgrade Temporal v1.24.0: - reproduced the error - applied this fix, and the error was gone. ## Potential risks <!-- Assuming the worst case, what can be broken when deploying this change to production? --> ## Documentation <!-- Have you made sure this change doesn't falsify anything currently stated in `docs/`? If significant new behavior is added, have you described that in `docs/`? --> ## Is hotfix candidate? <!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
|
Should be fixed now with v1.24.1. You will need to upgrade schema for that. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
Application should work normally after upgrading
Actual Behavior
Application is crashing after upgrading to version 1.24.0
Steps to Reproduce the Problem
Specifications
Sharing temporal-frontend logs:
The text was updated successfully, but these errors were encountered: