support trailing generic query responses #1441

gridnevvvit · 2024-01-30T20:43:39Z

Changelog entry

improve performance for single row responses in generic query

Changelog category

Improvement

Additional information

...

github-actions · 2024-01-30T20:46:59Z

⚪ 2024-01-30 20:46:59 UTC Pre-commit check for ded3818 has started.
⚪ 2024-01-30 20:47:01 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-30 21:23:06 UTC Build successful.
⚪ 2024-01-30 21:23:18 UTC Tests are running...
🔴 2024-01-30 23:03:45 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16022	15728	0	52	53	189

github-actions · 2024-01-30T20:47:00Z

⚪ 2024-01-30 20:47:00 UTC Pre-commit check for ded3818 has started.
⚪ 2024-01-30 20:47:03 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-30 21:22:45 UTC Build successful.
⚪ 2024-01-30 21:23:01 UTC Tests are running...
🔴 2024-01-30 22:52:49 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
60221	50730	0	44	9263	184

github-actions · 2024-01-31T18:01:47Z

⚪ 2024-01-31 18:01:47 UTC Pre-commit check for 95fc2b4 has started.
⚪ 2024-01-31 18:01:48 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-31 18:38:38 UTC Build successful.
⚪ 2024-01-31 18:38:49 UTC Tests are running...
🔴 2024-01-31 20:28:45 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
60412	49815	0	605	9397	595

github-actions · 2024-01-31T18:02:33Z

⚪ 2024-01-31 18:02:32 UTC Pre-commit check for 95fc2b4 has started.
⚪ 2024-01-31 18:02:34 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-31 18:38:38 UTC Build successful.
⚪ 2024-01-31 18:38:49 UTC Tests are running...
🔴 2024-01-31 20:35:22 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16086	14711	0	621	193	561

github-actions · 2024-01-31T22:30:46Z

⚪ 2024-01-31 22:30:45 UTC Pre-commit check for 0e90a02 has started.
⚪ 2024-01-31 22:30:48 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-31 23:09:38 UTC Build successful.
⚪ 2024-01-31 23:09:51 UTC Tests are running...
🔴 2024-01-31 23:15:22 UTC Test run completed, no test results found for commit aeab9d4. Please check build logs.
⚫ 2024-01-31 23:15:26 UTC Check cancelled

github-actions · 2024-01-31T22:30:53Z

⚪ 2024-01-31 22:30:53 UTC Pre-commit check for 0e90a02 has started.
⚪ 2024-01-31 22:30:55 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-31 23:07:37 UTC Build successful.
⚪ 2024-01-31 23:07:54 UTC Tests are running...
🔴 2024-01-31 23:15:34 UTC Test run completed, no test results found for commit aeab9d4. Please check build logs.
⚫ 2024-01-31 23:15:37 UTC Check cancelled

github-actions · 2024-01-31T23:16:56Z

⚪ 2024-01-31 23:16:56 UTC Pre-commit check for 9d1fa49 has started.
⚪ 2024-01-31 23:16:58 UTC Build linux-x86_64-release-asan is running...
🟢 2024-01-31 23:19:11 UTC Build successful.
⚪ 2024-01-31 23:19:24 UTC Tests are running...
🔴 2024-02-01 00:55:18 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
16040	15930	0	16	50	44

github-actions · 2024-01-31T23:19:11Z

⚪ 2024-01-31 23:19:11 UTC Pre-commit check for 9d1fa49 has started.
⚪ 2024-01-31 23:19:13 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-01-31 23:21:37 UTC Build successful.
⚪ 2024-01-31 23:21:51 UTC Tests are running...
🟢 2024-02-01 00:49:23 UTC Tests successful.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
60356	51043	0	0	9270	43

ydb/core/grpc_services/query/rpc_execute_query.cpp

ydb/core/kqp/common/events/query.h

spuchin · 2024-02-01T22:27:02Z

ydb/core/kqp/query_data/kqp_query_data.cpp

@@ -77,6 +77,19 @@ Ydb::ResultSet* TKqpExecuterTxResult::GetYdb(google::protobuf::Arena* arena, TMa
    return ydbResult;
 }

+Ydb::ResultSet* TKqpExecuterTxResult::GetTrailingYdb(google::protobuf::Arena* arena, TMaybe<ui64>) {


В чем смысл второго аргумента?

spuchin · 2024-02-01T22:33:06Z

ydb/core/kqp/session_actor/kqp_session_actor.cpp

@@ -1532,6 +1538,23 @@ class TKqpSessionActor : public TActorBootstrapped<TKqpSessionActor> {

        // Result for scan query is sent directly to target actor.
        Y_ABORT_UNLESS(response->GetArena());
+        if (QueryState->PreparedQuery && QueryState->IsStreamResult()) {


Эта секция очень похожа на следующую, где !QueryState->IsStreamResult().
Там есть какие-то принципиальные отличия?

ydb/core/kqp/executer_actor/kqp_executer_impl.h

spuchin · 2024-02-01T22:39:38Z

ydb/core/kqp/executer_actor/kqp_executer_impl.h

+                auto ackEv = MakeHolder<NYql::NDq::TEvDqCompute::TEvChannelDataAck>();
+                ackEv->Record.SetSeqNo(computeData.Proto.GetSeqNo());
+                ackEv->Record.SetChannelId(channel.Id);
+                ackEv->Record.SetFreeSpace(50_MB);


Мы же проверили что канал уже FINISHED, какая ему разница на FreeSpace?

да, просто нужно как-то заполнить финальный ак. не вижу тут проблемы какой-то.

Ну константы всегда вопросы вызывают, почему просто 0 не поставить? )

короче, шаг влево - тесты начинают зависать. похоже что есть какие-то проверки на получателе, могу покопать отдельно.

spuchin · 2024-02-01T22:43:38Z

ydb/core/kqp/executer_actor/kqp_executer_impl.h

+        this->Send(channelComputeActorId, ackEv.Release(), /* TODO: undelivery */ 0, /* cookie */ channel.Id);
+    }
+
+    void HandleStreamAck(TEvKqpExecuter::TEvStreamDataAck::TPtr& ev) {


Вот тут кстати не до конца понятно.

Если используется механизм trailing result, то аков нам сюда не прилетит.

Если не используется, то почему бы rpc актору не отправлять Ack'и напрямую в CA?

вообще идея норм, я поправлю, но пока я тут оставлю этот код на всякий.

короче, шаг влево - тесты начинают зависать. похоже что есть какие-то проверки на получателе, могу покопать отдельно.

ydb/core/grpc_services/query/rpc_execute_query.cpp

github-actions · 2024-02-02T16:16:26Z

⚪ 2024-02-02 16:16:26 UTC Pre-commit check for 2baeee6 has started.
⚪ 2024-02-02 16:16:28 UTC Build linux-x86_64-relwithdebinfo is running...
⚫ 2024-02-02 16:21:02 UTC Check cancelled

github-actions · 2024-02-02T16:18:43Z

⚪ 2024-02-02 16:18:43 UTC Pre-commit check for 2baeee6 has started.
⚪ 2024-02-02 16:18:45 UTC Build linux-x86_64-release-asan is running...
⚫ 2024-02-02 16:21:03 UTC Check cancelled

github-actions · 2024-02-02T16:24:39Z

⚪ 2024-02-02 16:24:38 UTC Pre-commit check for 7208485 has started.
⚪ 2024-02-02 16:24:41 UTC Build linux-x86_64-relwithdebinfo is running...
⚫ 2024-02-02 16:51:31 UTC Check cancelled

github-actions · 2024-02-02T16:26:59Z

⚪ 2024-02-02 16:26:59 UTC Pre-commit check for 7208485 has started.
⚪ 2024-02-02 16:27:01 UTC Build linux-x86_64-release-asan is running...
⚫ 2024-02-02 16:51:32 UTC Check cancelled

github-actions · 2024-02-02T16:52:36Z

⚪ 2024-02-03 10:39:14 UTC Pre-commit check for 12af35a has started.
⚪ 2024-02-03 10:39:17 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-02-03 10:40:31 UTC Build successful.
⚪ 2024-02-03 10:40:49 UTC Tests are running...
🔴 2024-02-03 12:01:45 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
58860	49567	0	3	9263	27

github-actions · 2024-02-02T16:55:07Z

⚪ 2024-02-03 14:44:11 UTC Pre-commit check for 50a622b has started.
⚪ 2024-02-03 14:44:14 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-02-03 14:45:26 UTC Build successful.
⚪ 2024-02-03 14:45:42 UTC Tests are running...
🔴 2024-02-03 14:53:13 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
58760	49458	0	6	9268	28

github-actions · 2024-02-02T16:55:12Z

⚪ 2024-02-02 16:55:11 UTC Pre-commit check for 50a622b has started.
⚪ 2024-02-02 16:55:14 UTC Build linux-x86_64-release-asan is running...
🟢 2024-02-02 17:10:14 UTC Build successful.
⚪ 2024-02-02 17:10:27 UTC Tests are running...
🔴 2024-02-02 18:46:22 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14533	14399	0	41	61	32

github-actions · 2024-02-03T10:39:25Z

⚪ 2024-02-03 10:39:25 UTC Pre-commit check for 12af35a has started.
⚪ 2024-02-03 10:39:27 UTC Build linux-x86_64-release-asan is running...
🟢 2024-02-03 10:40:37 UTC Build successful.
⚪ 2024-02-03 10:40:50 UTC Tests are running...
🔴 2024-02-03 12:16:48 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14530	14395	0	31	53	51

github-actions bot added the improvement label Jan 30, 2024

gridnevvvit force-pushed the trailing-generic-query-responses branch from bb1736b to 148e7bc Compare January 31, 2024 16:55

gridnevvvit force-pushed the trailing-generic-query-responses branch from 148e7bc to aeab9d4 Compare January 31, 2024 22:27

gridnevvvit force-pushed the trailing-generic-query-responses branch from aeab9d4 to ab89e4b Compare January 31, 2024 23:14

spuchin reviewed Feb 1, 2024

View reviewed changes

gridnevvvit force-pushed the trailing-generic-query-responses branch from ab89e4b to fb174eb Compare February 2, 2024 16:13

gridnevvvit force-pushed the trailing-generic-query-responses branch from fb174eb to e13b4f1 Compare February 2, 2024 16:20

gridnevvvit force-pushed the trailing-generic-query-responses branch from e13b4f1 to d12fb13 Compare February 2, 2024 16:51

support trailing generic query responses

66d80ab

gridnevvvit force-pushed the trailing-generic-query-responses branch from d12fb13 to 66d80ab Compare February 2, 2024 16:53

spuchin approved these changes Feb 2, 2024

View reviewed changes

gridnevvvit merged commit 5594dac into ydb-platform:main Feb 3, 2024
2 of 4 checks passed

gridnevvvit deleted the trailing-generic-query-responses branch February 3, 2024 16:34

FloatingCrowbar mentioned this pull request Feb 4, 2024

Сhange keys format for service partitions #1569

Merged

vitstn mentioned this pull request Feb 7, 2024

YQL-17725 drop b and t suffixes #1660

Merged

GrigoriyPA mentioned this pull request Feb 7, 2024

KqpRun add results into clear execution #1661

Merged

This was referenced Feb 7, 2024

YQL-17725 tune doc #1667

Merged

Switch pg_parser BC tests on again #1709

Merged

unmute test #1711

Merged

ildar-khisambeev mentioned this pull request Feb 8, 2024

LOGBROKER-8783 bugfix #1713

Merged

vitstn mentioned this pull request Feb 8, 2024

Mark mutable_id for YT writes #1744

Merged

niksaveliev mentioned this pull request Feb 9, 2024

Fix pq writer and few renames #1757

Merged

GrigoriyPA mentioned this pull request Feb 9, 2024

Moved optimizers from yql/providers/dq to yql/dq #1758

Merged

pavelvelikhov mentioned this pull request Feb 9, 2024

Cannonized two plans #1773

Merged

niksaveliev mentioned this pull request Feb 12, 2024

Workload read without consumer and metrics fixes #1792

Merged

starlinskiy mentioned this pull request Feb 12, 2024

runtime dispatching #1800

Closed

vitstn mentioned this pull request Feb 12, 2024

quick fix for column name of select version() #1801

Merged

GrigoriyPA mentioned this pull request Feb 12, 2024

YQ 2786 fix test fq restarts #1810

Merged

vitstn mentioned this pull request Feb 12, 2024

Support of database sys cache #1817

Merged

starlinskiy mentioned this pull request Feb 12, 2024

runtime dispatching (fixed) #1847

Merged

pavelvelikhov mentioned this pull request Feb 13, 2024

Fixed a problem with simplified plan JSONs #1878

Merged

This was referenced Feb 13, 2024

Case insensivity for pg_catalog tables/cluster, columns. Support of pg_class:relam, and some columns in pg_database & pg_namespace #1893

Merged

fixed pg_get_userbyid function #1936

Merged

support of AnyNonArray type, fixes for array_agg #1958

Merged

pavelvelikhov mentioned this pull request Feb 15, 2024

Fixed a problem with plans in PG ut #1974

Merged

niksaveliev mentioned this pull request Feb 16, 2024

Fix kafka with enabled proxy #2017

Merged

vitstn mentioned this pull request Feb 16, 2024

mock of savepoints #2026

Merged

aakulaga-ydb mentioned this pull request Feb 24, 2024

Fix TzTimestamp wrong encoding (YQL-17920) #2222

Merged

GrigoriyPA mentioned this pull request Feb 26, 2024

YQ-2898 fix query failed with unknown uncommitted upload issue #2240

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support trailing generic query responses #1441

support trailing generic query responses #1441

gridnevvvit commented Jan 30, 2024

github-actions bot commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

spuchin Feb 1, 2024

spuchin Feb 1, 2024

spuchin Feb 1, 2024

gridnevvvit Feb 2, 2024

spuchin Feb 2, 2024

gridnevvvit Feb 2, 2024

spuchin Feb 1, 2024

gridnevvvit Feb 2, 2024

gridnevvvit Feb 2, 2024

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 3, 2024 •

edited

Loading

support trailing generic query responses #1441

support trailing generic query responses #1441

Conversation

gridnevvvit commented Jan 30, 2024

Changelog entry

Changelog category

Additional information

github-actions bot commented Jan 30, 2024 • edited Loading

github-actions bot commented Jan 30, 2024 • edited Loading

github-actions bot commented Jan 31, 2024 • edited Loading

github-actions bot commented Jan 31, 2024 • edited Loading

github-actions bot commented Jan 31, 2024 • edited Loading

github-actions bot commented Jan 31, 2024 • edited Loading

github-actions bot commented Jan 31, 2024 • edited Loading

github-actions bot commented Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 2, 2024 • edited Loading

github-actions bot commented Feb 2, 2024 • edited Loading

github-actions bot commented Feb 2, 2024 • edited Loading

github-actions bot commented Feb 2, 2024 • edited Loading

github-actions bot commented Feb 2, 2024 • edited Loading

github-actions bot commented Feb 2, 2024 • edited Loading

github-actions bot commented Feb 2, 2024 • edited Loading

github-actions bot commented Feb 3, 2024 • edited Loading

github-actions bot commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 3, 2024 •

edited

Loading