Bug/57305 sorting by custom field has strong impact on performance for the project list #16841

toy · 2024-09-27T13:08:56Z

Ticket

OP#57305 + OP#57554

What are you trying to accomplish?

Screenshots

What approach did you choose and why?

Merge checklist

Added/updated tests
Added/updated documentation in Lookbook (patterns, previews, etc)
Tested major browsers (Chrome, Firefox, Edge, ...)

app/models/queries/projects/orders/custom_field_order.rb

toy · 2024-10-10T18:13:48Z

app/models/queries/work_packages/selects/custom_field_select.rb

+  def summable_statement
+    if %w[float int].include?(custom_field.field_format)
+      select = summable_select_statement
+
+      ->(query, grouped) {
+        Queries::WorkPackages::Selects::WorkPackageSelect
+          .scoped_column_sum(summable_scope(query), select, grouped:, query:)
+      }
    else
-      "COALESCE(ROUND(SUM(value::NUMERIC), 2)::FLOAT, 0.0) #{name}"
+      false
    end
  end


Diff may make it confusing, this is extracted from set_summable!

ulferts

The change looks good functionality and also performance wise, in the tests I have conducted. It is now possible to group by way more fields which when looking at community is nice.

There are some issues with the added grouping options, though. Of those, the first is but a display issue. But the second one leads to entries being sorted into the wrong group.

Grouping by integer cf

Here, the grouping happens with an integer cf. The groups however are presented as floats.

Grouping by link cf

The backend correctly returns just one item in the group. The frontend still places more entries into the group.

app/models/custom_field/order_statements.rb

toy · 2024-10-30T19:23:38Z

@ulferts I wasn't able to reproduce the problem with grouping, we should look at it together

toy · 2024-11-11T18:36:20Z

Grouping by link is fixed by 804986d, the issue was already there, but it was not possible to group by link custom field. Similar problem exists for grouping by bool custom fields, as absence of value will lead to nil used for grouping/ordering, but default value (which can only be true or false) used for work package custom field value, so currently there will be odd groups or adding to wrong group, the issue exists on dev.
Very confusing result is on the following screenshot, where only work packages with ids 40 and 5 have value set for boolean custom field, so only first 3 groups are correct (false, true and -), following groups are created because custom field value is available for type TASK, so defaults to false, but not available for other types, so has null value, and as frontend expects groups to be in order, it creates a group every time next row group value is different from one for current row. But the false group must be in the api response, as otherwise algorithm will not be be able to match it with row value and create it, so error doesn't show up, or at least that bad.

… the project list https://community.openproject.org/work_packages/57305

Previously it relied on receiving empty string

…QL 16

… is set after turbo:load too

toy · 2024-11-13T16:54:50Z

Last commit dc28e44 is also extracted as #17201

ulferts

👍

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch 4 times, most recently from f7fbb59 to 40034e9 Compare October 4, 2024 12:49

github-advanced-security bot found potential problems Oct 4, 2024

View reviewed changes

app/models/queries/projects/orders/custom_field_order.rb Fixed Show fixed Hide fixed

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch from 33ea0df to 29dac54 Compare October 4, 2024 13:14

github-advanced-security bot found potential problems Oct 4, 2024

View reviewed changes

app/models/queries/projects/orders/custom_field_order.rb Dismissed Show dismissed Hide dismissed

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch 4 times, most recently from 74aaecc to 73daf9f Compare October 10, 2024 18:07

toy commented Oct 10, 2024

View reviewed changes

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch 7 times, most recently from 172eb20 to 3dcfa9c Compare October 16, 2024 12:38

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch from 89a556f to 5bd80f6 Compare October 18, 2024 17:24

toy marked this pull request as ready for review October 18, 2024 17:26

ulferts requested changes Oct 21, 2024

View reviewed changes

app/models/custom_field/order_statements.rb Outdated Show resolved Hide resolved

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch from 5bd80f6 to e4acf8c Compare October 30, 2024 13:10

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch from e4acf8c to d103deb Compare November 1, 2024 17:37

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch 2 times, most recently from c45ae94 to 804986d Compare November 11, 2024 18:17

toy added 2 commits November 12, 2024 12:42

[#57305] Sorting by custom field has strong impact on performance for…

72dcb21

… the project list https://community.openproject.org/work_packages/57305

remove unused option if from work package property select

b6f64e2

toy added 21 commits November 12, 2024 12:42

rename CustomField#null_handling to order_null_handling

544e7d4

using joins for custom field ordering, grouping should be broken

edd62d8

move deciding which custom field can be grouped by to order statements

b886078

enable grouping by float, string and link too

4a72b89

remove replacing nil by false which doesn't seem to be needed

ac27483

handle joins for groupping by custom field

48eda6e

fix join statement for group by to return ids instead of positions

70b1acb

handle absent values in group by

119de6e

Previously it relied on receiving empty string

pass group_by_join_statement to WorkPackageSelect.scoped_column_sum

8b3ceff

disambigute value in summable_select_statement

924e9c8

use receive_messages as rubocop suggests

ac2c910

deduplicate queries using template

ac6202b

ensure group_by_join_statement is nil when group_by_statement is nil

76e4096

order empty values before present values for all custom field formats

526d287

fix order of list custom field groups to be same as when ordering

fbb246f

use MIN instead of ANY_VALUE, as latter is available only in PostgreS…

ae6702a

…QL 16

fix columns used in summing

e508268

use group_id instead of id to name the column used to group from summing

7e0ed44

extract method to calm rubocop

9c99bc8

add resulting join sqls from template in comment

bb9b32e

fix link strategy to return nil when value is not set

98ca48c

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch 2 times, most recently from 6f38bdc to 690b342 Compare November 13, 2024 13:59

move creation of angular boostrap __ng2-bootstrap-has-run mark, so it…

dc28e44

… is set after turbo:load too

toy force-pushed the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch from 690b342 to dc28e44 Compare November 13, 2024 14:20

toy requested a review from ulferts November 13, 2024 16:54

ulferts approved these changes Nov 15, 2024

View reviewed changes

ulferts merged commit 5a003d3 into dev Nov 15, 2024
13 checks passed

ulferts deleted the bug/57305-sorting-by-custom-field-has-strong-impact-on-performance-for-the-project-list branch November 15, 2024 08:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug/57305 sorting by custom field has strong impact on performance for the project list #16841

Bug/57305 sorting by custom field has strong impact on performance for the project list #16841

toy commented Sep 27, 2024 •

edited

Loading

toy Oct 10, 2024

ulferts left a comment

toy commented Oct 30, 2024

toy commented Nov 11, 2024 •

edited

Loading

toy commented Nov 13, 2024

ulferts left a comment

Bug/57305 sorting by custom field has strong impact on performance for the project list #16841

Bug/57305 sorting by custom field has strong impact on performance for the project list #16841

Conversation

toy commented Sep 27, 2024 • edited Loading

Ticket

What are you trying to accomplish?

Screenshots

What approach did you choose and why?

Merge checklist

toy Oct 10, 2024

Choose a reason for hiding this comment

ulferts left a comment

Choose a reason for hiding this comment

Grouping by integer cf

Grouping by link cf

toy commented Oct 30, 2024

toy commented Nov 11, 2024 • edited Loading

toy commented Nov 13, 2024

ulferts left a comment

Choose a reason for hiding this comment

toy commented Sep 27, 2024 •

edited

Loading

toy commented Nov 11, 2024 •

edited

Loading