[Flaky Test] Fix flaky test in range aggregation yaml test

[bowenlan-amzn]

Description

The newly added yaml test Double range profiler shows filter rewrite info is flaky

It's a supprise to me one bulk request with 3 documents can index into 2 shards.

Based on the profile API result, we can see 2 shards, and the optimization algorithm related profiling entry "optimized_segments": 1 and "inner_visited": 1 shows both shards have documents.

profile API result

{
  "body": {
    "took": 38,
    "timed_out": false,
    "_shards": {
      "total": 2,
      "successful": 2,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": {
        "value": 3,
        "relation": "eq"
      },
      "max_score": null,
      "hits": []
    },
    "aggregations": {
      "double_range": {
        "buckets": [
          {
            "key": "*-50.0",
            "to": 50.0,
            "doc_count": 1
          },
          {
            "key": "50.0-150.0",
            "from": 50.0,
            "to": 150.0,
            "doc_count": 2
          },
          {
            "key": "150.0-*",
            "from": 150.0,
            "doc_count": 0
          }
        ]
      }
    },
    "profile": {
      "shards": [
        {
          "id": "[1c8HW1JTTFafkVnpxeWY4Q][test_profile][0]",
          "inbound_network_time_in_millis": 0,
          "outbound_network_time_in_millis": 0,
          "searches": [
            {
              "query": [
                {
                  "type": "ConstantScoreQuery",
                  "description": "ConstantScore(*:*)",
                  "time_in_nanos": 332625,
                  "breakdown": {
                    "set_min_competitive_score_count": 0,
                    "match_count": 0,
                    "shallow_advance_count": 0,
                    "next_doc": 0,
                    "score_count": 0,
                    "compute_max_score_count": 0,
                    "advance": 0,
                    "advance_count": 0,
                    "score": 0,
                    "shallow_advance": 0,
                    "create_weight_count": 1,
                    "build_scorer": 0,
                    "set_min_competitive_score": 0,
                    "match": 0,
                    "next_doc_count": 0,
                    "compute_max_score": 0,
                    "build_scorer_count": 0,
                    "create_weight": 332625
                  },
                  "children": [
                    {
                      "type": "MatchAllDocsQuery",
                      "description": "*:*",
                      "time_in_nanos": 212334,
                      "breakdown": {
                        "set_min_competitive_score_count": 0,
                        "match_count": 0,
                        "shallow_advance_count": 0,
                        "next_doc": 0,
                        "score_count": 0,
                        "compute_max_score_count": 0,
                        "advance": 0,
                        "advance_count": 0,
                        "score": 0,
                        "shallow_advance": 0,
                        "create_weight_count": 1,
                        "build_scorer": 0,
                        "set_min_competitive_score": 0,
                        "match": 0,
                        "next_doc_count": 0,
                        "compute_max_score": 0,
                        "build_scorer_count": 0,
                        "create_weight": 212334
                      }
                    }
                  ]
                }
              ],
              "rewrite_time": 12042,
              "collector": [
                {
                  "name": "MultiCollector",
                  "reason": "search_multi",
                  "time_in_nanos": 2047167,
                  "children": [
                    {
                      "name": "EarlyTerminatingCollector",
                      "reason": "search_count",
                      "time_in_nanos": 9292
                    },
                    {
                      "name": "ProfilingAggregator: [double_range]",
                      "reason": "aggregation",
                      "time_in_nanos": 2048874
                    }
                  ]
                }
              ]
            }
          ],
          "aggregations": [
            {
              "type": "RangeAggregator",
              "description": "double_range",
              "time_in_nanos": 2882125,
              "breakdown": {
                "reduce": 0,
                "build_aggregation_count": 1,
                "post_collection": 2708,
                "initialize_count": 1,
                "reduce_count": 0,
                "collect_count": 0,
                "post_collection_count": 1,
                "build_leaf_collector": 2029083,
                "build_aggregation": 838209,
                "build_leaf_collector_count": 1,
                "initialize": 12125,
                "collect": 0
              },
              "debug": {
                "optimized_segments": 1,
                "unoptimized_segments": 0,
                "inner_visited": 1,
                "leaf_visited": 0
              }
            }
          ]
        },
        {
          "id": "[1c8HW1JTTFafkVnpxeWY4Q][test_profile][1]",
          "inbound_network_time_in_millis": 0,
          "outbound_network_time_in_millis": 0,
          "searches": [
            {
              "query": [
                {
                  "type": "ConstantScoreQuery",
                  "description": "ConstantScore(*:*)",
                  "time_in_nanos": 340417,
                  "breakdown": {
                    "set_min_competitive_score_count": 0,
                    "match_count": 0,
                    "shallow_advance_count": 0,
                    "next_doc": 0,
                    "score_count": 0,
                    "compute_max_score_count": 0,
                    "advance": 0,
                    "advance_count": 0,
                    "score": 0,
                    "shallow_advance": 0,
                    "create_weight_count": 1,
                    "build_scorer": 0,
                    "set_min_competitive_score": 0,
                    "match": 0,
                    "next_doc_count": 0,
                    "compute_max_score": 0,
                    "build_scorer_count": 0,
                    "create_weight": 340417
                  },
                  "children": [
                    {
                      "type": "MatchAllDocsQuery",
                      "description": "*:*",
                      "time_in_nanos": 225292,
                      "breakdown": {
                        "set_min_competitive_score_count": 0,
                        "match_count": 0,
                        "shallow_advance_count": 0,
                        "next_doc": 0,
                        "score_count": 0,
                        "compute_max_score_count": 0,
                        "advance": 0,
                        "advance_count": 0,
                        "score": 0,
                        "shallow_advance": 0,
                        "create_weight_count": 1,
                        "build_scorer": 0,
                        "set_min_competitive_score": 0,
                        "match": 0,
                        "next_doc_count": 0,
                        "compute_max_score": 0,
                        "build_scorer_count": 0,
                        "create_weight": 225292
                      }
                    }
                  ]
                }
              ],
              "rewrite_time": 11292,
              "collector": [
                {
                  "name": "MultiCollector",
                  "reason": "search_multi",
                  "time_in_nanos": 2071709,
                  "children": [
                    {
                      "name": "EarlyTerminatingCollector",
                      "reason": "search_count",
                      "time_in_nanos": 16208
                    },
                    {
                      "name": "ProfilingAggregator: [double_range]",
                      "reason": "aggregation",
                      "time_in_nanos": 2066710
                    }
                  ]
                }
              ]
            }
          ],
          "aggregations": [
            {
              "type": "RangeAggregator",
              "description": "double_range",
              "time_in_nanos": 2876917,
              "breakdown": {
                "reduce": 0,
                "build_aggregation_count": 1,
                "post_collection": 542,
                "initialize_count": 1,
                "reduce_count": 0,
                "collect_count": 0,
                "post_collection_count": 1,
                "build_leaf_collector": 2045625,
                "build_aggregation": 818625,
                "build_leaf_collector_count": 1,
                "initialize": 12125,
                "collect": 0
              },
              "debug": {
                "optimized_segments": 1,
                "unoptimized_segments": 0,
                "inner_visited": 1,
                "leaf_visited": 0
              }
            }
          ]
        }
      ]
    }
  }
}

The Fix

Explicitly set primary shard number to 1, considering we already disable the auto refresh by setting refresh_interval to -1. I expect there will always be just 1 segment in 1 shard for this test. On the other hand, I'm also looking forward to another flakyness to learn new things :)

Related Issues

Resolves #13865 (comment)

Check List

• [ ] Functionality includes testing.
• [ ] API changes companion pull request created, if applicable.
• [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 1c807cf: SUCCESS

[andrross]

@bowenlan-amzn Does this test require that there be a single segment? If so, it might be worth adding a step to explicitly force merge the index to a single segment so that there are no hidden assumptions.

[bowenlan-amzn]

@bowenlan-amzn Does this test require that there be a single segment? If so, it might be worth adding a step to explicitly force merge the index to a single segment so that there are no hidden assumptions.

Yes, it's required to be 1 segment as the assertion expect that.

From learning perspective, I was hoping to see if this can still be flaky without the force merge.
But considering people may not know how to fix it and this is brought up here, I am adding the force merge now.

[github-actions]

❌ Gradle check result for 59f30dd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 86fd641: SUCCESS

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.76%. Comparing base (b15cb0c) to head (86fd641).
Report is 471 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14486      +/-   ##
============================================
+ Coverage     71.42%   71.76%   +0.33%     
- Complexity    59978    62181    +2203     
============================================
  Files          4985     5124     +139     
  Lines        282275   292250    +9975     
  Branches      40946    42234    +1288     
============================================
+ Hits         201603   209719    +8116     
- Misses        63999    65201    +1202     
- Partials      16673    17330     +657     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andrross]

From learning perspective, I was hoping to see if this can still be flaky without the force merge. But considering people may not know how to fix it and this is brought up here, I am adding the force merge now.

@bowenlan-amzn My understanding, which could be wrong, is that it is possible for separate segments to get created on a shard even for trivially small documents here. However, the "merge on refresh" policy would cause these tiny segments to get merged into one when the refresh happened, so in practice you would never see multiple segments even without the force merge. However, if your test actually depends on a single segment then it is better to make that assumption explicit.