separate doc-level monitor query indices created by detectors #1324

sbcd90 · 2024-09-25T03:20:15Z

Description

separate doc-level monitor query indices created by detectors

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Subhobrata Dey <[email protected]>

eirsep · 2024-09-25T06:46:12Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

@@ -887,7 +887,7 @@ private IndexMonitorRequest createDocLevelMonitorMatchAllRequest(

        Monitor monitor = new Monitor(monitorId, Monitor.NO_VERSION, monitorName, false, detector.getSchedule(), detector.getLastUpdateTime(), null,
                Monitor.MonitorType.DOC_LEVEL_MONITOR.getValue(), detector.getUser(), 1, docLevelMonitorInputs, triggers, Map.of(),
-                new DataSources(detector.getRuleIndex(),
+                new DataSources(detector.getRuleIndex() + "_chained_findings",


why do we need separate index?

each doc-level monitor has its own query index now. So the chained findings doc-level monitor also has its own index. The Sigma Rules doc-level monitor & the bucket-level monitor's chained monitor run asynchronously & thus cannot share the same query index as it is created & deleted in every run.

eirsep · 2024-09-25T06:46:38Z

src/main/java/org/opensearch/securityanalytics/config/monitors/DetectorMonitorConfig.java

+    }
+
+    public static String getRuleIndexOptimized(String logType) {
+        return String.format(Locale.getDefault(), ".opensearch-sap-%s-detectors-queries-optimized-%s", logType, UUID.randomUUID());


we are moving from index patttern to index?

no. this method is utilized by few integ tests & hence the change.

eirsep · 2024-09-25T06:48:18Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

@@ -1369,7 +1369,11 @@ void onGetResponse(Detector currentDetector, User user) {
            request.getDetector().setAlertsHistoryIndexPattern(DetectorMonitorConfig.getAlertsHistoryIndexPattern(ruleTopic));
            request.getDetector().setFindingsIndex(DetectorMonitorConfig.getFindingsIndex(ruleTopic));
            request.getDetector().setFindingsIndexPattern(DetectorMonitorConfig.getFindingsIndexPattern(ruleTopic));
-            request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndex(ruleTopic));
+            if (currentDetector.getRuleIndex().contains("optimized")) {


if name contains optimized shouldn't we set rule index name as getRuleIndexOptimized value?

is this if else condition flipped?

the if-else condition checks if the detector is already using a dedicated query index, then keep on using it. else, switch from shared query index to new dedicated query index as the detector is updated.

eirsep · 2024-09-25T06:49:40Z

src/test/java/org/opensearch/securityanalytics/DetectorThreatIntelIT.java

-                "     }\n" +
-                "   }\n" +
-                "}";
-        SearchResponse response = executeSearchAndGetResponse(DetectorMonitorConfig.getRuleIndex(randomDetectorType()), request, true);


why has this been removed ?

Plz dont reduce test coverage..
fix test as necessary

query indices are created & deleted now. So, it is difficult to assert number of queries in query index now.

eirsep

what happens when someone updates a detector that doesnt have exclusive query index?

eirsep · 2024-09-25T07:05:06Z

src/test/java/org/opensearch/securityanalytics/alerts/AlertsIT.java

@@ -825,21 +825,10 @@ public void testMultipleAggregationAndDocRules_alertSuccess() throws IOException

        Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

-
-        String request = "{\n" +


plz revert and fix test as necessary

query indices are created & deleted now. So, it is difficult to assert number of queries in query index now.

eirsep · 2024-09-25T07:05:32Z

src/test/java/org/opensearch/securityanalytics/findings/FindingIT.java

@@ -1209,21 +1209,11 @@ public void testCreateDetectorWithNotCondition_verifyFindingsAndNoFindings_succe

        Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

-        String request = "{\n" +


query indices are created & deleted now. So, it is difficult to assert number of queries in query index now.

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 · 2024-09-25T11:18:59Z

what happens when someone updates a detector that doesnt have exclusive query index?

The detector is updated to use exclusive query index.

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 · 2024-09-26T00:49:25Z

security tests fail as docker image not updated.

eirsep · 2024-09-26T01:16:55Z

looks like we create aliases not indices from 'queryIndex` field in monitor datasources

eirsep · 2024-09-26T01:24:59Z

can we add tests for scenarios

setting is off. create 2 detectors. verify query index name. turn on setting. update 1 detector, verify query index name.
setting is on. create 1 detector. verify query index name. turn off setting. update detector, verify query index name. create new detector and verify query index(should be old style since setting is off)

eirsep · 2024-09-26T00:52:29Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

+                request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndexOptimized(ruleTopic));
+            } else {
+                String ruleTopicIndex = DetectorMonitorConfig.getRuleIndex(ruleTopic);
+                request.getDetector().setRuleIndex(ruleTopicIndex.substring(0, ruleTopicIndex.length()-1));


why did we change the existing code from
request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndex(ruleTopic));

ruleIndex is changing to dedicated ruleIndexOptimzied now. Thats why if-else condition.

eirsep · 2024-09-26T01:36:04Z

src/main/java/org/opensearch/securityanalytics/config/monitors/DetectorMonitorConfig.java

@@ -22,7 +24,11 @@ public class DetectorMonitorConfig {
    public static final String OPENSEARCH_SAP_RULE_INDEX_TEMPLATE = ".opensearch-sap-detectors-queries-index-template";

    public static String getRuleIndex(String logType) {
-        return String.format(Locale.getDefault(), ".opensearch-sap-%s-detectors-queries", logType);
+        return String.format(Locale.getDefault(), ".opensearch-sap-%s-detectors-queries*", logType);


why *
this is an alias

eirsep · 2024-09-26T01:37:21Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

@@ -1369,7 +1379,16 @@ void onGetResponse(Detector currentDetector, User user) {
            request.getDetector().setAlertsHistoryIndexPattern(DetectorMonitorConfig.getAlertsHistoryIndexPattern(ruleTopic));
            request.getDetector().setFindingsIndex(DetectorMonitorConfig.getFindingsIndex(ruleTopic));
            request.getDetector().setFindingsIndexPattern(DetectorMonitorConfig.getFindingsIndexPattern(ruleTopic));
-            request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndex(ruleTopic));
+            if (currentDetector.getRuleIndex().contains("optimized")) {
+                request.getDetector().setRuleIndex(currentDetector.getRuleIndex());


plz add comment that if we turn OFF setting after turning on, updating detector will not change from optimized to normal and we need to re-create detector

added a comment. added a test too.

eirsep · 2024-09-26T01:37:38Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

+                    request.getDetector().setRuleIndex(DetectorMonitorConfig.getRuleIndexOptimized(ruleTopic));
+                } else {
+                    String ruleTopicIndex = DetectorMonitorConfig.getRuleIndex(ruleTopic);
+                    request.getDetector().setRuleIndex(ruleTopicIndex.substring(0, ruleTopicIndex.length() - 1));


this line is very confusing

eirsep · 2024-09-26T01:38:15Z

src/test/java/org/opensearch/securityanalytics/SecurityAnalyticsRestTestCase.java

@@ -266,6 +266,7 @@ void setDebugLogLevel() throws IOException, InterruptedException {


        makeRequest(client(), "PUT", "_cluster/settings", Collections.emptyMap(), se, new BasicHeader("Content-Type", "application/json"));
+        updateClusterSetting("plugins.security_analytics.enable_detectors_with_dedicated_query_indices", "true");


are there tests with setting turned off

added a test for setting turned off.

eirsep · 2024-09-26T01:39:10Z

src/test/java/org/opensearch/securityanalytics/mapper/MapperRestApiIT.java

@@ -1722,12 +1725,16 @@ public void testAzureMappings() throws IOException {
        DetectorInput input = new DetectorInput("windows detector for security analytics", List.of(indexName), List.of(),
                getPrePackagedRules("azure").stream().map(DetectorRule::new).collect(Collectors.toList()));
        Detector detector = randomDetectorWithInputs(List.of(input), "azure");
-        createDetector(detector);
-
-        List<SearchHit> hits = executeSearch(".opensearch-sap-azure-detectors-queries-000001", matchAllSearchBody);


can you retain this test and add a new one - basically test for both setting turned. on and turned off

added scenario for flag turned off in test cases. updated this test case to fetch queries from index pattern. https://github.com/sbcd90/security-analytics/blob/query_idx/src/test/java/org/opensearch/securityanalytics/mapper/MapperRestApiIT.java#L1727

eirsep · 2024-09-26T01:40:02Z

src/test/java/org/opensearch/securityanalytics/resthandler/DetectorMonitorRestApiIT.java

@@ -1104,21 +1094,11 @@ public void testCreateDetector_verifyWorkflowCreation_success_WithGroupByRulesIn

        Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

-        String request = "{\n" +


we should not remove this code. we lose code coverage of tests. that's an anti-pattern. let's fix these tests

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 · 2024-09-26T08:57:48Z

can we add tests for scenarios

setting is off. create 2 detectors. verify query index name. turn on setting. update 1 detector, verify query index name. setting is on. create 1 detector. verify query index name. turn off setting. update detector, verify query index name. create new detector and verify query index(should be old style since setting is off)

added both test cases.

1st scenario: https://github.com/sbcd90/security-analytics/blob/c514a780d4b3f65ac828e3ff868c56f46f2a82b7/src/test/java/org/opensearch/securityanalytics/resthandler/DetectorRestApiIT.java#L1729
2nd scenario: https://github.com/sbcd90/security-analytics/blob/c514a780d4b3f65ac828e3ff868c56f46f2a82b7/src/test/java/org/opensearch/securityanalytics/resthandler/DetectorRestApiIT.java#L1831

eirsep · 2024-09-26T09:06:46Z

src/main/java/org/opensearch/securityanalytics/util/RuleTopicIndices.java

@@ -83,7 +83,8 @@ private void getAllRuleIndices(ActionListener<List<String>> listener) {
            listener.onResponse(
                    logTypes
                        .stream()
-                        .map(logType -> DetectorMonitorConfig.getRuleIndex(logType))
+                            // use index pattern here to define rule topic index template for all query indices which match the pattern
+                        .map(logType -> DetectorMonitorConfig.getRuleIndex(logType) + "*")


shouldn't this be detector specific?
why do we have this method? getAllRuleIndices

a single index template containing specific settings for sigma rule executions are applied to all rule indices.

eirsep · 2024-09-26T16:50:00Z

CI is failing

sbcd90 · 2024-09-26T17:50:56Z

CI is failing

ci passed now.

eirsep · 2024-09-26T19:45:37Z

Security tests failed

Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit 1ca4090) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

opensearch-trigger-bot · 2024-09-26T20:49:33Z

The backport to 2.15 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.15 2.15
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.15
# Create a new branch
git switch --create backport-1324-to-2.15
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1ca40908ac7b050cc7eb90df3108ab8a21a1f6d1
# Push it to GitHub
git push --set-upstream origin backport-1324-to-2.15
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.15

Then, create a pull request where the base branch is 2.15 and the compare/head branch is backport-1324-to-2.15.

opensearch-trigger-bot · 2024-09-26T20:49:34Z

The backport to 2.16 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security-analytics/backport-2.16 2.16
# Navigate to the new working tree
pushd ../.worktrees/security-analytics/backport-2.16
# Create a new branch
git switch --create backport-1324-to-2.16
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1ca40908ac7b050cc7eb90df3108ab8a21a1f6d1
# Push it to GitHub
git push --set-upstream origin backport-1324-to-2.16
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security-analytics/backport-2.16

Then, create a pull request where the base branch is 2.16 and the compare/head branch is backport-1324-to-2.16.

…arch-project#1324) Signed-off-by: Subhobrata Dey <[email protected]>

…#1330) Signed-off-by: Subhobrata Dey <[email protected]>

…#1330) Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit 038d60a)

…arch-project#1324) Signed-off-by: Subhobrata Dey <[email protected]>

…#1344) Signed-off-by: Subhobrata Dey <[email protected]>

…#1330) (#1343) Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit 038d60a) Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

separate doc-level monitor query indices created by detectors

8e6ac76

Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 requested review from amsiglan, AWSHurneyt, getsaurabh02, lezzago, praveensameneni, eirsep, jowg-amazon, engechas, goyamegh and riysaxen-amzn as code owners September 25, 2024 03:20

eirsep reviewed Sep 25, 2024

View reviewed changes

eirsep requested changes Sep 25, 2024

View reviewed changes

eirsep reviewed Sep 25, 2024

View reviewed changes

separate doc-level monitor query indices created by detectors

b3284e7

Signed-off-by: Subhobrata Dey <[email protected]>

finalize changes

1f3eb29

Signed-off-by: Subhobrata Dey <[email protected]>

eirsep requested changes Sep 26, 2024

View reviewed changes

address review comments

c514a78

Signed-off-by: Subhobrata Dey <[email protected]>

eirsep reviewed Sep 26, 2024

View reviewed changes

eirsep approved these changes Sep 26, 2024

View reviewed changes

AWSHurneyt approved these changes Sep 26, 2024

View reviewed changes

sbcd90 added backport 2.x backport 2.15 backport 2.16 labels Sep 26, 2024

sbcd90 merged commit 1ca4090 into opensearch-project:main Sep 26, 2024
13 of 19 checks passed

opensearch-trigger-bot bot mentioned this pull request Sep 26, 2024

[Backport 2.x] separate doc-level monitor query indices created by detectors #1330

Merged

opensearch-trigger-bot bot added the failed backport label Sep 26, 2024

sbcd90 added a commit to sbcd90/security-analytics that referenced this pull request Sep 26, 2024

separate doc-level monitor query indices created by detectors (opense…

521a424

…arch-project#1324) Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 pushed a commit that referenced this pull request Sep 30, 2024

separate doc-level monitor query indices created by detectors (#1324) (…

038d60a

…#1330) Signed-off-by: Subhobrata Dey <[email protected]>

opensearch-trigger-bot bot added a commit that referenced this pull request Oct 7, 2024

separate doc-level monitor query indices created by detectors (#1324) (…

89e03e2

…#1330) Signed-off-by: Subhobrata Dey <[email protected]> (cherry picked from commit 038d60a)

sbcd90 added a commit to sbcd90/security-analytics that referenced this pull request Oct 7, 2024

separate doc-level monitor query indices created by detectors (opense…

0ca18a9

…arch-project#1324) Signed-off-by: Subhobrata Dey <[email protected]>

sbcd90 added a commit that referenced this pull request Oct 7, 2024

separate doc-level monitor query indices created by detectors (#1324) (…

b003ebb

…#1344) Signed-off-by: Subhobrata Dey <[email protected]>

		@@ -825,21 +825,10 @@ public void testMultipleAggregationAndDocRules_alertSuccess() throws IOException

		Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));


		String request = "{\n" +

		@@ -1209,21 +1209,11 @@ public void testCreateDetectorWithNotCondition_verifyFindingsAndNoFindings_succe

		Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

		String request = "{\n" +

		@@ -266,6 +266,7 @@ void setDebugLogLevel() throws IOException, InterruptedException {


		makeRequest(client(), "PUT", "_cluster/settings", Collections.emptyMap(), se, new BasicHeader("Content-Type", "application/json"));
		updateClusterSetting("plugins.security_analytics.enable_detectors_with_dedicated_query_indices", "true");

		@@ -1104,21 +1094,11 @@ public void testCreateDetector_verifyWorkflowCreation_success_WithGroupByRulesIn

		Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector));

		String request = "{\n" +

separate doc-level monitor query indices created by detectors #1324

separate doc-level monitor query indices created by detectors #1324

Conversation

sbcd90 commented Sep 25, 2024

Description

Related Issues

Check List

Choose a reason for hiding this comment

sbcd90 Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbcd90 commented Sep 25, 2024

sbcd90 commented Sep 26, 2024

eirsep commented Sep 26, 2024

eirsep commented Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbcd90 Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbcd90 commented Sep 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep commented Sep 26, 2024

sbcd90 commented Sep 26, 2024

eirsep commented Sep 26, 2024

opensearch-trigger-bot bot commented Sep 26, 2024

opensearch-trigger-bot bot commented Sep 26, 2024

sbcd90 Sep 25, 2024 •

edited

Loading

eirsep Sep 25, 2024 •

edited

Loading

eirsep commented Sep 26, 2024 •

edited

Loading

sbcd90 Sep 26, 2024 •

edited

Loading