x-pack/filebeat/module/salesforce: Refactor a Salesforce module

[kush-elastic]

Description

This pull request aims to restructure the existing filebeat Salesforce module, transitioning it to utilize the new Salesforce input. The changes are subject to review once the new Salesforce input has been merged.

The current implementation of the Salesforce module employs httpjson and cometd inputs to capture historical and real-time data from Salesforce, respectively. Historical data is obtained from EventLogFiles, which stores data in files that can be retrieved from Salesforce using REST API and SOQL Queries. We use httpjson input to download and process this historical data. Real-time data is collected through real-time event monitoring, utilizing the pub-sub mechanism supported by Salesforce and cometd input can be used for the same.

During the collection of data from historical forms, we encountered challenges, with a significant issue being out-of-memory (OOM) issues. These errors could be reproduced with a large number of files from Salesforce, causing the httpjson input to utilize a significant amount of memory and eventually get killed by the kernel (oomk). To address this issue, we propose a solution by introducing a new Salesforce input. This new input is easily triggerable and capable of managing a large volume of data.

We also have plans to enhance the current Salesforce module to support more features as part of this improvement. Since most issues were closely tied to Salesforce, the decision was made to create a new input rather than repeatedly addressing problems with the existing httpjson input.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Configuration

  fileset-name:
    enabled: true
    var.api_version: 56

    var.authentication:
      jwt_bearer_flow:
        enabled: true
        client.id: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        client.username: "[email protected]"
        client.key_path: client_key.pem
        url: https://login.salesforce.com
      user_password_flow:
        enabled: true
        client.id: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        client.secret: client-secret
        token_url: "https://login.salesforce.com"
        user: "[email protected]"
        password: "P@$$W0₹D"

    var.url: "https://instance_id.my.salesforce.com"

    var.event_log_file: true
    var.elf_interval: 1h
    var.real_time: true
    var.real_time_interval: 5m

Salesforce Module Configuration Parameters:

• enabled:

  • Status: true
  • Description: Enables the Salesforce module fileset.

• var.api_version:

  • Version: 56
  • Description: Sets the Salesforce API version for interaction.

• var.authentication:

  • Parameters:
    • enabled : true
    • client.id: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    • client.username: "[email protected]"
    • client.key_path: client_key.pem
    • url: https://login.salesforce.com
  • Description: Enables JSON Web Token (JWT) authentication and specifies related details.

• Client Authentication Details:

  • Parameters:
    • enabled : true
    • client.id: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    • client.secret: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    • token_url: "https://login.salesforce.com"
    • username: "[email protected]"
    • password: "P@$$W0₹D"
  • Description: Provides client authentication for User Password Flow.

• var.url:

  • URL: "https://instance_id.my.salesforce.com"
  • Description: Specifies the base URL for the Salesforce instance.

• Event Log File Collection:

  • Enabled: true
  • Parameters:
    • var.elf_interval: 1 hour
  • Description: Enables the collection of event log files with a specified interval.

• Real-Time Event Monitoring:

  • Enabled: true
  • Parameters:
    • var.real_time_interval: 5 minutes
  • Description: Enables real-time event monitoring with a specified interval.

Related issues

• Relates https://github.com/elastic/obs-infraobs-team/issues/1248
• Closes https://github.com/elastic/obs-infraobs-team/issues/1277

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[belimawr]

LGTM.

I don't know much about Salesforce, but glancing over the PR it looks correct.

[shmsr]

Given the extensive size of the PR, reviewing all the changes thoroughly might pose a challenge. However, Kush and I have diligently gone through it, addressing numerous bugs along the way. @aliabbas-elastic and @niraj-elastic are helping in the multiple iterations of the review and testing process. Since this marks the initial alpha release, we have a window of opportunity to address any lingering bugs or implement enhancements.

cc: @lalit-satapathy Will require your approval to merge this.

[lalit-satapathy]

Approving to un-gate alpha release, per FF.

[ali786XI]

Approving on testing side. Refer here for the testing comments

[shmsr]

Approving on testing side. Refer here for the testing comments

Thanks @aliabbas-elastic!