Retrieve Bulk Job V2 error messages

[R0Wi]

Description

The new sf CLI data commands are using the Salesforce Bulk Job V2 API to upsert or delete records. This applies to

• sf data delete bulk
• sf data upsert bulk

Sometimes it happens that some datasets cannot be deleted or upserted, which results in a message, printed to the CLI like this:

Job XXX Status Job Complete Records processed 1. Records failed 1.

Unfortunately the CLI doesn't tell us what exactly went wrong. Also, since we're dealing with Bulk API V2, the results cannot be downloaded via Salesforce GUI. The only quite cumbersome solution is to use the REST API to check the results

Proposal

For the two aforementioned commands, it would be great to have some flag like --show-errors, which will download the errors after the action and either print them to the stdout or to some .csv-file. I think the current implementation is already downloading the job results via jsforce getAllResults() but the results are only used to print the summary shown above, not the details.

Current workaround

1. Use sf org display to get the Access Token
2. Figure out the JobId where records have failed
3. Use the REST API to query the CSV results which also contain the error messages coming from Salesforce

OR

Use the global --json flag. But this will print all results (succeeded, failed, not processed), which can be quite hard to read.

Contribution

If you're interested in this feature, I'd be glad to bring in a PR for this. We should just clarify the exact flag specification and which commands to be adjusted.

[github-actions]

Thank you for filing this feature request. We appreciate your feedback and will review the feature at our next grooming or sprint planning session. We prioritize feature requests with more upvotes and comments.

[git2gus]

This issue has been linked to a new work item: W-13599839

[WillieRuemmele]

thanks for filing this @R0Wi - sounds like a great enhancement

[R0Wi]

Thanks @WillieRuemmele . Would be great if we could discuss the CLI interface changes in detail, then I'd bring in a PR for this within the next weeks 👍

[WillieRuemmele]

@R0Wi - of course, we can definitely work on some designs. I bet they'd be pretty similar to what's printed when there's a failing component during a deploy, or a failing apex test like

Test Failures [1]
• FileUtilitiesTest.createFileSucceedsWhenCorrectInput
  message: System.AssertException: Assertion Failed: Expected: 1, Actual: 2
  stacktrace: 
    Class.FileUtilitiesTest.createFileSucceedsWhenCorrectInput: line 20, column 1

or

=== Component Failures [1]

 Type  Name              Problem                                                                                                                      
 ───── ───────────────── ─────────────────────────────────────────
 Error FileUtilitiesTest Method does not exist or inc...

I'm not sure what data will be available in that API response, but adding enough information to help someone figure out where the mistake is will be great. We can also add a --verbose flag (if it doesn't already exist) to show ALL of the data

[R0Wi]

Sounds good, thanks @WillieRuemmele for the quick feedback. Accoding to the docs the SF API will return the results directly in CSV format (this holds not only for the failed results but also for the success and "not processed" results).

So as a first improvement I could imagine using a --verbose-flag (which is not yet implemented for sf data delete bulk or sf data upsert bulk) and printing only the errors in a way like shown in your second example - in case there are some.

sf data delete bulk --help

USAGE
  $ sf data delete bulk -o <value> -f <value> -s <value> [--json] [--api-version <value>] [-w <value> | -a]

FLAGS
  -a, --async               Run the command asynchronously.
  -f, --file=<value>        (required) CSV file that contains the IDs of the records to delete.
  -o, --target-org=<value>  (required) [default: <some-email>] Org alias or username to use for the target org.
  -s, --sobject=<value>     (required) API name of the Salesforce object, either standard or custom, that you want to delete records from.
  -w, --wait=<value>        [default: 0 minutes] Number of minutes to wait for the command to complete before displaying the results.
  --api-version=<value>     Override the api version used for api requests made by this command

GLOBAL FLAGS
  --json  Format output as json.

Only disadvantage here is that the output cannot be further processed by any automation and it can fill your std-out quite a bit if you're dealing with a lot of records 😄

Later we could also think about writing back the results to a copy of the source CSV file, like SFDMU and Data Loader do it. So then the source CSV file would get two additional columns Id and Error and further processing would be possible.

[R0Wi]

@WillieRuemmele PR is here: salesforcecli/plugin-data#615. We might need to discuss some technical details there if needed 👍