Implement core dumps for notarized applications on macOS

[keithc-ca]

On macOS, core dumps are currently created by sending a signal to the process (see omrdump_create()). This works fine for an application built locally, but requires requesting the com.apple.security.get-task-allow entitlement during the code signing processes.

Unfortunately, Apple will normally refuse to notarize applications which have requested that entitlement. The qualification, 'normally' comes from a WWDC video on the topic, but the speaker did not elaborate.

So, it would seem, we either need to learn how to convince Apple to notarize applications requiring the com.apple.security.get-task-allow entitlement, or the production of core dumps on macOS cannot rely on the built-in signal-handler behavior.

The structure of a core file is reasonably well documented (OpenJ9 contains code that can read them). Implementing code here in OMR to write them explicitly should eliminate the need for a special entitlement and thus allow applications to be notarized.

[manqingl]

@keithc-ca @tajila @mikezhang1234567890 : if we are to implement our own mechanism to save memory content, do we know how feasible/reliable it would be? What would be the size of work?

Intuitively I assume Apple would not leave an easy backdoor for us if Apple decides to block it; and I assume Apple can block it unexpectedly in the future.

[keithc-ca]

It's not clear to me how Apple could block the creation of core files. The format is documented, and from the perspective of the OS, the application is simply writing a (large) file.

[manqingl]

@keithc-ca So we do not have to figure out which memories are for what, just have to write from address 0 to the end?

[keithc-ca]

We should be able to enumerate the regions of memory that need to be captured, using the same APIs as used by https://github.com/ePirat/gcore, but applied to the process itself rather than needing the entitlement required by using task_for_pid() to access a distinct process.

[keithc-ca]

This should be reopened: #6014 was reverted. @fjeremic can you do that, please?

[keithc-ca]

As @knn-k noted (eclipse-openj9/openj9#11164 (comment)), the solution needs to be extended for AArch64.