Identify the action items for adapting SageMaker containers in Flyte

[bnsblue]

Related PRs:
https://github.com/lyft/flyteplayground/pull/61
flyteorg/flytekit#156

• Verify we can pass Env Vars to the SM Job when submitting it. (We can either set SAGEMAKER_PROGRAM env var to a generated script for the SM Task inside the image or to a generic flytekit-provided script that then reads another flyte- specific env var to know which python func to run)

• How do we reconcile Spark Entrypoint with this behavior
  Looks like this might be easy because SM No longer requires an entrypoint to be set.

• How do we reconcile FlyteKit commands with this behavior. Related: Implement pyflyte-exec-alternative as the alternative entrypoints for SageMaker Custom Training tasks #479

SM will invoke docker run <image> train

• flyte_sagemaker_runner.py as the entry point to pass through the pyflyte execute command

[bnsblue]

While there's no way to explicitly inject an env var to a SM job via SM's k8s operator, we can inject a variable to a standalone training job or a training job underlying an hpo job by implicitly injecting it via the hyperParameters field of the TrainingJob CRD or the trainingJobDefinition.staticHyperParameters field of the HyperparameterTuningJob CRD.

For example,

...
 trainingJobDefinition:
    staticHyperParameters:
      - name: __FLYTE_ENTRYPOINT_SELECTOR__
        value: "SAGEMAKER"
...

What SageMaker does is that it will put a summarized map of hyperparameters and values (which includes the variable you want to inject) to the path /opt/ml/input/config/hyperparameters.json inside your container, and their wrapper script parses that file and passes the hyperparameters to the user script as command-line arguments. See the following example log from SageMaker:

SM_HP___FLYTE_ENTRYPOINT_SELECTOR__=SAGEMAKER
...

Invoking script with the following command:
/usr/bin/python3 flyte_entrypoint_selector.py --__FLYTE_ENTRYPOINT_SELECTOR__ SAGEMAKER

So you only need to prepare your entrypoint_selector script in a way that it is ready takes the injected variable from the command line.