Refactor dev/dev-env-setup.sh to separate mandatory and optional setup

[xq-yin]

What changes are proposed in this pull request?

dev/dev-env-setup.sh is a one-stop script for setting up development environment for MLflow. This is the most convenient way to set up and we recommend it in the contribution guide.

However, as the script gets longer, the fact that it can only be executed all at once becomes a friction. The motivation of this PR is to refactor the script to make it resumable from where it errored out.

This first PR only contains the refactoring change, which divides the script into different sections, wrap them with functions, and group them by mandatory vs optional. The following PR will implement the checkpoints to actually save the progress.

This PR also touches the logic of the original script by removing https://github.com/mlflow/mlflow/blob/master/dev/dev-env-setup.sh#L213-L222 , reason being it only prints out the dependencies without actually installing them. https://github.com/mlflow/mlflow/blob/master/dev/dev-env-setup.sh#L263-L278 has already covered both the printout and installation.

How is this PR tested?

• Existing unit/integration tests
• New unit/integration tests
• Manual tests

Passed pytest tests/tracking/test_client.py tests/tracking/test_rest_tracking.py
Passed pre-commit run --all-files

Ran dev/dev-env-setup.sh -d .venvs/mlflow-dev -q without commenting out https://github.com/mlflow/mlflow/blob/master/requirements/doc-requirements.txt#L2 on my local macOS, which fails the mandatory setup
Verified the script exited on error

Ran dev/dev-env-setup.sh -d .venvs/mlflow-dev -q when

1. correctly commenting out https://github.com/mlflow/mlflow/blob/master/requirements/doc-requirements.txt#L2 on my local macOS -> all mandatory setup should pass AND
2. making docker installation fail (it's an optional setup)
   Verified the script can finish

and source .venvs/mlflow-dev/bin/activate can run successfully

Does this PR require documentation update?

• No. You can skip the rest of this section.
• Yes. I've updated:
  • Examples
  • API references
  • Instructions

Release Notes

Is this a user-facing change?

• No. You can skip the rest of this section.
• Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/deployments: MLflow Deployments client APIs, server, and third-party Deployments integrations
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

Language

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

Integrations

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

• rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
• rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
• rn/feature - A new user-facing feature worth mentioning in the release notes
• rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
• rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

• Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
  Bug fixes, doc updates and new features usually go into minor releases.
• Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
  Bug fixes and doc updates usually go into patch releases.

• Yes (this PR will be cherry-picked and included in the next patch release)
• No (this PR will be included in the next minor release)

[github-actions]

Documentation preview for 0443d25 will be available when this CircleCI job
completes successfully.

• Top page
• Changed pages

More info

• Ignore this comment if this PR does not change the documentation.
• It takes a few minutes for the preview to be available.
• The preview is updated when a new commit is pushed to this PR.
• This comment was created by https://github.com/mlflow/mlflow/actions/runs/9382322769.

[BenWilson2]

An attempt to answer Feedback 1:

When this was first written, the M1 Mac was still fairly new and there were some 'interesting behaviors' with some of the gcc dependencies that were being configured and setup for compatibility for some of the pyc integrations (namely TensorFlow, LightGBM, and XGBoost, IIRC) were, shall we say, "flaky".
The intention of the "keep going if you can" option was to get most of the non-dependent components installed in one fell swoop, thereafter M1 users could do some digging and acquire the correct dependencies, compile them, and install them in order for the old pyc dependencies to function correctly.
Since these errors were non-critical for >95% of MLflow development (not many devs touch the flavor implementations), the intention was to revisit things in this script once the Metal ecosystem matured a bit.

It has since then. :)

Changing this to short-circuit and abort with state checkpointing is definitely the right move. We shouldn't be silently eating failures any longer.

Feedback 2:

Critical path:
Python Environment (pyenv, virtualenv) installation
Python version validation
Brew installation, configuration, and update
MLflow dev version install (core)
test requirements, lint requirements, and doc requirements
git hooks pre commit tooling
DCO config for GitHub signoffs (the validation - this is a Linux Foundation requirement)

Optional path:
Docker
pandoc
The remaining diff of requirements between dev-requirements.txt and the above mentioned core dev requirements (lint, test, doc) - some of these can cause installation issues in a fresh environment and some require manual env setup (gcc compilers, c libraries that are no longer default installed in MacOS, XCode requirements, etc.)
the requirements for frontend dev (which this script doesn't handle, but potentially could in the future - out of scope for this PR, though :) )

Please let me know if there are other parts of this that need further explanation :) I'm excited to see the greatly improved script!

[B-Step62]

Overall LGTM, with a few nit comments. Thank you for this refactoring, @xq-yin 😄

[WeichenXu123]

LGTM!