Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLI App Support #389

Merged
merged 20 commits into from
Sep 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 101 additions & 18 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -507,8 +507,7 @@ models. There are two primary use cases for Pydantic settings CLI:

By default, the experience is tailored towards use case #1 and builds on the foundations established in [parsing
environment variables](#parsing-environment-variable-values). If your use case primarily falls into #2, you will likely
want to enable [enforcing required arguments at the CLI](#enforce-required-arguments-at-cli) and [nested model default
partial updates](#nested-model-default-partial-updates).
want to enable most of the defaults outlined at the end of [creating CLI applications](#creating-cli-applications).

### The Basics

Expand Down Expand Up @@ -560,19 +559,7 @@ print(Settings().model_dump())
```

To enable CLI parsing, we simply set the `cli_parse_args` flag to a valid value, which retains similar conotations as
defined in `argparse`. Alternatively, we can also directly provide the args to parse at time of instantiation:

```py
from pydantic_settings import BaseSettings


class Settings(BaseSettings):
this_foo: str


print(Settings(_cli_parse_args=['--this_foo', 'is such a foo']).model_dump())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed suggestion of using private param _cli_parse_args. The "formal" equivalent is now:

CliApp.run(Settings, cli_args=['--this_foo', 'is such a foo']).model_dump()

#> {'this_foo': 'is such a foo'}
```
defined in `argparse`.

Note that a CLI settings source is [**the topmost source**](#field-value-priority) by default unless its [priority value
is customised](#customise-settings-sources):
Expand Down Expand Up @@ -875,6 +862,95 @@ sys.argv = ['example.py', 'gamma-cmd', '--opt-gamma=hi']
assert get_subcommand(Root()).model_dump() == {'opt_gamma': 'hi'}
```

### Creating CLI Applications

The `CliApp` class provides two utility methods, `CliApp.run` and `CliApp.run_subcommand`, that can be used to run a
Pydantic `BaseSettings`, `BaseModel`, or `pydantic.dataclasses.dataclass` as a CLI application. Primarily, the methods
provide structure for running `cli_cmd` methods associated with models.

`CliApp.run` can be used in directly providing the `cli_args` to be parsed, and will run the model `cli_cmd` method (if
defined) after instantiation:

```py
from pydantic_settings import BaseSettings, CliApp


class Settings(BaseSettings):
this_foo: str

def cli_cmd(self) -> None:
# Print the parsed data
print(self.model_dump())
#> {'this_foo': 'is such a foo'}

# Update the parsed data showing cli_cmd ran
self.this_foo = 'ran the foo cli cmd'


s = CliApp.run(Settings, cli_args=['--this_foo', 'is such a foo'])
print(s.model_dump())
#> {'this_foo': 'ran the foo cli cmd'}
```

Similarly, the `CliApp.run_subcommand` can be used in recursive fashion to run the `cli_cmd` method of a subcommand:
kschwab marked this conversation as resolved.
Show resolved Hide resolved
kschwab marked this conversation as resolved.
Show resolved Hide resolved

```py
from pydantic import BaseModel

from pydantic_settings import CliApp, CliPositionalArg, CliSubCommand


class Init(BaseModel):
directory: CliPositionalArg[str]

def cli_cmd(self) -> None:
print(f'git init "{self.directory}"')
#> git init "dir"
self.directory = 'ran the git init cli cmd'


class Clone(BaseModel):
repository: CliPositionalArg[str]
directory: CliPositionalArg[str]

def cli_cmd(self) -> None:
print(f'git clone from "{self.repository}" into "{self.directory}"')
self.directory = 'ran the clone cli cmd'


class Git(BaseModel):
clone: CliSubCommand[Clone]
init: CliSubCommand[Init]

def cli_cmd(self) -> None:
CliApp.run_subcommand(self)


cmd = CliApp.run(Git, cli_args=['init', 'dir'])
assert cmd.model_dump() == {
'clone': None,
'init': {'directory': 'ran the git init cli cmd'},
}
```

!!! note
Unlike `CliApp.run`, `CliApp.run_subcommand` requires the subcommand model to have a defined `cli_cmd` method.
kschwab marked this conversation as resolved.
Show resolved Hide resolved
kschwab marked this conversation as resolved.
Show resolved Hide resolved

For `BaseModel` and `pydantic.dataclasses.dataclass` types, `CliApp.run` will internally use the following
`BaseSettings` configuration defaults:

* `alias_generator=AliasGenerator(lambda s: s.replace('_', '-'))`
* `nested_model_default_partial_update=True`
* `case_sensitive=True`
* `cli_hide_none_type=True`
* `cli_avoid_json=True`
* `cli_enforce_required=True`
* `cli_implicit_flags=True`

!!! note
The alias generator for kebab case does not propagate to subcommands or submodels and will have to be manually set
kschwab marked this conversation as resolved.
Show resolved Hide resolved
kschwab marked this conversation as resolved.
Show resolved Hide resolved
in these cases.

### Customizing the CLI Experience

The below flags can be used to customise the CLI experience to your needs.
Expand Down Expand Up @@ -1241,7 +1317,7 @@ defined one that specifies the `root_parser` object.
import sys
from argparse import ArgumentParser

from pydantic_settings import BaseSettings, CliSettingsSource
from pydantic_settings import BaseSettings, CliApp, CliSettingsSource

parser = ArgumentParser()
parser.add_argument('--food', choices=['pear', 'kiwi', 'lime'])
Expand All @@ -1256,13 +1332,15 @@ cli_settings = CliSettingsSource(Settings, root_parser=parser)

# Parse and load CLI settings from the command line into the settings source.
sys.argv = ['example.py', '--food', 'kiwi', '--name', 'waldo']
print(Settings(_cli_settings_source=cli_settings(args=True)).model_dump())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed suggestions of using private param _cli_settings_source for its formal equivalent.

s = CliApp.run(Settings, cli_settings_source=cli_settings)
print(s.model_dump())
#> {'name': 'waldo'}

# Load CLI settings from pre-parsed arguments. i.e., the parsing occurs elsewhere and we
# just need to load the pre-parsed args into the settings source.
parsed_args = parser.parse_args(['--food', 'kiwi', '--name', 'ralph'])
print(Settings(_cli_settings_source=cli_settings(parsed_args=parsed_args)).model_dump())
s = CliApp.run(Settings, cli_args=parsed_args, cli_settings_source=cli_settings)
print(s.model_dump())
#> {'name': 'ralph'}
```

Expand All @@ -1281,6 +1359,11 @@ parser methods that can be customised, along with their argparse counterparts (t
For a non-argparse parser the parser methods can be set to `None` if not supported. The CLI settings will only raise an
error when connecting to the root parser if a parser method is necessary but set to `None`.

!!! note
The `formatter_class` is only applied to subcommands. The `CliSettingsSource` never touches or modifies any of the
kschwab marked this conversation as resolved.
Show resolved Hide resolved
kschwab marked this conversation as resolved.
Show resolved Hide resolved
external parser settings to avoid breaking changes. Since subcommands reside on their own internal parser trees, we
kschwab marked this conversation as resolved.
Show resolved Hide resolved
kschwab marked this conversation as resolved.
Show resolved Hide resolved
can safely apply the `formatter_class` settings without breaking the external parser logic.

## Secrets

Placing secret values in files is a common pattern to provide sensitive configuration to an application.
Expand Down
3 changes: 2 additions & 1 deletion pydantic_settings/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from .main import BaseSettings, SettingsConfigDict
from .main import BaseSettings, CliApp, SettingsConfigDict
from .sources import (
AzureKeyVaultSettingsSource,
CliExplicitFlag,
Expand All @@ -24,6 +24,7 @@
'BaseSettings',
'DotEnvSettingsSource',
'EnvSettingsSource',
'CliApp',
'CliSettingsSource',
'CliSubCommand',
'CliPositionalArg',
Expand Down
169 changes: 143 additions & 26 deletions pydantic_settings/main.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
from __future__ import annotations as _annotations

from typing import Any, ClassVar
from argparse import Namespace
from types import SimpleNamespace
from typing import Any, ClassVar, TypeVar

from pydantic import ConfigDict
from pydantic import AliasGenerator, ConfigDict
from pydantic._internal._config import config_keys
from pydantic._internal._utils import deep_update
from pydantic._internal._signature import _field_name_for_signature
from pydantic._internal._utils import deep_update, is_model_class
from pydantic.dataclasses import is_pydantic_dataclass
from pydantic.main import BaseModel

from .sources import (
Expand All @@ -17,9 +21,14 @@
InitSettingsSource,
PathType,
PydanticBaseSettingsSource,
PydanticModel,
SecretsSettingsSource,
SettingsError,
get_subcommand,
)

T = TypeVar('T')


class SettingsConfigDict(ConfigDict, total=False):
case_sensitive: bool
Expand All @@ -33,7 +42,6 @@ class SettingsConfigDict(ConfigDict, total=False):
env_parse_enums: bool | None
cli_prog_name: str | None
cli_parse_args: bool | list[str] | tuple[str, ...] | None
cli_settings_source: CliSettingsSource[Any] | None
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a bug. It's not possible to provide cli_settings_source as a configuration setting as the settings_cls cannot be defined (circular dependency).

cli_parse_none_str: str | None
cli_hide_none_type: bool
cli_avoid_json: bool
Expand Down Expand Up @@ -91,7 +99,8 @@ class BaseSettings(BaseModel):
All the below attributes can be set via `model_config`.

Args:
_case_sensitive: Whether environment variables names should be read with case-sensitivity. Defaults to `None`.
_case_sensitive: Whether environment and CLI variable names should be read with case-sensitivity.
Defaults to `None`.
_nested_model_default_partial_update: Whether to allow partial updates on nested model default object fields.
Defaults to `False`.
_env_prefix: Prefix for all environment variables. Defaults to `None`.
Expand Down Expand Up @@ -345,26 +354,24 @@ def _settings_build_values(
file_secret_settings=file_secret_settings,
) + (default_settings,)
if not any([source for source in sources if isinstance(source, CliSettingsSource)]):
if cli_parse_args is not None or cli_settings_source is not None:
cli_settings = (
CliSettingsSource(
self.__class__,
cli_prog_name=cli_prog_name,
cli_parse_args=cli_parse_args,
cli_parse_none_str=cli_parse_none_str,
cli_hide_none_type=cli_hide_none_type,
cli_avoid_json=cli_avoid_json,
cli_enforce_required=cli_enforce_required,
cli_use_class_docs_for_groups=cli_use_class_docs_for_groups,
cli_exit_on_error=cli_exit_on_error,
cli_prefix=cli_prefix,
cli_flag_prefix_char=cli_flag_prefix_char,
cli_implicit_flags=cli_implicit_flags,
cli_ignore_unknown_args=cli_ignore_unknown_args,
case_sensitive=case_sensitive,
)
if cli_settings_source is None
else cli_settings_source
if isinstance(cli_settings_source, CliSettingsSource):
sources = (cli_settings_source,) + sources
elif cli_parse_args is not None:
cli_settings = CliSettingsSource[Any](
self.__class__,
cli_prog_name=cli_prog_name,
cli_parse_args=cli_parse_args,
cli_parse_none_str=cli_parse_none_str,
cli_hide_none_type=cli_hide_none_type,
cli_avoid_json=cli_avoid_json,
cli_enforce_required=cli_enforce_required,
cli_use_class_docs_for_groups=cli_use_class_docs_for_groups,
cli_exit_on_error=cli_exit_on_error,
cli_prefix=cli_prefix,
cli_flag_prefix_char=cli_flag_prefix_char,
cli_implicit_flags=cli_implicit_flags,
cli_ignore_unknown_args=cli_ignore_unknown_args,
case_sensitive=case_sensitive,
)
sources = (cli_settings,) + sources
if sources:
Expand Down Expand Up @@ -401,7 +408,6 @@ def _settings_build_values(
env_parse_enums=None,
cli_prog_name=None,
cli_parse_args=None,
cli_settings_source=None,
cli_parse_none_str=None,
cli_hide_none_type=False,
cli_avoid_json=False,
Expand All @@ -420,3 +426,114 @@ def _settings_build_values(
secrets_dir=None,
protected_namespaces=('model_', 'settings_'),
)


class CliApp:
"""
A utility class for running Pydantic `BaseSettings`, `BaseModel`, or `pydantic.dataclasses.dataclass` as
CLI applications.
"""

@staticmethod
def _run_cli_cmd(model: Any, cli_cmd_method_name: str, is_required: bool) -> Any:
if hasattr(type(model), cli_cmd_method_name):
getattr(type(model), cli_cmd_method_name)(model)
elif is_required:
raise SettingsError(f'Error: {type(model).__name__} class is missing {cli_cmd_method_name} entrypoint')
return model

@staticmethod
def run(
model_cls: type[T],
cli_args: list[str] | Namespace | SimpleNamespace | dict[str, Any] | None = None,
cli_settings_source: CliSettingsSource[Any] | None = None,
cli_exit_on_error: bool | None = None,
cli_cmd_method_name: str = 'cli_cmd',
**model_init_data: Any,
) -> T:
"""
Runs a Pydantic `BaseSettings`, `BaseModel`, or `pydantic.dataclasses.dataclass` as a CLI application.
Running a model as a CLI application requires the `cli_cmd` method to be defined in the model class.

Args:
model_cls: The model class to run as a CLI application.
cli_args: The list of CLI arguments to parse. If `cli_settings_source` is specified, this may
also be a namespace or dictionary of pre-parsed CLI arguments. Defaults to `sys.argv[1:]`.
cli_settings_source: Override the default CLI settings source with a user defined instance.
Defaults to `None`.
cli_exit_on_error: Determines whether this function exits on error. If model is subclass of
`BaseSettings`, defaults to BaseSettings `cli_exit_on_error` value. Otherwise, defaults to
`True`.
cli_cmd_method_name: The CLI command method name to run. Defaults to "cli_cmd".
model_init_data: The model init data.

Returns:
The ran instance of model.

Raises:
SettingsError: If model_cls is not subclass of `BaseModel` or `pydantic.dataclasses.dataclass`.
SettingsError: If model_cls does not have a `cli_cmd` entrypoint defined.
"""

if not (is_pydantic_dataclass(model_cls) or is_model_class(model_cls)):
raise SettingsError(
f'Error: {model_cls.__name__} is not subclass of BaseModel or pydantic.dataclasses.dataclass'
)

cli_settings = None
cli_parse_args = True if cli_args is None else cli_args
if cli_settings_source is not None:
if isinstance(cli_parse_args, (Namespace, SimpleNamespace, dict)):
cli_settings = cli_settings_source(parsed_args=cli_parse_args)
else:
cli_settings = cli_settings_source(args=cli_parse_args)
elif isinstance(cli_parse_args, (Namespace, SimpleNamespace, dict)):
raise SettingsError('Error: `cli_args` must be list[str] or None when `cli_settings_source` is not used')

model_init_data['_cli_parse_args'] = cli_parse_args
model_init_data['_cli_exit_on_error'] = cli_exit_on_error
model_init_data['_cli_settings_source'] = cli_settings
if not issubclass(model_cls, BaseSettings):

class CliAppBaseSettings(BaseSettings, model_cls): # type: ignore
model_config = SettingsConfigDict(
alias_generator=AliasGenerator(lambda s: s.replace('_', '-')),
nested_model_default_partial_update=True,
case_sensitive=True,
cli_hide_none_type=True,
cli_avoid_json=True,
cli_enforce_required=True,
cli_implicit_flags=True,
)

model = CliAppBaseSettings(**model_init_data)
model_init_data = {}
for field_name, field_info in model.model_fields.items():
model_init_data[_field_name_for_signature(field_name, field_info)] = getattr(model, field_name)

return CliApp._run_cli_cmd(model_cls(**model_init_data), cli_cmd_method_name, is_required=False)

@staticmethod
def run_subcommand(
model: PydanticModel, cli_exit_on_error: bool | None = None, cli_cmd_method_name: str = 'cli_cmd'
) -> PydanticModel:
"""
Runs the model subcommand. Running a model subcommand requires the `cli_cmd` method to be defined in
the nested model subcommand class.

Args:
model: The model to run the subcommand from.
cli_exit_on_error: Determines whether this function exits with error if no subcommand is found.
Defaults to model_config `cli_exit_on_error` value if set. Otherwise, defaults to `True`.
cli_cmd_method_name: The CLI command method name to run. Defaults to "cli_cmd".

Returns:
The ran subcommand model.

Raises:
SystemExit: When no subcommand is found and cli_exit_on_error=`True` (the default).
SettingsError: When no subcommand is found and cli_exit_on_error=`False`.
"""

subcommand = get_subcommand(model, is_required=True, cli_exit_on_error=cli_exit_on_error)
return CliApp._run_cli_cmd(subcommand, cli_cmd_method_name, is_required=True)
Loading
Loading