Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to the latest OpenAPI generator #293

Open
tomplus opened this issue Nov 25, 2023 · 2 comments
Open

Upgrade to the latest OpenAPI generator #293

tomplus opened this issue Nov 25, 2023 · 2 comments
Assignees

Comments

@tomplus
Copy link
Owner

tomplus commented Nov 25, 2023

I've tried to use the latest version of OpenAPI generator to generate the library. That version uses Pydantic2 to generate models. Unfortunately kubernetes library contains ~600 models/api and loading time (at startup) is huge and not acceptable, especially for scripts, long running applications import modules once so it's not so painful. I tried to replace imports with lazy imports as a workaround but it's still 10 time worse in compare with current version using old generator.

My tests:

  1. kubectl, local cluster (as a reference), get pods (9) from all namespaces - 0.12 sec
  2. this library, current version, example1.py to list pods - 0.50 sec
  3. OpenAPIGenerator v 7.0.1, example1.py - 17.26 sec
  4. OpenAPIGenerator like above with some lazy imports (for models, apis) - 5.22 sec

The problem is known and may be resolved in the future: pydantic/pydantic#6748

For now I' m going to switch to the version just before Pydantic. It helps to resolve some pending issues, and this issue is for tracking ideas or attempts for future upgrades.

@tomplus tomplus self-assigned this Nov 25, 2023
@multani
Copy link
Contributor

multani commented Jan 2, 2024

@tomplus maybe it would be worth having a generator option to not load all the models from the __init__.py files (both client/__init__.py and client/models/__init__.py)

Typically, I'm using only a very small subset of all the available models, but importing kubernetes_asyncio always loads them all.

If we could reduce this default list to something smaller, and have to explicitly import the extra classes you need, I think it could help a lot.

@tomplus
Copy link
Owner Author

tomplus commented Jan 2, 2024

I did such tests using lazy imports (models were loaded only when needed) and it worked 3x faster but still 10x slower than the current version (see timings above). The problem is that each model requires other models and these models required others and still you need to load many of them...

@tomplus tomplus mentioned this issue Jan 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants