Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce number of dependencies for taps vs targets #391

Closed
MeltyBot opened this issue May 23, 2022 · 3 comments
Closed

Reduce number of dependencies for taps vs targets #391

MeltyBot opened this issue May 23, 2022 · 3 comments

Comments

@MeltyBot
Copy link
Contributor

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/394

Originally created by @laurentS on 2022-05-23 10:34:58


Summary

The sdk depends on a number of libraries that are only used in the tap or target context. This adds weight to virtual envs, potential for dependency conflicts, security issues, etc...

While I don't have an immediate solution, it would make sense to work towards a way to split dependencies according to the use-case of the sdk.

Proposed benefits

  • smaller code footprint (weight, security, conflicts...)
  • faster venv installs -> happy devs 😄

Proposal details

The singer recommendation is to install each tap/target in its own virtual env to minimize potential for dependency conflicts, etc...
As an example, we run tap-github and tap-gitlab, each of which pulls in singer-sdk and transitively sqlalchemy (and its own dependencies). As these are taps, they make no use of sqlalchemy (and likely never will). sqlalchemy is interesting because it also requires a bunch of build dependencies as it contains C code that's built upon install.
I suspect there are similar dependencies that are not used by the target part of the sdk.

I'm wondering if it would be possible to break up the dependency list into something like deps-tap and deps-target so that non-development installations of the sdk fetch fewer dependencies.

Best reasons not to build

I'm conscious this is probably not trivial, and might lead to headaches by creating multiple "shades" of the sdk.

poetry has an extras mechanism, but I don't know if it can handle variants, unless dependencies are individually declared as extras, which may also make installation less straightforward.

@MeltyBot
Copy link
Contributor Author

@stale
Copy link

stale bot commented Jul 18, 2023

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

@edgarrmondragon
Copy link
Collaborator

Superseded by #2335

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants