Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Design a way to more quickly write AWS sync plugins #349

Closed
achantavy opened this issue Jul 13, 2020 · 7 comments
Closed

Design a way to more quickly write AWS sync plugins #349

achantavy opened this issue Jul 13, 2020 · 7 comments
Assignees
@achantavy
Labels
keep fresh Disables stalebot from closing an issue long-term-improvement A better way of doing things

Comments

@achantavy
Copy link
Contributor

Design a way to more quickly write AWS sync plugins

Description:

Describe your idea. Please be detailed. If a feature request, please
describe the desired behavior, what scenario it enables, and how it
would be used.

Background

An intel module needs to perform a common set of steps:

  1. Get data with a certain set of fields from an external API

  2. Perform a set of data transformations such as generating unique IDs if vendor-provided ones (e.g. AWS ARNs) are not available.

  3. Load new nodes and relationships to the graph and connect them to all related nodes such as AWS accounts and others

  4. Make indexes for the new node types

  5. Create integration tests to ensure the nodes and connections are loaded as expected

  6. Update our schema documents

This is a lot of boilerplate code to write and there are many repeated sections across intel modules.

The idea

To improve our developer experience, we could create an abstraction layer that accepts as input what data types the developer wants to retrieve and their expected forms in the graph (steps 1-4). This will greatly improve our velocity on adding new plugins.

This issue will initially be scoped to AWS syncs and we will revisit this approach for the other modules.

The task

Write a specification document on what this abstraction interface could look like and submit requests for comment.
@achantavy achantavy self-assigned this Jul 13, 2020
@stale
Copy link

stale bot commented Jul 27, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the stale stalebot believes this issue/PR is no longer active label Jul 27, 2020
@stale
Copy link

stale bot commented Aug 21, 2020

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new change or reopen this one.

@stale stale bot closed this as completed Aug 21, 2020
@marco-lancini
Copy link
Contributor

@achantavy I still thinks this could add some value and speed-up writing new sync modules.
Thoughts?

@marco-lancini marco-lancini reopened this Aug 22, 2020
@stale stale bot removed the stale stalebot believes this issue/PR is no longer active label Aug 22, 2020
@achantavy
Copy link
Contributor Author

@marco-lancini 100%.

I have some ideas on how to do this and have opened up #395 as foundation for this work.

@stale
Copy link

stale bot commented Sep 5, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the stale stalebot believes this issue/PR is no longer active label Sep 5, 2020
@stale
Copy link

stale bot commented Sep 27, 2020

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new change or reopen this one.

@stale stale bot closed this as completed Sep 27, 2020
@marco-lancini marco-lancini added keep fresh Disables stalebot from closing an issue and removed stale stalebot believes this issue/PR is no longer active labels Oct 5, 2020
@marco-lancini marco-lancini reopened this Oct 5, 2020
@achantavy achantavy added the long-term-improvement A better way of doing things label Oct 14, 2020
@achantavy achantavy mentioned this issue Nov 28, 2022
Merged
achantavy added a commit that referenced this issue Dec 16, 2022
@achantavy @ramonpetgrave64
#349: Generalize graph write queries (#1038)
9179f59 
* Build ingest query

* Linter

* Save cleanup query for another PR

* Implement schema

* bump mypy to 0.981 for python/mypy#13398

* linter

* make load_graph_data interface make more sense

* fix comment

* Docs and some better names

* add a todo

* Doc updates, rename some fields

* Fix pre-commit

* Code commment suggestions

Co-authored-by: Ramon Petgrave <[email protected]>

* Stackoverflow comment for clarity)

* Support ingesting only parts of a schema without breaking the others

* Doc comment

* Linter

* Support matching on one or more properties

* Correctly name test

* Change key_refs to TargetNodeMatcher to enforce it as a mandatory field

* Remove use of hacky default_field()

* Support subset of schema relationships for query generation, test multiple node labels

* Docstrings

* Comments in tests

* Better comments

* Test for exception conditions

* Remove irrelevant comment

Co-authored-by: Ramon Petgrave <[email protected]>
@achantavy
Copy link
Contributor Author

Done! We will need to refactor existing code using #1038, but we have the capability now!

chandanchowdhury pushed a commit to juju4/cartography that referenced this issue Jun 26, 2024
@achantavy @ramonpetgrave64 @chandan-cl
lyft#349: Generalize graph write queries (lyft#1038)
1bddd61 
* Build ingest query

* Linter

* Save cleanup query for another PR

* Implement schema

* bump mypy to 0.981 for python/mypy#13398

* linter

* make load_graph_data interface make more sense

* fix comment

* Docs and some better names

* add a todo

* Doc updates, rename some fields

* Fix pre-commit

* Code commment suggestions

Co-authored-by: Ramon Petgrave <[email protected]>

* Stackoverflow comment for clarity)

* Support ingesting only parts of a schema without breaking the others

* Doc comment

* Linter

* Support matching on one or more properties

* Correctly name test

* Change key_refs to TargetNodeMatcher to enforce it as a mandatory field

* Remove use of hacky default_field()

* Support subset of schema relationships for query generation, test multiple node labels

* Docstrings

* Comments in tests

* Better comments

* Test for exception conditions

* Remove irrelevant comment

Co-authored-by: Ramon Petgrave <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@achantavy achantavy

Labels
keep fresh Disables stalebot from closing an issue long-term-improvement A better way of doing things
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@marco-lancini @achantavy