Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GraphRAG with llama-index #793

Merged
merged 50 commits into from
Oct 30, 2024
Merged

GraphRAG with llama-index #793

merged 50 commits into from
Oct 30, 2024

Conversation

rbrugaro
Copy link
Collaborator

Description

GraphRAG components using llama-index:

comps/dataprep/neo4j/llama-index/extract_graph_neo4j.py
comps/retrievers/neo4l/llama-index/retriever_community_answers_neo4j.py

later i will add a GenAIExample that will combine these 2 with the DocSum to generate the final answers from the community summaries

rbrugaro and others added 14 commits October 15, 2024 13:53
@rbrugaro
graphRAG dataprep llama-index validated w openai endpoints
b0a95a3 
Signed-off-by: rbrugaro <[email protected]>
@rbrugaro
llama-index graphRAG retrieval validated with openai models
4e1db1f 
Signed-off-by: rbrugaro <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
ba1cffb 
for more information, see https://pre-commit.ci
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
dc97611 
for more information, see https://pre-commit.ci
@rbrugaro
validated code usng TGI/TEI instead of openai
d4a6459 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
compose.yaml for dataprep validated with neo4j, TGI/TEI, openai
012a503 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
fix merge conflict
a0668b7 
Signed-off-by: Rita Brugarolas <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
f345bb6 
for more information, see https://pre-commit.ci
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
0cf55a8 
for more information, see https://pre-commit.ci
@rbrugaro
graphRAG retriever validated and full compose.yaml
1ceb659 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
minor fix
8e034cd 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
Merge branch 'graphRAG_LI' of https://github.com/rbrugaro/GenAIComps …
3ca5298 
…into graphRAG_LI
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
9eee5cb 
for more information, see https://pre-commit.ci
@rbrugaro
add graphRAGGateway WIP
109a04e 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro rbrugaro mentioned this pull request Oct 22, 2024
Merged
ichbinblau
ichbinblau reviewed Oct 23, 2024
View reviewed changes
Copy link
Contributor

@ichbinblau ichbinblau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @rbrugaro Thanks for enabling Graphrag with llama-index. I found some inconsistent places in the commits during the test. PTAL.

comps/retrievers/neo4j/llama_index/README.md Outdated Show resolved Hide resolved
comps/retrievers/neo4j/llama_index/README.md Outdated Show resolved Hide resolved
comps/retrievers/neo4j/llama_index/README.md Outdated Show resolved Hide resolved
comps/retrievers/neo4j/llama_index/README.md Outdated Show resolved Hide resolved
comps/dataprep/neo4j/llama_index/README.md Outdated Show resolved Hide resolved
@rbrugaro
Copy link
Collaborator Author

@ichbinblau Thanks for catching this!, PR still in draft since i need to do some clean up and will take care of the READMEs

@codecov Codecov
Copy link

codecov bot commented Oct 23, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 9.52381% with 38 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
comps/cores/mega/gateway.py 7.31% 38 Missing ⚠️
Files with missing lines Coverage Δ
comps/cores/mega/constants.py 98.41% <100.00%> (+0.02%) ⬆️
comps/cores/mega/gateway.py 30.24% <7.31%> (-2.07%) ⬇️

... and 1 file with indirect coverage changes

@rbrugaro rbrugaro marked this pull request as ready for review October 23, 2024 21:19
@rbrugaro
rename test containers for cicd
c1d3361 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro rbrugaro mentioned this pull request Oct 28, 2024
Open
4 tasks
@rbrugaro
set 4 hpu for 70B model in ci test
be2b3e5 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
add extra time for large model loading cicd
d0ae033 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
fix tgi gaudi shard args
5791644 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
switch to chat cause chat template is needed
2501099 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
enable logs in test
e4c839e 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
Merge branch 'main' into graphRAG_LI
d752dd5
@rbrugaro
use locally downloaded model in CI machine
057dde2 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
Merge branch 'graphRAG_LI' of https://github.com/rbrugaro/GenAIComps …
a0eb001 
…into graphRAG_LI
@rbrugaro
use local model path and reduce wait time
847e132 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
clear ports before ci run
fee0735 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
fix cache model access
7acdfad 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
fix cache model access
f9ac6ba 
Signed-off-by: Rita Brugarolas <[email protected]>
@rbrugaro
incrased wait time for tgi shards ready
09d011f 
Signed-off-by: Rita Brugarolas <[email protected]>
ashahba
ashahba requested changes Oct 29, 2024
View reviewed changes
Copy link
Collaborator

@ashahba ashahba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rbrugaro for this PR!
This looks like a great PR and my comments and change requests are mostly BKMs for code consistency.

comps/cores/mega/gateway.py Show resolved Hide resolved
comps/cores/mega/gateway.py Show resolved Hide resolved
comps/cores/mega/gateway.py Show resolved Hide resolved
comps/dataprep/neo4j/llama_index/README.md Outdated Show resolved Hide resolved
comps/dataprep/neo4j/llama_index/README.md Outdated Show resolved Hide resolved
tests/dataprep/test_dataprep_neo4j_llama_index_on_intel_hpu.sh Outdated Show resolved Hide resolved
tests/dataprep/test_dataprep_neo4j_llama_index_on_intel_hpu.sh Outdated Show resolved Hide resolved
tests/dataprep/test_dataprep_neo4j_llama_index_on_intel_hpu.sh Outdated Show resolved Hide resolved
tests/retrievers/test_retrievers_neo4j_llama_index_on_intel_hpu.sh Outdated Show resolved Hide resolved
tests/retrievers/test_retrievers_neo4j_llama_index_on_intel_hpu.sh Outdated Show resolved Hide resolved
@rbrugaro rbrugaro requested a review from ashahba October 30, 2024 03:34
ashahba
ashahba approved these changes Oct 30, 2024
View reviewed changes
Copy link
Collaborator

@ashahba ashahba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ashahba ashahba requested review from lvliang-intel and removed request for lvliang-intel October 30, 2024 05:43
@ashahba ashahba merged commit 19330ea into opea-project:main Oct 30, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chensuyue chensuyue chensuyue left review comments

@ichbinblau ichbinblau ichbinblau left review comments

@XuhuiRen XuhuiRen XuhuiRen approved these changes

@ashahba ashahba ashahba approved these changes

@lvliang-intel lvliang-intel Awaiting requested review from lvliang-intel lvliang-intel is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@rbrugaro @endomorphosis @ichbinblau @ashahba @XuhuiRen @chensuyue