Add biomedical entity normalization #3180
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t basic text and Ab3P pre-processing to the new structure; fix bug in Ab3P abbreviation detection
…m text and (2) entity / concept names from an knowledge base or ontology
- improve name consistency - make code more pythonic - dictionaries always do lazy loading - consistency in dictionary parsing: always yield (cui,name) - clean up loading w/ CONSTANTS (easily swap models) - allow access to sparse and dense search
sg-wbi
changed the title
added 2 commits
April 27, 2023 18:28
- yet better naming - add batched search - fix dicionary loading
- predict only on mentions of give entity type
sg-wbi
changed the title
mariosaenger
commented
May 8, 2023
- fix mypy typing - fix typos - update docstrings - rm faiss from requirements - better naming - allow user to specify annotation layer in predict - allow no mentions
sg-wbi
reviewed
May 11, 2023
mariosaenger
commented
May 15, 2023
| import faiss | ||
| except ImportError as error: | ||
| raise ImportError( | ||
| f"You need to install to run the biomedical entity linking: `pip faiss faiss-cpu=={FAISS_VERSION}`" |
There was a problem hiding this comment.
Typo: Install command should be "pip install faiss-cpu.." (instead of "pip faiss faiss-cpu").
Moreover, I would recommend to adjust the warning and refer to the GPU version of faiss too, i.e. add "pip install faiss-gpu..."
There was a problem hiding this comment.
I have removed the option to place the index on the GPU.
Large dictionaries require a lot of GPU RAM and unless we offer some compression it does not make too much sense.
We can leave it to as a next feature in a separate PR.
- better naming - unique cache name
- add option to time search - change error to warning if pre-trained model is not hybrid - check if there are mentions to predict
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements a named entity recognition model focussing on the biomedical domain.
The main contribution is a entity linking model which uses dense (transformer-based) embeddings and (optionally) sparse character-based representations, for normalizing an entity mention to specific identifiers in a knowledge base / dictionary. To this end, the model embeds the entity mention text and all concept names from the knowledge base and outputs the k best-matching concepts based on embedding similarity.