Kafka integration #84
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
geoffreyaldebert
commented
Jun 10, 2022
geoffreyaldebert
commented
- Kafka Integration (only consumer)
- Read message from udata-analysis-service
- Parse file (could be from minio instead of downloading again resource)
- Add csv-detective type detection to help agate to store resource into sqlite
- Add pandas profiling analysis (minimal) and generation of json report
- Store new infos into sqlite in new tables :
- general_infos : basic info on resource
- column_infos : basic info on each column of resource
- categorical_infos : categorical values for each columns (limit to 10)
- top_infos : top values for each columns (limit to 10)
- numeric_infos : basic info on each numeric column of resource (mean, std, min, max)
- numeric_plot_infos : repartition of values of numeric column in a plot
- Update API to list those new info if we have them
…al, store to sqlite, modify api with new infos
maudetes
reviewed
Jul 18, 2022
| #url = r.json()['url'] | ||
| if((message is not None) & (message['service'] == 'csvdetective')): | ||
| #try: | ||
| url = 'https://www.data.gouv.fr/fr/datasets/r/{}'.format(key) |
There was a problem hiding this comment.
Why do we need to build an url instead of using the minio location?
This data.gouv.fr location url is environment dependent (dev / demo / prod).
* Switch to poetry - Explicitely upgrade to python >= 3.9 - Upgrade pandas and pandas-profiling * Cleanup - Overhaul CI file - Remove useless files - Update License attribution - Remove obsolete ansible roles * trigger CI * fix local tests on macos * add linting * add linting * upgrade flake8 and pytest * lint all the thingz * fix tests with strict asyncio mode * really really fix the tests * Update README * Use CI template, publish kafka-integration, bump 1.3.0 * poetry update * invalidate cache * CI: cache-prefix param
|
This branch is now published on pypi https://app.circleci.com/pipelines/github/etalab/csvapi/91/workflows/09dba6e2-b91f-4cf2-af03-71a9daee9bbb/jobs/605 |
* Check message structure to prevent errors * Add pandas profiling analysis (optional) in api * Update requirements (csv-detective) * Update message structure format * Add requirements * Remove requirements, switch to poetry * Add poetry lock file * Lint code * setuptools * upgrade and clean deps * lint test Co-authored-by: Geoffrey Aldebert <[email protected]> Co-authored-by: Alexandre Bulté <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.