docs: add end2end example on creating a basic text-classification dataset #4208

plaguss · 2023-11-13T11:12:05Z

Description

This PR includes 2 features towards the #4178 issue.

A tutorial for the creation of a FeedbackDataset for text-classification.
A new script has been added to run the notebooks automatically, via end2end.yml workflow.

Closes #4179 and #4220

Type of change

(Remember to title the PR according to the type of change)

Documentation update

How Has This Been Tested

(Please describe the tests that you ran to verify your changes.)

sphinx-autobuild (read Developer Documentation for more details)

Checklist

I added relevant documentation
I followed the style guidelines of this project
I did a self-review of my code
I made corresponding changes to the documentation
My changes generate no new warnings
I filled out the contributor form (see text above)
I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

…t classification

…2end-text-classification

plaguss · 2023-11-15T13:29:44Z

Example running locally (with elasticsearch and argilla quickstart images):

argilla on  docs/end2end-text-classification [!?] via 🐍 v3.10.13 (.venv) on ☁️  (us-east-1) 
❯ python scripts/end2end_examples.py --api-key admin.apikey
Executing: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 46/46 [00:19<00:00,  2.33cell/s]
✅  text-classification-create-dataset
Removed output notebook: output-notebook
Removed output folder: output_notebooks

And an example forcing the process to fail with an error in a cell:

❯ python scripts/end2end_examples.py
/home/agustin/github_repos/argilla-io/argilla/.venv/lib/python3.10/site-packages/nbformat/__init__.py:93: MissingIDFieldWarning: Code cell is missing an id field, this will become a hard error in future nbformat versions. You may want to use `normalize()` on your notebooks before validations (available since nbformat 5.1.4). Previous versions of nbformat are fixing this issue transparently, and will stop doing so in the future.
  validate(nb)
Executing:  75%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                        | 3/4 [00:01<00:00,  1.86cell/s]
❌  test_notebook
Traceback (most recent call last):
  File "/home/agustin/github_repos/argilla-io/argilla/scripts/end2end_examples.py", line 65, in <module>
    main()
  File "/home/agustin/github_repos/argilla-io/argilla/scripts/end2end_examples.py", line 56, in main
    example.run()
  File "/home/agustin/github_repos/argilla-io/argilla/scripts/end2end_examples.py", line 39, in run
    raise e from None
  File "/home/agustin/github_repos/argilla-io/argilla/scripts/end2end_examples.py", line 35, in run
    papermill.execute_notebook(str(self.src_filename), str(self.dst_filename), parameters=self.parameters)
  File "/home/agustin/github_repos/argilla-io/argilla/.venv/lib/python3.10/site-packages/papermill/execute.py", line 134, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/home/agustin/github_repos/argilla-io/argilla/.venv/lib/python3.10/site-packages/papermill/execute.py", line 241, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError: 
---------------------------------------------------------------------------
Exception encountered at "In [3]":
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 assert 1==2

AssertionError:

scripts/end2end_examples.py

davidberenstein1957 · 2023-11-15T13:58:51Z

.github/workflows/end2end-examples.yml

+        run: |
+          echo "ARGILLA_SEARCH_ENGINE=opensearch" >> "$GITHUB_ENV"
+          echo "Configure opensearch engine"
+      - name: Run end2end examples 📈


Because of running this every time, do you think we can filter ut a but and only run it when there are changes to src or examples.py? Also, perhaps we can use a subset of the datasets and/or setup a persistent cache for the datasets and set this equal to our "Cache pip 👜" step?

This might be relevant in other places we download 'datasets' for our cache.

Will update that

davidberenstein1957 · 2023-11-15T14:00:40Z

scripts/end2end_examples.py

+        "hf_token": hf_token,
+    }
+
+    examples = [


perhaps it is better to do this with glob and select everything in our folder as examples?

I wasn't sure whether we could add other files here, but I think that's better

@davidberenstein1957 I think we would like to have these run in a specific order. For that should we name them using something that can be sorted? (maybe just a number at the end works fine)

…files

plaguss · 2023-11-16T09:33:32Z

@gabrielmbmb I requested your review to know your opinion on the workflow, but feel free to skip the remaining content of the PR

…ore running

for more information, see https://pre-commit.ci

dvsrepo · 2023-11-16T10:21:48Z

@plaguss @davidberenstein1957 if the plan is to run this as part of CI/CD or periodically or for testing/QA purposes please make sure we DON'T track any telemetry as this will affect our understanding of real usage/errors, etc.

…/argilla-io/argilla into docs/end2end-text-classification

plaguss · 2023-11-16T10:33:33Z

@plaguss @davidberenstein1957 if the plan is to run this as part of CI/CD or periodically or for testing/QA purposes please make sure we DON'T track any telemetry as this will affect our understanding of real usage/errors, etc.

Sure @dvsrepo, that should be taken into account in the workflow:

        env:
          ARGILLA_ENABLE_TELEMETRY: 0
        run: |
          pip install -e .
          pip install papermill
          python scripts/end2end_examples.py

dvsrepo · 2023-11-16T10:55:01Z

@plaguss @davidberenstein1957 if the plan is to run this as part of CI/CD or periodically or for testing/QA purposes please make sure we DON'T track any telemetry as this will affect our understanding of real usage/errors, etc.

Sure @dvsrepo, that should be taken into account in the workflow:
        env:
          ARGILLA_ENABLE_TELEMETRY: 0
        run: |
          pip install -e .
          pip install papermill
          python scripts/end2end_examples.py

perfect @plaguss !

…docs/end2end-text-classification

github-actions · 2023-11-16T17:24:24Z

The URL of the deployed environment for this PR is https://argilla-quickstart-pr-4208-ki24f765kq-no.a.run.app

…ion dataset (#4342)  # Description Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change. Closes #4184 **Type of change** (Remember to title the PR according to the type of change) - [ ] Documentation update **How Has This Been Tested** (Please describe the tests that you ran to verify your changes.) - [x ] `sphinx-autobuild` (read [Developer Documentation](https://docs.argilla.io/en/latest/community/developer_docs.html#building-the-documentation) for more details) **Checklist** - [ ] I added relevant documentation - [ ] I followed the style guidelines of this project - [ ] I did a self-review of my code - [ ] I made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK) (see text above) - [ ] I have added relevant notes to the `CHANGELOG.md` file (See https://keepachangelog.com/)

# Description Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change. Closes #4187 **Type of change** (Remember to title the PR according to the type of change) - [ ] Documentation update **How Has This Been Tested** (Please describe the tests that you ran to verify your changes.) - [ ] `sphinx-autobuild` (read [Developer Documentation](https://docs.argilla.io/en/latest/community/developer_docs.html#building-the-documentation) for more details) **Checklist** - [ ] I added relevant documentation - [ ] I followed the style guidelines of this project - [ ] I did a self-review of my code - [ ] I made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK) (see text above) - [ ] I have added relevant notes to the `CHANGELOG.md` file (See https://keepachangelog.com/)

…tion dataset (#4350)  # Description Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change. Closes #4185 **Type of change** (Remember to title the PR according to the type of change) - [ x] Documentation update **How Has This Been Tested** (Please describe the tests that you ran to verify your changes.) - [ x] `sphinx-autobuild` (read [Developer Documentation](https://docs.argilla.io/en/latest/community/developer_docs.html#building-the-documentation) for more details) **Checklist** - [ ] I added relevant documentation - [ ] I followed the style guidelines of this project - [ ] I did a self-review of my code - [ ] I made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK) (see text above) - [ ] I have added relevant notes to the `CHANGELOG.md` file (See https://keepachangelog.com/)

docs: changed to spacy training instead of trf

* develop: (41 commits) chore: update dev version chore: update CHANGELOG.md before release v1.20.0 (#4357) docs: temporal update to indicate persistent storage (#4355) docs: add suggestions and responses filters and sorting (#4345) docs: add end2end example on creating a basic text-classification dataset (#4208) Fix/responses suggestions filter fine tune (#4356) Fix/responses suggestions filter fine tune (#4356) fix: Accept draft responses on dataset records creation (#4354) Feature/responses operator (#4352) Feature/responses operator (#4352) chore: increase dev version release to 1.21.0 chore: remove dev suffix for release branch fix: responses and suggestions filter QA (#4337) feat: delete suggestion from record on search engine (#4336) feat: update suggestion from record on search engine (#4339) bug: fix bug and update test (#4341) fix: preserve `TextClassificationSettings.label_schema` order (#4332) Update issue templates feat: 🚀 support for filtering and sorting by responses and suggestions (#4160) fix: handling errors for non-existing endpoints (#4325) ... # Conflicts: # frontend/v1/domain/entities/question/Question.ts # frontend/v1/domain/entities/record/Record.ts

* develop: (21 commits) ✨ Fix error handling in axios plugin for 401 (#4362) docs: Change `telemetry` section in tutorials to directly executable cells (#4399) docs: add faq files (#4363) fix: pinning `pytest-asyncio` to version `0.21.1` to avoid problems running unit tests on GitHub workflows (#4395) docs: add making most of markdown to tutorial page (#4376) Fixing typo in Fine Tuning LLMs Practical Guides (#4392) Token Classification epochs parameter trainer changed (#4393) docs: align practical guidescreate datasethtml with end2end examples structure (#4375) docs: hugging face space url (#4379) docs: extend using proxy section (#4368) chore: update dev version chore: update CHANGELOG.md before release v1.20.0 (#4357) docs: temporal update to indicate persistent storage (#4355) docs: add suggestions and responses filters and sorting (#4345) docs: add end2end example on creating a basic text-classification dataset (#4208) Fix/responses suggestions filter fine tune (#4356) Fix/responses suggestions filter fine tune (#4356) fix: Accept draft responses on dataset records creation (#4354) Feature/responses operator (#4352) Feature/responses operator (#4352) ...

plaguss added 7 commits November 13, 2023 12:09

docs: first tutorial for end2end examples on feedback dataset for tex…

4f9bbb6

…t classification

Merge branch 'develop' of github.com:argilla-io/argilla into docs/end…

a663698

…2end-text-classification

docs: update with task templates and push to argilla hf hub

c8681ba

feat: parametrize notebook to run via papermill

eb35dd3

feat: script to run end2end notebooks

212752c

refactor: grab apikey based on quickstart image

66c55cc

feat: initial version of end2end examples from ci/cd

fea8b11

plaguss requested review from frascuchon and davidberenstein1957 November 15, 2023 13:30

davidberenstein1957 reviewed Nov 15, 2023

View reviewed changes

plaguss and others added 2 commits November 15, 2023 19:09

feat: apply comments from code review

fd0ba5b

feat: update workflow to run depending on the result from check-repo-…

3130fdf

…files

plaguss requested a review from gabrielmbmb November 16, 2023 09:32

Agustin Piqueres added 2 commits November 16, 2023 11:14

feat: update naming convention for examples to allow sorting them bef…

5a676fb

…ore running

refactor: update glob call and sort notebooks before running

a6fc67a

plaguss force-pushed the docs/end2end-text-classification branch from 7be9cd5 to a6fc67a Compare November 16, 2023 10:15

[pre-commit.ci] auto fixes from pre-commit.com hooks

6398947

for more information, see https://pre-commit.ci

Agustin Piqueres added 2 commits November 16, 2023 11:21

chore: remove deleted file

bb5c580

Merge branch 'docs/end2end-text-classification' of https://github.com…

a9c893a

…/argilla-io/argilla into docs/end2end-text-classification

Agustin Piqueres added 3 commits November 16, 2023 17:34

Merge branch 'develop' of https://github.com/argilla-io/argilla into …

3ed3992

…docs/end2end-text-classification

feat: add telemetry for the tutorial

b4f61cb

refactor: set python3 as the default kernel for running the notebooks

6acfe0c

plaguss marked this pull request as ready for review November 16, 2023 17:01

tests: updated en2end workflow

72c3ddf

sdiazlor and others added 6 commits November 29, 2023 09:11

docs: updated references

0f6d36d

docs: added reference to parpermill description

2ebd343

docs: updated tabs

c8739a3

docs: marked cells as parameters

94436ec

docs: assign records

1ae526c

davidberenstein1957 force-pushed the docs/end2end-text-classification branch from 18815fd to 1ae526c Compare November 29, 2023 10:47

davidberenstein1957 and others added 18 commits November 29, 2023 11:50

chore: add utils module

e92e80c

docs: assign records

6bc5fff

Merge branch 'develop' into docs/end2end-text-classification

63aa5d8

chore: updated cell id

c1d3f49

chore: added kernelspec info

0279821

chore: updated add vectors

a520efa

docs: updated suggestions and responses

01c5d97

docs: updated token

9050330

docs: added updated to end2end training

bfcc207

chore: review notebook validator

a5cbedb

docs: updated end2end to running examples

b2da325

docs: updated tutorials and images

dedadae

docs: updated message used

18a62ee

docs: added reference to dataset

1a298a9

docs: changed to spacy training instead of trf

docs: push with local model instead

dfd9bad

docs: updated practical guides image

89c0c50

davidberenstein1957 merged commit cf7f67a into develop Nov 29, 2023
2 of 3 checks passed

davidberenstein1957 deleted the docs/end2end-text-classification branch November 29, 2023 16:16

davidberenstein1957 mentioned this pull request Nov 30, 2023

[DOCS] add visualisation to docs w.r.t. Argilla structure #3609

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add end2end example on creating a basic text-classification dataset #4208

docs: add end2end example on creating a basic text-classification dataset #4208

plaguss commented Nov 13, 2023 •

edited

Loading

plaguss commented Nov 15, 2023 •

edited

Loading

davidberenstein1957 Nov 15, 2023

davidberenstein1957 Nov 15, 2023

plaguss Nov 15, 2023

davidberenstein1957 Nov 15, 2023

plaguss Nov 15, 2023

plaguss Nov 16, 2023

plaguss commented Nov 16, 2023

dvsrepo commented Nov 16, 2023

plaguss commented Nov 16, 2023

dvsrepo commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

docs: add end2end example on creating a basic text-classification dataset #4208

docs: add end2end example on creating a basic text-classification dataset #4208

Conversation

plaguss commented Nov 13, 2023 • edited Loading

Description

plaguss commented Nov 15, 2023 • edited Loading

davidberenstein1957 Nov 15, 2023

Choose a reason for hiding this comment

davidberenstein1957 Nov 15, 2023

Choose a reason for hiding this comment

plaguss Nov 15, 2023

Choose a reason for hiding this comment

davidberenstein1957 Nov 15, 2023

Choose a reason for hiding this comment

plaguss Nov 15, 2023

Choose a reason for hiding this comment

plaguss Nov 16, 2023

Choose a reason for hiding this comment

plaguss commented Nov 16, 2023

dvsrepo commented Nov 16, 2023

plaguss commented Nov 16, 2023

dvsrepo commented Nov 16, 2023

github-actions bot commented Nov 16, 2023

plaguss commented Nov 13, 2023 •

edited

Loading

plaguss commented Nov 15, 2023 •

edited

Loading