Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Create a new endpoint supporting bulk annotation #4367

Closed
Tracked by #4050
frascuchon opened this issue Nov 30, 2023 · 1 comment · Fixed by #4380
Closed
Tracked by #4050

[FEATURE] Create a new endpoint supporting bulk annotation #4367

frascuchon opened this issue Nov 30, 2023 · 1 comment · Fixed by #4380
Assignees
Labels
area: api Indicates that an issue or pull request is related to the Fast API server or REST endpoints team: backend Indicates that the issue or pull request is owned by the backend team type: enhancement Indicates new feature requests
Milestone

Comments

@frascuchon
Copy link
Member

No description provided.

@frascuchon frascuchon added this to the v1.21.0 milestone Nov 30, 2023
@dosubot dosubot bot added area: api Indicates that an issue or pull request is related to the Fast API server or REST endpoints team: backend Indicates that the issue or pull request is owned by the backend team type: enhancement Indicates new feature requests labels Nov 30, 2023
Copy link

dosubot bot commented Nov 30, 2023

🤖

Hello @frascuchon! 👋

I'm Dosu, a friendly bot here to assist you with bugs, answer your queries, and guide you on becoming a contributor. While we wait for a human maintainer, feel free to ask me anything about the Argilla repository, I'm here to help!

Thank you for your feature request. Based on the current structure of the Argilla codebase, it seems feasible to create a new endpoint for supporting bulk annotation.

Currently, the handling of annotations is done through the AnnotationProgress model, which tracks the progress of individual annotations. To support bulk annotation, we would need to modify this model or create a new one that can handle multiple annotations at once.

However, without information on the current API structure and the endpoint used for creating a single annotation, it's challenging to provide a detailed plan on how to implement this feature. The implementation would largely depend on these details.

For instance, if the current API uses REST, the new endpoint could be something like POST /annotations/bulk, which accepts an array of annotations in the request body. On the other hand, if the API uses GraphQL, we would need to create a new mutation that accepts a list of annotations.

In both cases, the server-side logic would need to be updated to process multiple annotations at once. This could involve iterating over the array of annotations and creating/updating each one in the database. Error handling would also need to be considered, such as what to do if some annotations are successfully processed but others are not.

Here's a rough example of what the server-side code could look like:

@app.route('/annotations/bulk', methods=['POST'])
def create_bulk_annotations():
    annotations = request.json
    results = []
    for annotation in annotations:
        try:
            # This is a placeholder for the actual logic to create an annotation
            result = create_annotation(annotation)
            results.append(result)
        except Exception as e:
            # Handle the error appropriately
            pass
    return jsonify(results), 200

Please note that this is a very simplified example and the actual implementation would likely be more complex.

If you could provide more details about the current API structure and the endpoint used for creating a single annotation, I could give a more accurate and detailed plan for this feature. Dosu

Sources


This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

jfcalvo added a commit that referenced this issue Dec 14, 2023
# Description

This PR adds a new endpoint to create responses in bulk to support the
new bulk annotation feature on Argilla.

The new endpoint is `POST /api/v1/me/responses/bulk` with the following
implementations details:
* It can accept a maximum of `50` responses to be created or updated.
* Responses are created or updated for the current user doing the
request.
* The endpoint is not transactional.
* One response failing to be created or updated is not avoiding to
process the rest of the responses in the request.
* If the request is successful the number and order of the items in the
request body and response will match.

Example of a body request with a response to update and a second
response to update (but using an invalid question):

```json
{
  "items": [
    {
      "values": {"prompt-quality": {"value": 10}},
      "status": "submitted",
      "record_id": "8df12fc4-aaa4-4c5c-a883-1b50ab877c97"
    },
    {
      "values": {"non-existent-question": {"value": 1}},
      "status": "draft",
      "record_id": "f62786d4-eba7-4ecb-8225-730509150668"
    }
  ]
}
```

> [!NOTE]
> For updating responses we don't need the `id` of the response, from
the context of the request we have the current `user_id` and the
`record_id` is mandatory using this endpoint. These two attributes are
the ones used to unambiguously know when to create or update a response.

A response for the previous request with the first response successfully
updated and an error trying to create the second response:

```json
{
  "items": [
    {
      "item": {
        "id": "e27cdcde-1393-4bfb-a42a-d557a1975b35",
        "values": {"prompt-quality": {"value": 10}},
        "status": "submitted",
        "user_id": "3063e7c6-5b2a-4952-83b1-8e05277e8b51",
        "record_id": "8df12fc4-aaa4-4c5c-a883-1b50ab877c97",
        "inserted_at": "2023-12-05T12:39:19.165603",
        "updated_at": "2023-12-05T12:50:26.593117",
      },
      "error": null
    },
    {
      "item": null,
      "error": {
        "detail": "found responses for non configured questions: ['non-existent-question']"
      }
    }
  ]
}
```

/cc @frascuchon @gabrielmbmb <del>there are some `TODO` comments on the
code to discuss possible improvements.</del>
/cc @damianpumar 

Closes #4367 

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [x] New feature (non-breaking change which adds functionality)

**How Has This Been Tested**

- [x] Running tests locally using SQLite as database.
- [x] Running tests locally using PostgreSQL as database.

**Checklist**

- [ ] I added relevant documentation
- [ ] follows the style guidelines of this project
- [x] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [x] I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)
jfcalvo added a commit that referenced this issue Dec 18, 2023
# Generic Pull Request Template

This PRs include a small change increasing the maximum number of items
that can be part of a bulk responses request from `50` to `100`.

Records pagination is expected to be `100` as maximum so it makes sense
that the endpoint for bulk annotation/responses also allow to get a
maximum of `100` responses to create/update.

Refs #4367 

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (change restructuring the codebase without changing
functionality)
- [ ] Improvement (change adding some improvement to an existing
functionality)
- [ ] Documentation update

**How Has This Been Tested**

- [x] Running tests locally.

**Checklist**

- [ ] I added relevant documentation
- [x] follows the style guidelines of this project
- [x] I did a self-review of my code
- [ ] I made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK)
(see text above)
- [ ] <del>I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)</del>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: api Indicates that an issue or pull request is related to the Fast API server or REST endpoints team: backend Indicates that the issue or pull request is owned by the backend team type: enhancement Indicates new feature requests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants