…elop

…elop

…elop

…elop

…elop

- Add relationship to User and Workspace models.
- Minor fixes.

…elop

…ecognai/rubrix into feat/add-basic-database-support

Changes in tests to integrate with the new database system.

Also, correct the `workspaces` ORM -> View mapping.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

- Remove `email` and `password_reset_token` from `users` table
migration.
- Set `first_name` from `users` table as non nullable.
- Modify `User` Pydantic model mapping `full_name` and `workspaces`
using SQLAlchemy `User` model.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

- Add new `POST /api/users` endpoint to create users.
- Set `api_key` field as mandatory on Pydantic User model.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Improve how database sessions are used in context and endpoints:
- Using a new `get_db` function yielding a `SessionLocal` instance and
ensuring the session is closed.
- All endpoints that require it now depends on `get_db`.
- Accounts context functions now receive a `db` session argument.
- We have improved how `SessionLocal` is used on seeds functions.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

This PR adds a new API endpoint to list all the users present in the
database.

It doesn't includes pagination. We need to discuss if it's something
that we need right now, maybe it can be implemented later.

Add new endpoint to delete users.
- The endpoint returns a 404 error when the provided `user_id` is not
found.
- I have found a problem with `UUID` type not being correctly serialized
to JSON for some reason. We need to find an alternative solution to this
(more info in the code).

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…elop

This PR includes the first iteration for the new workspaces API.

The included endpoints allow to:
- `GET /api/workspaces`: list workspaces.
- `POST /api/workspaces`: create a new workspace.
- `DELETE /api/workspaces/:workspace_id`: delete a workspace.
- `GET /api/workspaces/:workspace_id/users`: list users members of the
workspace.
- `POST /api/workspaces/:workspace_id/users/:user_id`: add a new user as
member of the workspace.
- `DELETE /api/workspace/:workspace_id/users/:user_id`: remove/delete a
user as member of the workspace.

It also includes changes to `users_workspaces` table migration adding on
delete cascade. With this change once a record on `users` or
`workspaces` table is deleted we are deleting the associated
`users_workspaces` record too.

There are some missing things that are not included and we will fix
later:
- When deleting workspaces we are not checking if they have associated
datasets or not.
- Pagination (we need to discuss the response format for this).
- Some missing errors that are not managed yet (for example conflict
error when an user is already member of a workspace and you are trying
to create it).
- Timestamps (`inserted_at`, `udpdated_at` columns).
- Tests (we want to do some improvements to our testing suite first).

@frascuchon tell me if you want to discuss some of these missing points.

…elop

This PR include the following changes:
* Add `inserted_at` and `updated_at` columns to all database table
migrations.
* Add `inserted_at` and `updated_at` fields to database models and
Pydantic schemas.

Missing things:
* It was not possible to add `inserted_at`, and `updated_at` fields to
`User` Pydantic schema due to some errors in old code. We can fix this
later.

Gotchas:
* In order to have the same exact values for `inserted_at` and
`updated_at` when a record is inserted I have created a
`default_inserted_at` function that get the context of the operation to
extract the value of `inserted_at` and use it on `update_at`.
* I can see microseconds level date times inserted on SQLite so I guess
we are covered on that sense (maybe we can check this once we test the
changes with PostgreSQL).

…elop

…database (#2395)

In this PR we are making changes on how tests are configured to be used
with the database:
* Moved test seeds to a new `test_seeds` function at
`argilla.server.seeds` package.
* One the new `test_seeds` function we are defining a default "argilla"
user. In the future `test_seeds` should only have proper seeds necessary
for the application to work normally and users and other stuff should be
added as test fixtures.
* We didn't find a proper way to use transactions and rollback changes
in every unit test and allow endpoints access the same data so we are
truncating all the database tables after every unit test execution.
* We added a test for one of the new workspaces endpoint that interact
with the database so we can check that everything is working as
expected.
* Now our GitHub actions run (alembic) database migrations in the tests
section.

…oints (#2397)

This PR add explicit ordering by db insertion time (using `inserted_at`
column) to endpoints that are returning collections from database. Right
now those are for users and workspaces.

This explicit order avoid leaving the order to the database so it is
consistent between SQLite and PostgreSQL or other databases, and always
returning the same result.

This PR add an unique constraint to column `name` on `workspaces` table.

This restriction is necessary right now because we are using workspaces
name inside elastic-search indexes and it's not possible to have
duplications. In the future we will have information about elastic
indexes in the database (using ids) so we can rethink is the uniqueness
is necessary.

I have added a missing `mapped_column(unique=True)` for `username` field
on `User` model too.

…ction (#2401)

A small fix for a wrong method name used when user is not found on
`authenticate_user`.

Waiting for changes on test settings to be merged so we can write some
new tests for this one.

The pydantic `User` class contains id, and timestamp (`inserted_at` and
`updated_at` attributes).

Also, old user components (service, dao,...) are removed, and all
affected tests removed/adapted.

---------

Co-authored-by: José Francisco Calvo <[email protected]>

…elop

This PR add tests for new endpoints and also:
- Adding [`factory-boy`](https://factoryboy.readthedocs.io/en/stable/)
as development dependency.
- Adding factories: `UserFactory`, `WorkspaceFactory`, and
`UserWorkspaceFactory`.
- Fixing relationship warnings for `UserWorkspace` SQLAlchemy model.

Additionally some of the tests are marked as pending for two different
reasons:
- SQLite is by default [not enforcing referencial integrity with foreign
keys](https://docs.sqlalchemy.org/en/20/dialects/sqlite.html#foreign-key-support),
we have different options here:
1. Enforce foreign key support executing `PRAGMA` statement on
connection.
  2. Add additional checks to the endpoints where necessary.
- Pydantic is by default accepting empty strings as valid values for
fields that are not optional. This is causing for example that create an
user with an empty string as `first_name` is totally valid.
- We need to check if it's necessary to add a constraint to our models
or an alternative solution (it is a dangerous default behavior).

We have discussed these points and we will solve them as follows:
1. We will enforce SQLite foreign keys integrity using a `PRAGMA`
statement in every connection.
    - The changes are included in this PR.
- Even with this I have added some additional checks in some endpoints
to give more information to the user when necessary.
2. For Pydantic models accepting empty strings for non-optional string
fields:
- We will create a new PR with a solution for this, adding more
constraints of changing this Pydantic default behavior.
3. We will change `users_workspaces` table name and models to
`workspaces_users`, ' WorkspaceUser`, etc. :
     - The changes will be done in a different PR.
     - This has sense because the most important entity is workspace.

This PR adds `___repr___` implementation for all database models so we
can easily debug on console.

In this PR I'm redefining the order of the entities in
`users_workspaces` table to `workspaces_users`. This will improve the
semantics of our endpoints and make it more clear.

I have changed too the associated ORM models and Pydantic schemas to
reflect this change.

Everything else should remain the same, behavior is not changing.

We found that Pydantic schema string fields allow empty strings by
default, being the field optional or not.

More info about that here:
pydantic/pydantic#1626

With this behavior we can finish with empty strings used in fields like
`user.first_name`, `user.last_name` and others. Something that can have
a bad impact in our application.

I have explored different solutions, like defining a common validation
or a custom field type. At the end I think the most simple solution is
to be explicit and define the constraint that we want for the fields. In
this case `min_length=1` when necessary.

Additional changes:
* Added new max length constraint for `user.password` field.
* Notice how I have changed `constr` for `str = Field(...)`. If I'm not
wrong looks like they are similar but `Field` is used to generate a
validation response when errors are found.

Still failing 4 tests. All failed tests are related to workspace
operations.

```bash
FAILED tests/client/test_api.py::test_dataset_copy_to_another_workspace - argilla.client.sdk.commons.errors.NotFoundApiError: Argilla server returned an error with http status: 404

FAILED tests/functional_tests/test_log_for_text_classification.py::test_log_data_in_several_workspaces - Exception: Wrong provided workspace my-fun-workspace
FAILED tests/functional_tests/datasets/test_delete_records_from_datasets.py::test_delete_records_without_permission - Exception: Wrong provided workspace argilla

FAILED tests/server/datasets/test_api.py::test_delete_records - assert 404 == 403
```


Running tests locally I found an error with DB rollback when a test is
failing. I've stopped using current seeds for tests (adding users).
Those users are created as a fixture where they are required. This
change affects to some users tests

…elop

Failing tests were trying to launch actions using a workspace linked to
the `argilla` user. Now, those tests will create this link explicitly.

…ngs (#2447)

With this PR we are making the necessary changes to create database
connections using a new `ARGILLA_DATABASE_URL` environment variable for
SQLAlchemy and Alembic migrations.

Considerations:
- Right now the value of `ARGILLA_DATABASE_URL` is get on
`server/database.py` file. Maybe we should think into moving it to some
settings file.
- A new `postgresql` section has been added to `pyproject.toml`. We
should add something to the documentation about installing dependencies
with `pip` for this section if the user wants to use PostgreSQL.
- I have moved the connection argument `check_same_thread=False` to the
default `ARGILLA_DATABASE_URL` value. If a users wants to use a custom
SQLite URL the same connection argument should be used (maybe we need to
add something about this to the documentation).
- API endpoint tests are passing locally using PostgreSQL.

## How to run the application using PostgreSQL

Installing PostgreSQL and creating `argilla` database:

```sh
$ brew install postgresql
$ createdb argilla
```

Installing Python PostgreSQL dependencies and run migrations:

```sh
$ pip install -e .[postgresql]
$ ARGILLA_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/argilla alembic upgrade head
```

Running tests

```sh
$ ARGILLA_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/argilla pytest tests
```

Simply add a new accounts context function to get workspaces by name.

We remove extra things from the `User` pydantic model.

- The `email` attribute
- Extra not needed validators, like username or workspace patterns,
since they are already done in the `UserCreate` and `WorkspaceCreate`
classes

In a separate PR, workspace validations will be moved to a proper
validation module/class used for that purpose.

This PR adds the following changes:
- Add a new string `role` column to `users` table. The column is non
nullable and indexed.
- Add `UserRole` class as enumerable. The class inherits from `Enum` and
`str` (to avoid some JSON encoding problems, maybe we can create a
custom JSON encoder to avoid possible problems).
- Add `role` as enumerable field to `User` SQLAlchemy model with
`annotator` as default.
- Add `role` as optional for `UserCreate` Pydantic schema.

Changes has been tested locally and are working with PostgreSQL too.

closes #2049

Remove the extra `workspaces` field and the unused `disabled` field.

This PR adds `ARGILLA_HOME_PATH` new environment variable with a default
value equal to `~/.argilla`.

After that `ARGILLA_HOME_PATH` is used to define the default value for
`ARGILLA_DATABASE_URL`.

I have added these changes to `_const.py` file but maybe we should
create a `settings.py` file or something similar.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

This PR adds a new users migration task that will allow to move users
defined at a YAML file to database.

After checking that this is working as expected and it's what we really
want we can made the necessary changes to our documentation (maybe in
this current PR).

## Details about the migration process:

- It's executed inside a database transaction so an error in one
specific user should rollback all the changes.
- I have added some Pydantic basic schemas to validate the content from
the YAML file. Please check if some additional checks or validations can
be added.
- The code is capturing the exception when the file is not found.
Knowing that this is a task to be manually executed, maybe we should not
capture the exception and let it crash.

## How to execute the migration process:

```sh
$ ARGILLA_LOCAL_AUTH_USERS_DB_FILE=users.yml python -m argilla.tasks.users_migrator
```

…und (#2481)

With this PR we will add a default `argilla` user (with `argilla`) when
there are not users on the database.

We also show a warning message indicating that it could be dangerous to
have default `argilla` user in a production environment and what to do
in that case.

In the case that we already have users on the database (because of the
migration) we check if the `argilla` default user is using default
credentials (password and api key) and in that case we show the same
previous warning.

This PR adds a new task to run alembic database migration in the same
way that we do with `alembic upgrade head` but using an argilla task.

To execute the task you need to do:

```sh
$ python -m argilla.tasks.database.migrate
```

This PR removes the creation of the default user from the server startup
process.

The idea is to move this functionality to a new task that will allow
users to create the default `argilla` user from command line.

Adding a task to create users from command line.

```bash 
# To create an user
 python -m argilla.tasks.create_user --username 3233edrito2df --role admin --password 12345678932

# To show help
python -m argilla.tasks.create_user --help
```



TODO:
- [x] Add tests

---------

Co-authored-by: José Francisco Calvo <[email protected]>

This is a first proposal of the new authorization policies after
visiting them for the first time on @frascuchon PR:
#2469

The general idea is to have a `policies` package where we can create
different policies classes including the authorization logic for every
model in our application.

The `policies` package will include also two method to facilitate check
authorization policies on API handlers.
- `authorized`: that will raise a `ForbiddenOperationError` exception if
the policy does not check.
- `is_authorized`: that will return `True` or `False` depending if the
policy checks or not.

Things to have into account:
- The user that we get as `current_user` on API handlers is not an
SQLAlchemy ORM `User` model (we need to take a look to this
@frascuchon).
- The creation of default user on server startup it's breaking some of
my tests so I have disabled it (we can discuss if with the new user
creation task this can be removed).
- There is an alternative version of authorization policies that we can
implement using closures. It has a lot of flexibility but adds a little
bit of complexity to the code.

This PR adds a new task to create user with defaults credentials on
database.

```sh
# Get help
$ python -m argilla.tasks.users.create_default --help

# Create the default user
$ python -m argilla.tasks.users.create_default

# Create the default user without output
$ python -m argilla.tasks.users.create_default --quiet
```

I have refactored a little bit some `contexts.accounts` functions
related with passwords, maybe it could be applied later to some other
PRs on the go.

This PR improves the state of the users tasks package:
- Move `users_migrator` to `tasks.users` package and renamed it to
`migrate`.
- Rename `create_user` to `create`.

At the end we have the following tasks available:

```sh
# Creates a new user in the database
$ python -m argilla.tasks.users.create

# Creates a user with default credentials into the database
$ python -m argilla.tasks.users.create_default

# Migrates from a YAML users file to database
$ python -m argilla.tasks.users.migrate
```

And additionally (not related with this PR changes but adding it
anyway):

```sh
# Run pending database migrations
$ python -m argilla.tasks.database.migrate
```

Using the `DELETE /api/datasets/:name`, this PR shows how the
integration with the new database, models, and policies could be
applied.

- A new `DatasetPolicy` class has been created
- The current user used for this endpoint is now a `models.User`
instance (using the new method `provider.get_current_user`)

… package

This PR include changes to allow alembic database migrations work as
expected with argilla built as a Python package.

I have moved `alembic.ini` from root project to
`src/argilla/alembic.ini` . With this change we can't execute `alembic`
anymore without specifying where `alembic.ini` is with `--config`
parameter.

So in order to execute `alembic` command in development we need to set
`--config parameter`:

```sh
$ alembic --config src/argilla/alembic.ini upgrade head
```

Alternatively we can set `ALEMBIC_CONFIG`:

```sh
$ ALEMBIC_CONFIG=src/argilla/alembic.ini alembic upgrade head
```

I followed the steps below to test the changes:

## Create a testing environment to build the argilla package

```sh
$ conda create -n argilla-testing python=3.8
$ conda activate argilla-testing
```

## Build the package

Only building the python code for testing purposes:

```sh
$ python -m build
```

## Install the package

```sh
$ pip install dist/argilla-1.4.0.dev0.tar.gz[server,listeners,postgresql]
```

## Testing that changes are working

These command should be executed outside argilla root project path, to
be sure that everything is working running with the installed Python
package.

Running database migrations (default SQLite):

```sh
$ python -m argilla.tasks.database.migrate
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 74694870197c, create users table
INFO  [alembic.runtime.migration] Running upgrade 74694870197c -> 82a5a88a3fa5, create workspaces table
INFO  [alembic.runtime.migration] Running upgrade 82a5a88a3fa5 -> 1769ee58fbb4, create workspaces_users table
```

Running database migrations with PostreSQL:

```
$ createdb argilla
$ ARGILLA_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/argilla python -m argilla.tasks.database.migrate
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 74694870197c, create users table
INFO  [alembic.runtime.migration] Running upgrade 74694870197c -> 82a5a88a3fa5, create workspaces table
INFO  [alembic.runtime.migration] Running upgrade 82a5a88a3fa5 -> 1769ee58fbb4, create workspaces_users table
```

Adding user with default credentials (default SQLite):

```
$ python -m argilla.tasks.users.create_default
User with default credentials succesfully created:
• username: 'argilla'
• password: '1234'
• api_key:  'argilla.apikey'
```

Adding user with default credentials with PostgreSQL:

```
$ ARGILLA_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/argilla python -m argilla.tasks.users.create_default
User with default credentials succesfully created:
• username: 'argilla'
• password: '1234'
• api_key:  'argilla.apikey'
```

…argilla-io/argilla into feat/add-basic-database-support

This PR integrates new DB logic and policies with the current
implementation.

- The main changes are applied to the dataset service class
- The `auth.get_current_user` returns a `models.User` instance
- The ONLY change in endpoints is using this new `auth.get_current_user`
method

In a separate PR the pydantic User class will be cleaned and normalized

---------

Co-authored-by: José Francisco Calvo <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: José Francisco Calvo <[email protected]>

- Removing extra method and computed properties for pydantic
`User`class.
- import `models.User` instead of security. model

…ault task (#2516)

This PR includes the following changes:
* Adds `--api-key` and `--password` parameters to
`argilla.tasks.users.create_default`.
* Now `argilla.tasks.users.create_default` checks if there is a user
using default username before creating it and showing a message in such
case (but not raising an error).

The current client will check if a user can work on a specific workspace
by using workspaces info returning in `/api/me` Returning all workspaces
from the `/api/me` for admin users keeps the backward compatibility with
the client flow.

---------

Co-authored-by: José Francisco Calvo <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…elop

The workspace name will be mandatory for those requests working with
datasets.

All except the dataset list, which will remain optional.

Also, the dataset tests are upgraded using the `test_client` fixture,
instead of custom mocked_client

---------

Co-authored-by: José Francisco Calvo <[email protected]>

This PR will include changes to our Dockerfile images to have support to
the new database integration.

Only adding two new env vars `ADMIN_PASSWORD` and `ADMIN_API_KEY`
allowing the configuration of the password and api key when using
`release.Dockerfile`.

This PR adds a new optional `--workspace` parameter to the
`argilla.tasks.user.create` task. The parameter can be used multiple
times to add the user to multiple workspaces.

If the workspaces doesn't exist these will be created.

You can use the new parameter as follows:
```sh
$ python -m argilla.tasks.users.create --first-name John --last-name Doe --role admin --username johndoe --password 12345678 --workspace workspace-a --workspace workspace-b
User succesfully created:
• first_name: 'John'
• last_name: 'Doe'
• username: 'johndoe'
• role: 'admin'
• api_key: 'hZQKQa-zWzXSaw1VMgy8Ego6B8u2qoPFb7bp_BgKOKWnueYFzcvvevihs6YqlNOuHzFC_MQNwpmo41J5Ce3Htgj8mT6KIrYrt3mk4EFJ5S8'
• workspaces: ['workspace-a', 'workspace-b']
```

It's what it is

@jfcalvo I see also that the `script.mako` file is not included by
default. Do we need it?