Feat/add basic database support

.github/workflows/package.yml

@@ -19,6 +19,11 @@ on:
       - "feature/**"
       - "feat/**"
 
+env:
+  # Increase this value to reset cache if etc/example-environment.yml has not changed
+  CACHE_NUMBER:  5
+  ALEMBIC_CONFIG: src/argilla/alembic.ini
+
 jobs:
   test-elastic:
     name: Tests ElasticSearch
@@ -71,9 +76,6 @@ jobs:
         with:
           path: ${{ env.CONDA }}/envs
           key: conda-${{ runner.os }}--${{ runner.arch }}--${{ steps.get-date.outputs.today }}-${{ hashFiles('environment_dev.yml') }}-${{ env.CACHE_NUMBER }}
-        env:
-          # Increase this value to reset cache if etc/example-environment.yml has not changed
-          CACHE_NUMBER: 2
 
       - name: Update environment
         if: steps.filter.outputs.python_code == 'true' && steps.cache.outputs.cache-hit != 'true'
@@ -82,9 +84,6 @@ jobs:
       - name: Cache pip 👜
         uses: actions/cache@v2
         if: steps.filter.outputs.python_code == 'true'
-        env:
-          # Increase this value to reset cache if pyproject.toml has not changed
-          CACHE_NUMBER: 0
         with:
           path: ~/.cache/pip
           key: ${{ runner.os }}-pip-${{ env.CACHE_NUMBER }}-${{ hashFiles('pyproject.toml') }}
@@ -98,6 +97,8 @@ jobs:
       - name: Run tests 📈
         if: steps.filter.outputs.python_code == 'true'
         run: |
+          pip install -e ".[server,listeners]"
+          alembic upgrade head
           pytest --cov=argilla --cov-report=xml
           pip install "spacy<3.0" && python -m spacy download en_core_web_sm
           pytest tests/monitoring/test_spacy_monitoring.py
@@ -115,6 +116,7 @@ jobs:
   test-opensearch:
     name: Test OpenSearch
     runs-on: ubuntu-latest
+
     strategy:
       matrix:
         version: [ 1.3, 2.3 ]
@@ -161,9 +163,6 @@ jobs:
         with:
           path: ${{ env.CONDA }}/envs
           key: conda-${{ runner.os }}--${{ runner.arch }}--${{ steps.get-date.outputs.today }}-${{ hashFiles('environment_dev.yml') }}-${{ env.CACHE_NUMBER }}
-        env:
-          # Increase this value to reset cache if etc/example-environment.yml has not changed
-          CACHE_NUMBER: 2
 
       - name: Update environment
         if: steps.filter.outputs.python_code == 'true' && steps.cache.outputs.cache-hit != 'true'
@@ -172,9 +171,6 @@ jobs:
       - name: Cache pip 👜
         uses: actions/cache@v2
         if: steps.filter.outputs.python_code == 'true'
-        env:
-          # Increase this value to reset cache if pyproject.toml has not changed
-          CACHE_NUMBER: 0
         with:
           path: ~/.cache/pip
           key: ${{ runner.os }}-pip-${{ env.CACHE_NUMBER }}-${{ hashFiles('pyproject.toml') }}
@@ -188,6 +184,8 @@ jobs:
       - name: Run tests 📈
         if: steps.filter.outputs.python_code == 'true'
         run: |
+          pip install -e ".[server,listeners]"
+          alembic upgrade head
           pytest --cov=argilla --cov-report=xml
           pip install "spacy<3.0" && python -m spacy download en_core_web_sm
           pytest tests/monitoring/test_spacy_monitoring.py

.gitignore

@@ -130,9 +130,11 @@ sw.*
 # Vim swap files
 *.swp
 
-
 yarn.lock
 package-lock.json
 
 # App generated files
 src/**/server/static/
+
+# Old users db file
+.users.yml

CHANGELOG.md

@@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- `ARGILLA_HOME_PATH` new environment variable ([#2564]).
+- `ARGILLA_DATABASE_URL` new environment variable ([#2564]).
+- Basic support for user roles with `admin` and `annotator` ([#2564]).
+- `id`, `first_name`, `last_name`, `role`, `inserted_at` and `updated_at` new user fields ([#2564]).
+- `/api/users` new endpoint to list and create users ([#2564]).
+- `/api/users/{user_id}` new endpoint to delete users ([#2564]).
+- `/api/workspaces` new endpoint to list and create workspaces ([#2564]).
+- `/api/workspaces/{workspace_id}/users` new endpoint to list workspace users ([#2564]).
+- `/api/workspaces/{workspace_id}/users/{user_id}` new endpoint to create and delete workspace users ([#2564]).
+- `argilla.tasks.users.migrate` new task to migrate users from old YAML file to database ([#2564]).
+- `argilla.tasks.users.create` new task to create a user ([#2564]).
+- `argilla.tasks.users.create_default` new task to create a user with default credentials ([#2564]).
+- `argilla.tasks.database.migrate` new task to execute database migrations ([#2564]).
+- `release.Dockerfile` and `quickstart.Dockerfile` now creates a default `argilladata` volume to persist data ([#2564]).
+
+### Changed
+
+- `ARGILLA_USERS_DB_FILE` environment variable now it's only used to migrate users from YAML file to database ([#2564]).
+- `full_name` user field is now deprecated and `first_name` and `last_name` should be used instead ([#2564]).
+- `password` user field now requires a minimum of `8` and a maximum of `100` characters in size ([#2564]).
+- `quickstart.Dockerfile` image default users from `team` and `argilla` to `admin` and `annotator` including new passwords and API keys ([#2564]).
+- Datasets to be managed only by users with `admin` role ([#2564]).
+
+### Removed
+
+- `email` user field ([#2564]).
+- `disabled` user field ([#2564]).
+- Support for private workspaces ([#2564]).
+- `ARGILLA_LOCAL_AUTH_DEFAULT_APIKEY` and `ARGILLA_LOCAL_AUTH_DEFAULT_PASSWORD` environment variables. Use `python -m argilla.tasks.users.create_default` instead ([#2564]).
+
+[#2564]: https://github.com/argilla-io/argilla/issues/2564
+
 ## [1.5.0](https://github.com/recognai/rubrix/compare/v1.4.0...v1.5.0) - 2023-03-21
 
 ### Added

MANIFEST.in

@@ -1,2 +1,3 @@
+include src/argilla/alembic.ini
 graft src/argilla/server/static
-prune docs
+prune docs

docker-compose.yaml

@@ -8,17 +8,24 @@ services:
     ports:
       - "6900:6900"
     environment:
+      ARGILLA_HOME_PATH: /var/lib/argilla
       ARGILLA_ELASTICSEARCH: http://elasticsearch:9200
-      # Opt-out for telemetry https://docs.argilla.io/en/latest/reference/telemetry.html
-      # ARGILLA_ENABLE_TELEMETRY: 0
+      # ARGILLA_ENABLE_TELEMETRY: 0 # Opt-out for telemetry https://docs.argilla.io/en/latest/reference/telemetry.html
 
       # Set user configuration https://docs.argilla.io/en/latest/getting_started/installation/user_management.html
       # ARGILLA_LOCAL_AUTH_USERS_DB_FILE: /config/.users.yaml
       # volumes:
       #- ${PWD}/.users.yaml:/config/.users.yaml
+
+      # DEFAULT_USER_ENABLED: false # Uncomment this line to disable the creation of the default user
+      # DEFAULT_USER_PASSWORD: custom-password # Uncomment this line to set a custom password for the default user
+      # DEFAULT_USER_API_KEY: custom-api-key # Uncomment this line to set a custom api-key for the default user
     networks:
       - argilla
-
+    volumes:
+      # ARGILLA_HOME_PATH is used to define where Argilla will save it's application data.
+      # If you change ARGILLA_HOME_PATH value please copy that same value to argilladata volume too.
+      - argilladata:/var/lib/argilla
   elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:8.5.3
     environment:
@@ -48,8 +55,10 @@ services:
       ELASTICSEARCH_HOSTS: '["http://elasticsearch:9200"]'
     networks:
       - argilla
+
 networks:
   argilla:
     driver: bridge
 volumes:
+  argilladata:
   elasticdata:

docs/_source/getting_started/installation/configurations/elasticsearch.md

@@ -49,9 +49,9 @@ Within Elastic, it is possible to create snapshots of a running cluster. We high
 
 ### Mount back-up volume
 
-When deploying Elastic, we need to define a `path.repo` via setting this as an environment variable in your `docker-compose.yml` or by setting this in your `elasticsearch.yml`, and passing this as config. Additionally, we need to pass the same `path.repo` to a mounted volume. By default, we set this `elasticdata:/usr/share/elasticsearch/backups` because the `elasticsearch` user needs to have full permisions to act on the repo. Hence, setting the volume to something different might require some additional permission configurations. Note that the `minimum_master_nodes` need to be explicitly set when bound on a public IP.
+When deploying Elastic, we need to define a `path.repo` via setting this as an environment variable in your `docker-compose.yaml` or by setting this in your `elasticsearch.yml`, and passing this as config. Additionally, we need to pass the same `path.repo` to a mounted volume. By default, we set this `elasticdata:/usr/share/elasticsearch/backups` because the `elasticsearch` user needs to have full permisions to act on the repo. Hence, setting the volume to something different might require some additional permission configurations. Note that the `minimum_master_nodes` need to be explicitly set when bound on a public IP.
 
-#### `docker-compose.yml`
+#### `docker-compose.yaml`
 
 ```yaml
 elasticsearch:

docs/_source/getting_started/installation/configurations/server_configuration.md

@@ -32,6 +32,10 @@ You can set following environment variables to further configure your server and
 
 ### Server
 
+- `ARGILLA_HOME_PATH`: The directory where Argilla will store all the files needed to run. If the path doesn't exists it will be automatically created (Default: `~/.argilla`).
+
+- `ARGILLA_DATABASE_URL`: A URL string that contains the necessary information to connect to a database. Argilla uses SQLite by default, PostgreSQL is also officially supported (Default: `sqlite:///$ARGILLA_HOME_PATH/argilla.db?check_same_thread=False`).
+
 - `ARGILLA_ELASTICSEARCH`: URL of the connection endpoint of the Elasticsearch instance (Default: `http://localhost:9200`).
 
 - `ARGILLA_ELASTICSEARCH_SSL_VERIFY`: If "False", disables SSL certificate verification when connection to the Elasticsearch backend.