Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files endpoint #360

Merged
merged 36 commits into from
Jun 2, 2022
Merged
Changes from 35 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
318c2b5
Adding a scaffold to describe files endpoint.
merkys Jun 7, 2021
6c37306
Describing attributes of files endpoint.
merkys Jun 8, 2021
5fc402c
Listing files endpoint under 'Relationships Used by Multiple Entry Ty…
merkys Jun 8, 2021
6a06bde
Simplifying the description of relationships with Files entries.
merkys Jun 8, 2021
32c23c2
Merge branch 'develop' into files-endpoint
merkys Jun 10, 2021
c1b1227
Update optimade.rst
merkys Sep 8, 2021
55cac28
Merge branch 'develop' into files-endpoint
merkys Sep 8, 2021
57b18d6
Update optimade.rst
merkys Sep 8, 2021
f3ced3e
Introducing `description` property for free-form text.
merkys Sep 27, 2021
0419374
Describing `media_type` as RFC 6838 string.
merkys Sep 27, 2021
24e2907
Including `atime`, `ctime` and `mtime` as per @sauliusg request.
merkys Nov 22, 2021
fb187a7
Redefining `url` as JSON API links object.
merkys Nov 26, 2021
82f3d11
Including `size` as per @ml-evs request.
merkys Nov 26, 2021
eb5aa2e
Merge branch 'develop' into files-endpoint
merkys Nov 26, 2021
e535aa1
Starting each sentence with a capital letter. Removing excess periods.
merkys Nov 26, 2021
d680370
Revert "Redefining `url` as JSON API links object."
merkys Dec 8, 2021
32e265e
Removing excess period ('.').
merkys Dec 8, 2021
467d117
Making support for 'name' mandatory.
merkys Dec 8, 2021
574a27c
File name extension is an integral part of a file name.
merkys Dec 9, 2021
63b492d
Mentioning POSIX in `xtime` definitions.
merkys Dec 9, 2021
1b3bbfd
Describing the relation between `mtime` and `last_modified`.
merkys Dec 9, 2021
6959f5f
Introducing `md5sum` and `sha1sum`.
merkys Dec 9, 2021
5d7e9d4
Adding `url_stable_until`.
merkys Dec 10, 2021
d3e5c9b
Adding `modification_timestamp`.
merkys Dec 10, 2021
3d6f3b2
Describing the relation between `last_modified`, `modification_timest…
merkys Dec 10, 2021
b85e16d
Values for `modification_timestamp` SHOULD be increasing (as per @JPB…
merkys Dec 14, 2021
466b9e3
Merge branch 'develop' into files-endpoint
merkys Apr 1, 2022
fe97a05
Merge branch 'develop' into files-endpoint
merkys Jun 1, 2022
1b3d80f
Clarifying that an URL has to point to actual raw contents of a file.
merkys Jun 1, 2022
ae07299
Moving the explanation of restrictions for `url` usage lower.
merkys Jun 1, 2022
93e92ca
Substituting: "octet stream" -> "byte stream".
merkys Jun 1, 2022
210131f
Escaping `_` in property names used for subsection titles.
merkys Jun 1, 2022
e9d0274
Update optimade.rst
merkys Jun 1, 2022
575ef59
Merge branch 'develop' into files-endpoint
merkys Jun 1, 2022
e05b0f3
Moving all checksums to `checksums` property.
merkys Jun 1, 2022
f6f2fea
Merge branch 'develop' into files-endpoint
merkys Jun 2, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 193 additions & 0 deletions optimade.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2404,6 +2404,165 @@ Example:
}
}

Files Entries
-------------

The :entry:`files` entries describe files.
The following properties are used to do so:

url
~~~

- **Description**: The URL to get the contents of a file.
- **Type**: string
- **Requirements/Conventions**:

- **Support**: MUST be supported by all implementations, MUST NOT be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.
- **Response**: REQUIRED in the response.
- The URL MUST point to the actual contents of a file (i.e. byte stream), not an intermediate (preview) representation.
For example, if referring to a file on GitHub, a link should point to raw contents.

- **Examples**:

- :val:`"https://example.org/files/cifs/1000000.cif"`

url\_stable\_until
~~~~~~~~~~~~~~~~~~

- **Description**: Point in time until which the URL in `url` is guaranteed to stay stable.
- **Type**: timestamp
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.
- :val:`null` means that there is no stability guarantee for the URL in `url`.
Indefinite support could be communicated by providing a date sufficiently far in the future, for example, :val:`9999-12-31`.

name
~~~~

- **Description**: Base name of a file.
- **Type**: string
- **Requirements/Conventions**:

- **Support**: MUST be supported by all implementations, MUST NOT be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.
- File name extension is an integral part of a file name and, if available, MUST be included.

- **Examples**:

- :val:`"1000000.cif"`

size
~~~~

- **Description**: Size of a file in bytes.
- **Type**: integer
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.
- If provided, it MUST be guaranteed that either exact size of a file is given or its upper bound.
This way if a client reserves a static buffer or truncates the download stream after this many bytes the whole file would be received.
Such provision is included to allow the providers to serve on-the-fly compressed files.

media\_type
~~~~~~~~~~~

- **Description**: Media type identifier for a file as per `RFC 6838 Media Type Specifications and Registration Procedures <https://datatracker.ietf.org/doc/html/rfc6838>`__.
- **Type**: string
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.

- **Examples**:

- :val:`"chemical/x-cif"`

version
~~~~~~~

- **Description**: Version information of a file (e.g. commit, revision, timestamp).
- **Type**: string
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.
- If provided, it MUST be guaranteed that file contents pertaining to the same combination of :field:`id` and :field:`version` are the same.

modification\_timestamp
~~~~~~~~~~~~~~~~~~~~~~~

- **Description**: Timestamp of the last modification of file contents.
A modification is understood as an addition, change or deletion of one or more bytes, resulting in file contents different from the previous.
- **Type**: timestamp
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.
- Timestamps of subsequent file modifications SHOULD be increasing (not earlier than previous timestamps).

description
ml-evs marked this conversation as resolved.
Show resolved Hide resolved
~~~~~~~~~~~

- **Description**: Free-form description of a file.
- **Type**: string
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.

- **Examples**:

- :val:`"POSCAR format file"`

checksums
~~~~~~~~~

* **Description**: Dictionary providing checksums of file contents.
* **Type**: dictionary with keys identifying checksum functions and values (strings) giving the actual checksums
* **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.
- Supported dictionary keys: :property:`md5`, :property:`sha1`, :property:`sha224`, :property:`sha256`, :property:`sha384`, :property:`sha512`.
Checksums outside this list MAY be used, but their names MUST be prefixed by database-provider-specific namespace prefix (see appendix `Database-Provider-Specific Namespace Prefixes`_).

atime
~~~~~

- **Description**: Time of last access of a file as per POSIX standard.
- **Type**: timestamp
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.

ctime
~~~~~

- **Description**: Time of last status change of a file as per POSIX standard.
- **Type**: timestamp
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.

mtime
~~~~~
ml-evs marked this conversation as resolved.
Show resolved Hide resolved

- **Description**: Time of last modification of a file as per POSIX standard.
- **Type**: timestamp
- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations, i.e., MAY be :val:`null`.
- **Query**: Support for queries on this property is OPTIONAL.
- It should be noted that the values of :field:`last_modified`, :field:`modification_timestamp` and :field:`mtime` do not necessary match.
JPBergsma marked this conversation as resolved.
Show resolved Hide resolved
:field:`last_modified` pertains to the modification of the OPTIMADE metadata, :field:`modification_timestamp` pertains to file contents and :field:`mtime` pertains to the modification of the file (not necessary changing its contents).
For example, appending an empty string to a file would result in the change of :field:`mtime` in some operating systems, but this would not be deemed as a modification of its contents.

Database-Provider-Specific Entry Types
--------------------------------------

Expand Down Expand Up @@ -2495,6 +2654,40 @@ Relationships with calculations MAY be used to indicate provenance where a struc

At the moment the database providers are suggested to extend their API the way they choose, always using their database-provider-specific prefix in non-standardized fields.

Files
~~~~~

Relationships with files may be used to relate an entry with any number of :entry:`files` entries.

.. code:: jsonc

{
"data": {
"type": "structures",
"id": "example.db:structs:1234",
ml-evs marked this conversation as resolved.
Show resolved Hide resolved
"attributes": {
"chemical_formula_descriptive": "H2O"
},
"relationships": {
"files": {
"data": [
{ "type": "files", "id": "example.db:files:1234" }
]
}
}
},
"included": [
{
"type": "files",
"id": "example.db:files:1234",
"attributes": {
"media_type": "chemical/x-cif",
"url": "https://example.org/files/cifs/1234.cif"
}
}
]
}

Appendices
==========

Expand Down