Allows extraction of search fields from dataset metadata without dependencies on
db driver.

To avoid circular dependency issues, `Field` is used as input to `MetadataType`
constructor, `Expression` is output of `Field` methods, but then Field needs
`Range` which is defined in `model`

- Test supplying credentials from outside for session creation
- Test failure to obtain credentials

This is needed for `s3_{dump|fetch}` running in "remote" mode, i.e. with dask

no need to derive from object

Change to import mock from unittest.mock, which has been available in
the standard library since Python 3.3, well below our minimum.

* Utility to function to obtain IAM auth token.

* Hook IAM auth into engine instantiation.

* Return last token if not getting a new one.

* Cleanup coding style.

* Cleanup coding style.

* Cleanup coding style - how deep an indent does it want?

* Pass region name in test.

* s/region/region_name/  oops.

* Pass region name when creating boto3 client.

* Fix obtain_new_iam_auth_token() function signature.

* Move iam config into config layer.

* Make options and env var names more consistent.

* Test environment parser.

* Removed hard-to-test and unnecessary exception conversion.

* Add documentation.

* Update whats_new.rst

Co-authored-by: phaesler <[email protected]>

In memory index driver

…ast commit.

Typehint cleanup - mypy compliance

…h. (#1459)

* GitHub actions fixes (#1433)

* Github action fixes, backported from develop-1.9

* Updates to whats_new.rst

* Capitalise "Dependabot" in whats_new.rst

* Remove duplicate GHA Workflow for docs

We used to deploy to netlify for docs previews, but now that's done to
Read The Docs. The old Workflow should be removed.

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/adrienverge/yamllint.git: v1.30.0 → v1.31.0](https://github.com/adrienverge/yamllint.git/compare/v1.30.0...v1.31.0)

* Display error message instead of help message if a required argument isn't provided

* update whats_new

* print error message as well as usage info

* add license hook to pre-commit

* update license template and instructions, and whats_new

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add > and < to lark grammar

* refine logic

* use timestamp 0 as lowest bound instead of hardcoded date

* update whats_new

* update doco

* update whats_new

* Allow open date range in dc load and find_datasets (#1443)

* support open ended date range in query init

* allow open ended time ranges in load() and find_datasets(), also simplify logic for cli

* update doco and whats_new

* get end of datetime.now() to avoid failing tests due to second mismatches

* Minor update to documentation

Even with open bounds, dates are still inclusive of the start and end dates. Minor update to wording to make this clearer

---------

Co-authored-by: Ariana Barzinpour <[email protected]>
Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/adrienverge/yamllint.git: v1.31.0 → v1.32.0](https://github.com/adrienverge/yamllint.git/compare/v1.31.0...v1.32.0)

* add archive_less_mature option to add and update

* update whats_new

* add warning message in memory driver

* Pass X and Y Scale factors through to rasterio.warp.project. (#1450)

* Pass X and Y Scale factors through to rasterio.warp.project. Update whats_new

* Update PR number in whats_new.rst

* Remove unused import.

* Cleanup.

* Cleanup.

* Should probably just add it to the dictionary tbh.

* Respond to Kirill's comments.

* remove lineage from docs

* move archive_less_mature to abstract and allow for postgres

* move find dupes logic into a separate function

* update whats_new

* allow for a bit of leniency in datetime comparison when searching for less mature, add test case

* update whats_new

* properly add new files

* fix failing tests

* refactor doc_to_ds without adding dataset logic

* Add missing PR's to whats_new.rst and prepare for 1.8.13 release. (#1453)

* Fix gha pypi publishing condition (#1454)

* fix gha pypi publishing condition

* update whats_new

---------

Co-authored-by: Ariana Barzinpour <[email protected]>

* update ubntu installation instructions

* update wordlist

* update readme

* update wordlist again

* add a bit more info on db env variables; other misc improvements

* update barebones metadata type requirements

* fix typos, update wordlist

* fix some wording

* update integration db names

* rename agdcintegration.conf

* Always use XSCALE=1,YSCALE=1 in warp. (#1457)

* Use SCALEX=1,SCALEY=1 in both warp code-paths.

* remove data preparation page, add links to indexing guide

* fix typo, del data preparation scripts page

* increase buffer to 500ms

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Damien Ayers <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ariana Barzinpour <[email protected]>
Co-authored-by: Ariana-B <[email protected]>
Co-authored-by: Robbi Bishop-Taylor <[email protected]>

…h. (#1459)

* GitHub actions fixes (#1433)

* Github action fixes, backported from develop-1.9

* Updates to whats_new.rst

* Capitalise "Dependabot" in whats_new.rst

* Remove duplicate GHA Workflow for docs

We used to deploy to netlify for docs previews, but now that's done to
Read The Docs. The old Workflow should be removed.

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/adrienverge/yamllint.git: v1.30.0 → v1.31.0](https://github.com/adrienverge/yamllint.git/compare/v1.30.0...v1.31.0)

* Display error message instead of help message if a required argument isn't provided

* update whats_new

* print error message as well as usage info

* add license hook to pre-commit

* update license template and instructions, and whats_new

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add > and < to lark grammar

* refine logic

* use timestamp 0 as lowest bound instead of hardcoded date

* update whats_new

* update doco

* update whats_new

* Allow open date range in dc load and find_datasets (#1443)

* support open ended date range in query init

* allow open ended time ranges in load() and find_datasets(), also simplify logic for cli

* update doco and whats_new

* get end of datetime.now() to avoid failing tests due to second mismatches

* Minor update to documentation

Even with open bounds, dates are still inclusive of the start and end dates. Minor update to wording to make this clearer

---------

Co-authored-by: Ariana Barzinpour <[email protected]>
Co-authored-by: Robbi Bishop-Taylor <[email protected]>

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/adrienverge/yamllint.git: v1.31.0 → v1.32.0](https://github.com/adrienverge/yamllint.git/compare/v1.31.0...v1.32.0)

* add archive_less_mature option to add and update

* update whats_new

* add warning message in memory driver

* Pass X and Y Scale factors through to rasterio.warp.project. (#1450)

* Pass X and Y Scale factors through to rasterio.warp.project. Update whats_new

* Update PR number in whats_new.rst

* Remove unused import.

* Cleanup.

* Cleanup.

* Should probably just add it to the dictionary tbh.

* Respond to Kirill's comments.

* remove lineage from docs

* move archive_less_mature to abstract and allow for postgres

* move find dupes logic into a separate function

* update whats_new

* allow for a bit of leniency in datetime comparison when searching for less mature, add test case

* update whats_new

* properly add new files

* fix failing tests

* refactor doc_to_ds without adding dataset logic

* Add missing PR's to whats_new.rst and prepare for 1.8.13 release. (#1453)

* Fix gha pypi publishing condition (#1454)

* fix gha pypi publishing condition

* update whats_new

---------

Co-authored-by: Ariana Barzinpour <[email protected]>

* update ubntu installation instructions

* update wordlist

* update readme

* update wordlist again

* add a bit more info on db env variables; other misc improvements

* update barebones metadata type requirements

* fix typos, update wordlist

* fix some wording

* update integration db names

* rename agdcintegration.conf

* Always use XSCALE=1,YSCALE=1 in warp. (#1457)

* Use SCALEX=1,SCALEY=1 in both warp code-paths.

* remove data preparation page, add links to indexing guide

* fix typo, del data preparation scripts page

* increase buffer to 500ms

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Damien Ayers <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ariana Barzinpour <[email protected]>
Co-authored-by: Ariana-B <[email protected]>
Co-authored-by: Robbi Bishop-Taylor <[email protected]>

…nkering in validation

…m core