Please describe the use case that requires this feature.

Our CI only has jobs that run on the newest possible set of dependencies as resolved by either pip or mamba.

We should also have a CI test testing on the oldest still supported core depdencies, e.g. once we merge #2580 , we
should have a CI job running on numpy 1.23, astropy 5.x, etc.

Describe the solution you'd like

Add a job to the CI matrix running jobs for the oldest still supported core dependencies.