TST: SDSS remote data not found (404), failing 6 tests

[pllim]

Example log: https://github.com/astropy/specutils/actions/runs/8994189000/job/24707287554

FAILED ../../specutils/tests/test_loaders.py::test_sdss_spspec - urllib.error.HTTPError: HTTP Error 404: Not Found
FAILED ../../specutils/tests/test_loaders.py::test_sdss_spspec_stream - urllib.error.HTTPError: HTTP Error 404: Not Found
FAILED ../../specutils/tests/test_loaders.py::test_sdss_compressed[gzip] - urllib.error.HTTPError: HTTP Error 404: Not Found
FAILED ../../specutils/tests/test_loaders.py::test_sdss_compressed[bzip2] - urllib.error.HTTPError: HTTP Error 404: Not Found
FAILED ../../specutils/tests/test_loaders.py::test_sdss_compressed[xz] - urllib.error.HTTPError: HTTP Error 404: Not Found
FAILED ../../specutils/tests/test_loaders.py::test_sdss_spplate - urllib.error.HTTPError: HTTP Error 404: Not Found

1. Ask SDSS if this is temporary.
2. If not, find replacement data or create fake local data to mimic the desired format.

[eteq]

I don't have any insider knowledge from SDSS, but I know this server goes up and down some of the time - not the first time this particular source of data has been 404ing. (But the last time I remember this happening it was down for a while, like days or maybe even weeks until someone realized it...)

[pllim]

Is there a more stable source?

[eteq]

I have a vague memory of a "better" URL that essentially goes for the same place but sometimes falls back on something else that is against a more modern API instead of direct URL? But I don't remember much more than that. Maybe @weaverba137 or @havok2063 have insight here (I could be just imagining this, but I think one of them might have been the person to tell me this)?

[eteq]

Oh, wait, I think I found part of it - the suggested solution was to use the SAS server: https://dr17.sdss.org/optical/spectrum/search - the problem was that it doesn't expose exactly the same data, and I think we couldn't figure out exactly how to get the same spectra. So the test would have to be re-written for a different file (which is not impossible but is non-trivial work)

[havok2063]

Some of the tests that are failing use this url, EBOSS_SPECTRUM_URL = 'https://data.sdss.org/sas/dr16/eboss/spectro/redux/v5_13_0/spectra/lite/4055/spec-4055-55359-0596.fits' , which is still accessible, https://data.sdss.org/sas/dr16/eboss/spectro/redux/v5_13_0/spectra/lite/4055/. So maybe there was a temporary hiccup.

Other tests that are failing use this old url, 'http://das.sdss.org/spectro/1d_26/0273/1d/ which does appear to be down, but this is a really old site. I'm not even sure who maintains that domain. If possible, I would update that to the SAS, e.g. https://data.sdss.org/sas/dr16/sdss/spectro/redux/26/0273/ which has the spPlate file, but spSpec files I don't think exist anymore. At least I can't find any reference. I think they are now just spec files.

[weaverba137]

I believe a simple http-> https replacement will fix issues with das.sdss.org.

[eteq]

It looks like the das server is back up as of right sometime in the last ~ hour... but the http -> https shift is probably a good idea anyway!

but spSpec files I don't think exist anymore. At least I can't find any reference. I think they are now just spec files.

I wonder if that means we should just grab that one example file and host it somewhere else? I don't think we want to deprecate the reader itself because there's still plenty of these files out there for various science use cases, but if they're not "officially" being served, we shouldn't have the tests depend on that as the authoritative copy.

[weaverba137]

No the spSpec files definitely still exist. Just follow the link & you'll see them.

[pllim]

Thanks! I confirmed locally that just upgrading to HTTPS works, so I opened #1136