Possibly fix derived param aliases test

[taldcroft]

Description

This might finally fix the intermittent problems with derived param aliases:

• Getting no samples with the original dt=200 was due to the fact that PITCH_FSS is not defined out of the FSS field of view, beyond pitch > 135. So sometimes there really was no data.
  • Changed to PITCH_CSS. This is basically always available.
• Reduced interval from 86400 to 2000: I think that 86400 was too long and this was actually sampling realtime when the CXC archive was within a day of current.
  • Notice that the final time is increasing during the tests, and indeed CxoTime(7.71253565e+08).date = '2022:161:13:04:55.816' was during a realtime comm pass.

=================================== FAILURES ===================================
___________ test_fetch_derived_param_aliases[sources1-DP_piTch_fss] ____________

msid = 'DP_piTch_fss', sources = ('maude',)

    @pytest.mark.parametrize('msid', ['DP_piTch_fss', 'Calc_pitCH_fss'])
    @pytest.mark.parametrize('sources', (('cxc',), ('maude',), ('cxc', 'maude')))
    def test_fetch_derived_param_aliases(msid, sources):
        cxc_tstop = fetch.get_time_range('dp_pitch_fss', 'secs')[1]
        with fetch.data_source(*sources):
            # Get data from +/- 1 day from end of CXC data
            dt = 86400  # seconds
            d1 = fetch.Msid('piTch_fss', cxc_tstop - dt, cxc_tstop + dt)
            d2 = fetch.Msid(msid, cxc_tstop - dt, cxc_tstop + dt)
        assert d2.msid == msid  # version as the user provide
        assert d2.MSID == d1.MSID  # normalized version for accessing databases
>       assert np.all(d1.times == d2.times)
E       assert False
E        +  where False = <function all at 0x7f1933413d30>(array([7.7110...71253564e+08]) == array([7.7110...71253565e+08])
E        +    where <function all at 0x7f1933413d30> = np.all
E           +array([7.71105988e+08, 7.71105989e+08, 7.71105990e+08, ...,\n
E           +       7.71253561e+08, 7.71253562e+08, 7.71253564e+08])
E           -array([7.71105988e+08, 7.71105989e+08, 7.71105990e+08, ...,\n
E           -       7.71253562e+08, 7.71253564e+08, 7.71253565e+08])
E           Full diff:
E             array([7.71105988e+08, 7.71105989e+08, 7.71105990e+08, ...,
E           -        7.71253562e+08, 7.71253564e+08, 7.71253565e+08],
E           +        7.71253561e+08, 7.71253562e+08, 7.71253564e+08],
E             ))

Ska/engarchive/tests/test_fetch.py:166: AssertionError
------------------------------ Captured log call -------------------------------
DEBUG    maude:maude.py:625 Getting URL http://telemetry.cfa.harvard.edu/maude/mrest/FLIGHT/msid.bin?m=DP_PITCH_FSS&ts=2022159200518241&tp=2022161200518241
DEBUG    maude:maude.py:625 Getting URL http://telemetry.cfa.harvard.edu/maude/mrest/FLIGHT/msid.bin?m=DP_PITCH_FSS&ts=2022159200518241&tp=2022161200518241
__________ test_fetch_derived_param_aliases[sources1-Calc_pitCH_fss] ___________

msid = 'Calc_pitCH_fss', sources = ('maude',)

    @pytest.mark.parametrize('msid', ['DP_piTch_fss', 'Calc_pitCH_fss'])
    @pytest.mark.parametrize('sources', (('cxc',), ('maude',), ('cxc', 'maude')))
    def test_fetch_derived_param_aliases(msid, sources):
        cxc_tstop = fetch.get_time_range('dp_pitch_fss', 'secs')[1]
        with fetch.data_source(*sources):
            # Get data from +/- 1 day from end of CXC data
            dt = 86400  # seconds
            d1 = fetch.Msid('piTch_fss', cxc_tstop - dt, cxc_tstop + dt)
            d2 = fetch.Msid(msid, cxc_tstop - dt, cxc_tstop + dt)
        assert d2.msid == msid  # version as the user provide
        assert d2.MSID == d1.MSID  # normalized version for accessing databases
>       assert np.all(d1.times == d2.times)
E       assert False
E        +  where False = <function all at 0x7f1933413d30>(array([7.7110...71253567e+08]) == array([7.7110...71253568e+08])
E        +    where <function all at 0x7f1933413d30> = np.all
E           +array([7.71105988e+08, 7.71105989e+08, 7.71105990e+08, ...,\n
E           +       7.71253565e+08, 7.71253566e+08, 7.71253567e+08])
E           -array([7.71105988e+08, 7.71105989e+08, 7.71105990e+08, ...,\n
E           -       7.71253566e+08, 7.71253567e+08, 7.71253568e+08])
E           Full diff:
E             array([7.71105988e+08, 7.71105989e+08, 7.71105990e+08, ...,
E           -        7.71253566e+08, 7.71253567e+08, 7.71253568e+08],
E           +        7.71253565e+08, 7.71253566e+08, 7.71253567e+08],
E             ))

Interface impacts

None

Testing

Unit tests

• Mac

Independent check of unit tests by [REVIEWER NAME]

• [PLATFORM]:

Functional tests

Changed the unit test to ensure failure, and got some semi-useful output:

----------------------------------------------------- Captured stdout call ------------------------------------------------------
CxoTime(cxc_tstop).date='2022:160:20:05:18.241' dt=2000
CxoTime(cxc_tstop - dt).date='2022:160:19:31:58.241'
CxoTime(cxc_tstop + dt).date='2022:160:20:38:38.241'
CxoTime.now().date='2022:163:11:29:51.692'
CxoTime.now().iso='2022-06-12 11:29:51.693'
sources=('maude',) msid='Calc_pitCH'

[javierggt]

This test just failed for me on HEAD. I created an environment on /proj/sot/ska/jgonzalez/miniconda3/envs/ska3-flight-test with ska3-flight and ska3-perl, created a link to /proj/sot/ska/data in it, and installed this branch. I just did this as user kadi to make sure it would work for you:

source /proj/sot/ska/jgonzalez/miniconda3/envs/ska3-flight-test/bin/ska_envs.sh 
cd /proj/sot/ska/jgonzalez/miniconda3/envs/ska3-flight-test/lib/python3.8/site-packages/
pytest Ska/engarchive/tests/test_fetch.py::test_fetch_derived_param_aliases

[taldcroft]

After all this time I finally was able to see a failure. And immediately found the root problem which is that the stop time for CXC telemetry was being computed from the wrong MSID!

I also made an improvement of using fetch.MSID instead of fetch.Msid to ensure that no data filtering is done in case there are bad values. @javierggt - please give this another try.

[taldcroft]

I also just fixed the merge conflict and rebased.

[javierggt]

Tests now pass for me on HEAD (same test env mentioned above, with the latest commits)