Event filtering gives unexpected exception for inverted fetch time range

[jeanconn]

I get an error when trying to remove ltt bads from an empty (inverted range) contained within a day that has an entry in the ltt_bads file. This is obviously a special case, but isn't a problem for the other kadi event intervals.

from kadi import events
from Ska.engarchive import fetch
d = events.dwells.filter(obsid=51641)[0]
# this returns an empty set contained within the day 2015:252 which should be bad for AACCCDPT
telem = fetch.Msid('AOATTER1', d.tstart + 500, d.stop)
# I can "remove" dumps and tsc moves without error (even though there were already no values)
telem.remove_intervals(events.dumps)
telem.remove_intervals(events.tsc_moves)
# but removing the ltt bads throws a range error
telem.remove_intervals(events.ltt_bads)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-50-caf4420f10a3> in <module>()
----> 1 telem.remove_intervals(events.ltt_bads)

/proj/sot/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/Ska.engarchive-0.36.5-py2.7.egg/Ska/engarchive/fetch.pyc in remove_intervals(self, intervals, copy)
    789         """
    790         obj = self.copy() if copy else self
--> 791         obj._filter_times(intervals, exclude=True)
    792         if copy:
    793             return obj

/proj/sot/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/Ska.engarchive-0.36.5-py2.7.egg/Ska/engarchive/fetch.pyc in _filter_times(self, intervals, exclude)
    862             if tstart > tstop:
    863                 raise ValueError("Start time %s must be less than stop time %s"
--> 864                                  % (tstart, tstop))
    865 
    866             if tstop < self.times[0] or tstart > self.times[-1]:

ValueError: Start time 558163554.332 must be less than stop time 558163164.007

[taldcroft]

This is unrelated to ltt_bads (renamed issue accordingly):

telem = fetch.Msid('AOATTER1', d.tstart + 50, d.tstart + 40)
telem.remove_intervals(events.dwells)
...
ValueError: Start time 558163104.332 must be less than stop time 558163094.332

I believe that it only works for dumps and TSC moves because there is no intersection of the event interval(s) with the fetch data interval.

The question is whether to issue an error or warning when a fetch query has effectively invalid inputs, or whether to allow invalid time intervals in remove intervals. Maybe a warning would be useful to discover potential errors in the calling code. In this particular case I would say there is a bug in the upstream code for not checking that the fetch interval is valid.

[jeanconn]

When you say "because there is no intersection", which do you mean?

[taldcroft]

Comment updated.

[jeanconn]

I'm still a little confused about the "fetch data interval". If I had a bad time range for fetch or I have removed intervals so I now have an empty set of times and values, is the "fetch data interval" useful? Also, should a "backwards" fetch range be able to intersect with a valid interval?