-
Notifications
You must be signed in to change notification settings - Fork 267
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Discussion on restructuring Event Container structure #1165
Comments
Does anyone remember why |
I think the docu of |
I think this was just a mistake |
Can you clarify again what is meant to be the difference between |
We are still missing a place to store dl1b (hillas etc.) |
I would switch hierarchies to make things clearer: The toplevel should be an Telescope events have all the necessary telescope-wise information and maybe a backref to its array event if available. This is a few advantages imho: Telescope-Reconstruction algorithms like calibration and image parameterization can get just the telescope event and do their thing, no strange looping over tel_ids accessing multiple telescope-wise maps. compare e.g.: calibrate(data) # calibrate has an internal loop over all telescope events
for tel_id in data.tels_with_data:
pointing = data.pointing.tel[tel_id]
r0 = data.r0.tel[tel_id]
dl0 = data.dl0.tel[tel_id]
print(pointing, r0, dl0) with for telescope_event in array_event.telescope_events:
calibrate(telescope_event) # no loop over events, just calibrate this telescope event
print(telescope_event.r0, telescope_event.pointing, telescope_event.dl0)
stereo_reconstruction(array_event) Observed data event sources could then produce I haven't yet found a draw back of this and everything else seems to get much nicer. This is also about data locality. It keeps data that belongs together closer together in the code, as opposed to keeping data of similar kind but from different location together. |
@maxnoe I agree . The old scheme was trying to follow the top-level data model naming scheme, which wasn't really meant to be a storage hierarchy. The only question for me is whether or not to swap the data level and telescope, since separating things by data level is still useful (since you don't need to pass DL0 info on to higher steps, etc). So another option (perhaps a bit more complex) is:
as opposed to:
I'm thinking mostly of the use case of parallelization, where you want to be able to send around the least amount of info to a remote process. So if I'm doing the reconstruction step, I only need to "scatter" the dl1 info to multiple processors, and I don't want to also keep the dl0 info. I suppose it depends on where the scatter operation happens, at the telescope event level, or array event level (certainly for reconstruction, it's at the array event level) However, some thing like where to put data shared between data levels (pointing, MC shower info, etc) works better in the second case (your proposal), since you could have |
I think the rational was this:
|
I am also hugely in favor of Maxs proposal. In another issue @maxnoe also mentioned the service approach used e.g. in fact-tools to get information, that is not naturally synced with the event timing, such as pointing. I think telescope events should also carry refs to their respective services. I did not follow this discussion closely but I assume these services would be accessed lazily, right? |
There's another discussion about that in the various calibration issues, but no real resolution. What you are talking about is basically the difference between Monitoring data, and Event data that has been interpolated from Monitoring data. Monitoring data (a time-series of measurement that are not synced to the event stream, but rather taken once a second or so) is so far not treated in a nice way. But the idea (perhaps similar to the FACTTools services) is that you would have a Component that loads up monitoring data, constructs an interpolator, and writes the interpolated result to the Therefore what is in the event are instantaneous values of those quantities, interpolated to the current event time. Other COmponents that use them do not need to know how they were interpolated (or even if they were computed on-the-fly like with pedestals, perhaps), just that the value in the Container is the right one for this event. Therefore they don't need to distinguish between data coming from a "service" or from an event stream. |
For @maxnoe's proposal, the question is wether it is better to have a single "event data" as shown above, or something like we have now (but cleaned up), where there is a clear separation of input and output data. What we have now is like this: Which means you could parallelize (or just serialize to disk) one step of the data flow, by data level. If we invert the tree, this is harder (even if nicer conceptually in memory). What's of course missing in this diagram (and important), is where is the data that is shared between data levels? |
I'm sorry but I do not see the clear separation in the image above:
So what, I ask, can prevent me from writing a Component or accidentally misusing a Component, that reads from DL0 and writes back into DL0? |
@dneise Yes, both your statement are correct. I'm just showing the current design, where everything within a single Tool reads and writes to one central data structure called "Event". There is nothing stopping a rogue component from modifying another's data (other than regression tests, etc). Are you suggesting another design? In the suggestion from @maxnoe, it's even more linked, since the data levels cannot be separated into separate objects (and there can only be one central "array event" object shared with each stage of the analysis). In principle we could implement a "owner" system to prevent writing to places where one shouldn't, but not sure it's worth the effort. There are definitely good and bad points to this design, but so far it's not been a major problem. Especially if we later separate larger stages of analysis by on-disk files, then the ownership and input/output is clear (only in-memory it needs to be considered) |
I was trying to point out that this statement here:
seemed not entirely true to me. From my personal point of view, both Maxs proposal and the current solution are equal with respect to the question how linked the data levels are. I had the impression you were pointing out a downside of Maxs proposal and I wanted to point out that this is not a unique downside of Maxs proposal but of both solutions. I did not want to say this is an issue, that needs to be fixed. |
Perhaps it's useful to look at the simplified data flow just from DL0 → DL1, right now the situation looks (something) like this, where the order of calling each component goes from left to right: (of course right now we don't do the monitoring interpolation, the realtime calibration coefficient production part. In #1163 , the image processing is 2 components and the writing is not yet split into a component. But this is the design we are working toward so far). The question is: does this all still make sense, and is the container structure designed correctly? (clearly not so far). It gets more complex when you start to think of further stages:
|
I like the "event.array..." approach (Max's) for the event container structure, but I am not sure I understand (probably I don't) why it poses a problem when sending around data of specific levels - the structure does not force one to fill it completely, right? Is it just that you expect it to be much less efficient if you have to strip it of all the unneeded data, e.g. DL0? |
I also like @maxnoe proposal, one could also think to have r1, dl0 and dl1 at the telescope level and the dl2, dl3 at the array level: event.array.telescope.r1/dl0/dl1 Just, the "array" name is perhaps a bit misleading (given the numpy array), perhaps better "cta_array" or even only "cta" : event.cta.telescope.dl0 , event.cta.dl2 ...) |
@FrancaCassol yes exactly. DL2 / DL3 is array event wise |
Also, I don't mean event.array I mean array_event as the top level container. |
yes @maxnoe, I was looking at Karl's naming scheme which reflets the standard data model categories. The intermediate level is useful if part of the telescopes are pointing to different targets. I don't know how the data streams will be handled then, but in the case they will not be totally separated, one could introduce the "observation/target/subarray" level (call it as you like): event : with everything concerning the whole array (e.g. atmo, (?) ) |
There are still telescope-wise data at the DL2 level (but not DL3) - recall the discussion at the f2F ASWG meeting. But really, we don't need any DL3 data in this structure - that won'y be done event-wise. |
I don't think we should mix observation-level things in here - all processing is done for a single observation at most, with a single sub-array. |
I'd like to keep this as close as possible to the top-level CTA data model, rather than invert it. The point of this conversation was more about how to fix places where we are not consistent, rather than fully re-designing a new data model. I see some nice points of going for "telescope → dl0/dl1..., subarray → dl0/dl1", but at the same time it makes writing and reading the data much more complex (since it will be stored closer to how it is organized now) |
@kosack I don't think the CTA data model is meant or should be treated as something that should be exactly represented in code. It's about the general concepts and entities, not about how to layout storage in memory or in file. Different use cases might prefer the one or the other hierarchy in memory or in a file. That does not change how they relate to each other. I discussed with @FrancaCassol again, and I am much in favor now of using for the in memory representation for the preprocessing until DL2 a version that looks like this:
I think this as several key advantages:
You are referring to single-telescope files? How is this easier with the structure you proposed? I see the same challenges for both, not a general difference. I think we should make splitting and merging full telescope events easy. Currently this is hard, has you have to update lot's of telescope |
Yes, I think it might be clearer this way, and since the DL1 (and DL2) writer components I'm working on don't really care about the in-memory structure, reading and writing shouldn't be too hard to accomplish. However, can you explain a bit your diagram/structure above? I'm not sure I get it. For example, trigger_info is something right now that is written by the SWAT, so it's array-level information. Or is that the telescope-trigger information? and then where does the array-trigger go? Perhaps if you add one-level of detail lower, it will be clear to me. From the standpoint of data acquisition you will have:
So an in-memory structure that doesn't require too much manipulation to support loading these is useful. Also, where is the separation of monitoring and event data here? I guess this is all event (including monitoring parameters interpolated to events?). Would be nice to see also where non-interpolated monitoring info goes in this scheme (it's wasn't pretty in the old one either) |
Ah sorry, that was not really meant to be an exhaustive or even clear diagram. I just wanted to make the point array_event → list (or mapping) of telescope_events + array wide info and telescope_event → data_levels. Both the array event and the telescope_event will have trigger info. Array with central array information and telescope with telescope trigger info (e.g. trigger type, time, fired trigger patches...) |
OK, yes that makes more sense. So basically you have:
The same could be use for (non-interpolated) monitoring:
So you'd have something like: event.tel[n].dl0.waveform
event.tel[n].dl1.image
event.tel[n].dl2.shower.energy # mono
event.subarray.trigger.tels_with_data
event.subarray.dl2.shower.origin
event.subarray.dl2.shower.energy |
Monitoring is more service-like, isn't it? So it will probably be an api like: pedestal = monitoring.camera.get_pedestal(
tel_id=tel_id,
timestamp=tel_event.trigger.timestamp,
strategy='closest', # or earlier to only allow data known previously
) and this then may be do some database lookup or just access the values stored in the simtel event source depending on if what kind of data you are analyzing. |
Yes, that is what I intended for monitoring, in fact. However, you still need to write out some monitoring data (pedestals computed every few seconds, data quality parametersetc), so having a hierarchy would be nice for output. For input, you'd just load up an interpolater. I think somewhere there is an issue (or presentation)? Where I basically wanted to do exactly what you show above for reading monitoring data. Internally, it just loads all the data with |
One more question: how do we deal with the fact that the list of telescopes changes between data levels? At R1/DL0 you have all telescopes that triggered and read out. At DL1 you might only have parameters for a subset that passed cleaning. At DL2 you only have reconstructions for a further subset that had good parameters, etc. Right now, that is easy to deal with, but in this scheme, we would need to add a way to keep track of this, so you don't loop over things that are missing/dropped in the previous step. |
Yes, this is the thing that is harder in the current scheme. However, with using proper missing value markers or flags, this should be easy, right? Or just setting the respective dl1/dl2 containers to We probably should implement something to be able to disambiguate defaults from unset fields in containers. The current behaviour of setting the default for things not found in the input file seems dangerous. |
perhaps defining a common place to set an algorithm-specific quality flag is one option, or setting the Container to None if it is not included (but that might cause problems for writing). I just want to think about how algorithms will look after this change. For example, before we just could pass a list of params = {tel_id: t.dl1.parameters.hillas for tel_id, t in event.tel.items() if t.dl1.parameters.hillas]
shower = reconstruct(params)
event.subarray.dl2.shower = shower or you force all algorithms to know about the event structure (less ideal, especially if there are more than one algorithm that produces the output): reconstruct(event.tel) here a loop over tel events is made inside the Reconstructor, but then it need to know to get the parameters from |
Also, I guess when you say "list of telescopes", it's really a dict by tel_id, otherwise you have trouble connecting the telescope to the instrument description |
Anyhow, I'd still like to see an example of how this makes the loops for cleaning, parameterization, and reconstruction cleaner than before, since I still feel like it requires a lot of bookkeeping, given that the algorithms should not be too tied to the event structure. |
Yes, I think we have now seen that there are usecases that are easier and simpler in either format. So without an actual implementation, discussions will probably not be able to solve this. |
But especially from the single telescope's perspective, it would be nicer if they would just fill their This is also probably close to what real data taking will look like, right? First there are single telescope events that are then merged into array wide data structures. |
Yes, I do like that the TelescopeEvent can be separated (at least at the R1-DL0-DL1 level) |
I think that all events should be kept until actual selection at DL2 level, as you suggest with a reconstruction quality flag. |
It's not that easy. "Keeping all events" is something else than "using all events for stereo reconstruction". You can make all kinds of (telescope) event selections for the different reconstruction algorithms, which will probably be necessary to achieve a good performance. I agree we should keep all telescope events, but we also need to have a way to do the reconstruction only on a subset of those events (more than N pixels, less than X leakage, etc). So already at the DL2 level, you will have discarded some telescope events for some stereo reconstruction. (Or even Dl1 telescope event reconstruction in case of no or just a couple pixels after cleaning) |
Yes, and I meant keeping them not using all of them. Those not passing cleaning, for example, are obviously not used. Yet they should not disappear between DL0 and DL1/2 but can be flagged as such. |
After working on the LST EventSource again and all the calibration steps connected to that (e.g. time_shift), I came to the conclusion that the hierarchy
will be much easier to work with. Right now, LST defines it's own Especially the usecase of merging the different telescope streams will be much easier. |
@maxnoe In the inverted hierarchy above, can you expand the example to show where all subarray info as well as monitoring goes? I think it will work well, but would be nice to describe it in more detail. Some other nice points:
|
Since it's come up a few times that things in the current Event container structure are a bit haphazard, let's start a discussion on what should move. For reference, here is what it is now:
event_type
Event typeindex
event indexing informationevent_id
event identifierobs_id
observation identifierr0
Raw Dataobs_id
observation IDevent_id
event id numbertels_with_data
list of telescopes with datatel[X]
map of tel_id to R0CameraContainertrigger_time
Telescope trigger time, start of waveform readout, None for MCstrigger_type
camera's event trigger type if applicablenum_trig_pix
Number of trigger groups (sectors) listedtrig_pix_id
pixels involved in the camera triggerwaveform
numpy array containing ADC samples(n_channels, n_pixels, n_samples)r1
R1 Calibrated Dataobs_id
observation IDevent_id
event id numbertels_with_data
list of telescopes with datatel[X]
map of tel_id to R1CameraContainertrigger_time
Telescope trigger time, start of waveform readouttrigger_type
camera trigger typewaveform
numpy array containing a set of images, one per ADC sampleShape: (n_channels, n_pixels, n_samples)selected_gain_channel
Numpy array containing the gain channel chosen for each pixel. Shape: (n_pixels)dl0
DL0 Data Volume Reduced Dataobs_id
observation IDevent_id
event id numbertels_with_data
list of telescopes with datatel[X]
map of tel_id to DL0CameraContainertrigger_time
Telescope trigger time, start of waveform readouttrigger_type
camera trigger typewaveform
numpy array containing data volume reduced p.e. samples(n_pixels, n_samples). Note this may be a masked array, if pixels or time slices are zero-suppresseddl1
DL1 Calibrated imagetel[X]
map of tel_id to DL1CameraContainerimage
Numpy array of camera image, after waveform extraction.Shape: (n_pixel)pulse_time
Numpy array containing position of the pulse as determined by the extractor.Shape: (n_pixel, n_samples)dl2
Reconstructed Shower Informationshower[X]
Map of algorithm name to shower infoalt
reconstructed altitude [deg]alt_uncert
reconstructed altitude uncertainty [deg]az
reconstructed azimuth [deg]az_uncert
reconstructed azimuth uncertainty [deg]core_x
reconstructed x coordinate of the core position [m]core_y
reconstructed y coordinate of the core position [m]core_uncert
uncertainty of the reconstructed core position [m]h_max
reconstructed height of the shower maximumh_max_uncert
uncertainty of h_maxis_valid
direction validity flag. True if the shower directionwas properly reconstructed by the algorithmtel_ids
list of the telescope ids used in the reconstruction of the showeraverage_intensity
average intensity of the intensities used for reconstructiongoodness_of_fit
measure of algorithm success (if fit)energy[X]
Map of algorithm name to energy infoenergy
reconstructed energy [TeV]energy_uncert
reconstructed energy uncertainty [TeV]is_valid
energy reconstruction validity flag. True if the energy was properly reconstructed by the algorithmtel_ids
array containing the telescope ids used in the reconstruction of the showergoodness_of_fit
goodness of the algorithm fitclassification[X]
Map of algorithm name to classification infoprediction
prediction of the classifier, defined between [0,1], where values close to 0 are more gamma-like, and values close to 1 more hadron-likeis_valid
classificator validity flag. True if the predition was successful within the algorithm validity rangetel_ids
array containing the telescope ids used in the reconstruction of the showergoodness_of_fit
goodness of the algorithm fitmc
Monte-Carlo dataenergy
Monte-Carlo Energy [TeV]alt
Monte-carlo altitude [deg]az
Monte-Carlo azimuth [deg]core_x
MC core position [m]core_y
MC core position [m]h_first_int
Height of first interactionx_max
MC Xmax value [g / cm2]shower_primary_id
MC shower primary ID 0 (gamma), 1(e-),2(mu-), 100A+Z for nucleons and nuclei,negative for antimatter.*tel[X]
map of tel_id to MCCameraEventContainerphoto_electron_image[X]
reference image in pure photoelectrons, with no noisephoto_electron_image[X]
(ERROR): this is a Map that couldn't be accessed: 'NoneType' object is not callablereference_pulse_shape
reference pulse shape for each channeltime_slice
width of time slice [ns]dc_to_pe
DC/PE calibration arrays from MC filepedestal
pedestal calibration arrays from MC fileazimuth_raw
Raw azimuth angle [radians from N->E] for the telescopealtitude_raw
Raw altitude angle [radians] for the telescopeazimuth_cor
the tracking Azimuth corrected for pointing errors for the telescopealtitude_cor
the tracking Altitude corrected for pointing errors for the telescopemcheader
Monte-Carlo run header datarun_array_direction
the tracking/pointing direction in [radians]. Depending on 'tracking_mode' this either contains: [0]=Azimuth, [1]=Altitude in mode 0, OR [0]=R.A., [1]=Declination in mode 1.corsika_version
CORSIKA version * 1000simtel_version
sim_telarray version * 1000energy_range_min
Lower limit of energy range of primary particle [TeV]energy_range_max
Upper limit of energy range of primary particle [TeV]prod_site_B_total
total geomagnetic field [uT]prod_site_B_declination
magnetic declination [rad]prod_site_B_inclination
magnetic inclination [rad]prod_site_alt
height of observation level [m]prod_site_array
site arrayprod_site_coord
site (long., lat.) coordinatesprod_site_subarray
site subarrayspectral_index
Power-law spectral index of spectrumshower_prog_start
Time when shower simulation started,CORSIKA: only date
shower_prog_id
CORSIKA=1, ALTAI=2, KASCADE=3, MOCCA=4detector_prog_start
Time when detector simulation starteddetector_prog_id
simtelarray=1num_showers
Number of showers simulatedshower_reuse
Numbers of uses of each showermax_alt
Maximimum shower altitude [rad]min_alt
Minimum shower altitude [rad]max_az
Maximum shower azimuth [rad]min_az
Minimum shower azimuth [rad]diffuse
Diffuse Mode On/Offmax_viewcone_radius
Maximum viewcone radius [deg]min_viewcone_radius
Minimum viewcone radius [deg]max_scatter_range
Maximum scatter range [m]min_scatter_range
Minimum scatter range [m]core_pos_mode
Core Position Mode (fixed/circular/...)injection_height
Height of particle injection [m]atmosphere
Atmospheric model numbercorsika_iact_options
Detector MC informationcorsika_low_E_model
Detector MC informationcorsika_high_E_model
Detector MC informationcorsika_bunchsize
Number of photons per bunchcorsika_wlen_min
Minimum wavelength of cherenkov light [nm]corsika_wlen_max
Maximum wavelength of cherenkov light [nm]corsika_low_E_detail
Detector MC informationcorsika_high_E_detail
Detector MC informationtrig
central trigger informationgps_time
central average time stamptels_with_trigger
list of telescopes with datacount
number of events processedinst
instrumental informationsubarray
SubarrayDescription from the instrument modulepointing[X]
Telescope pointing positionsazimuth
Azimuth, measured N->E [rad]altitude
Altitude [rad]The text was updated successfully, but these errors were encountered: