FDataIrregular discussion

[eliegoudout]

Hello,

Is there a pointer towards a discussion regarding the structure and some implementation choices of the FDataIrregular feature?

In particular, I was wondering about the choice to stack all samples' evaluation points, which requires keeping track of start_indices. I don't say I'm against this choice, which is handy in some cases, but I think there is little coherence with the representation choice of grid_points and domain_range for FDataGrid (see #563 for example).

Thanks!

[vnmabus]

The original issue is #498, although we have discussed the details in some online meetings. The pull request is #536, as you probably already know. It is almost finished, and only requires a final review.

The choice of stacking points is because we can vectorize operations this way, such as elementwise operations and per-function reduction operations (using ufunc.reduceat). That said, I am not opposed to hearing alternatives. If you have a better proposal, it would be a good time to say so, before the final decision is made.

About #563 I have been thinking on your proposal as part of an effort I am trying to make to extract the function representation utilities to their own package, and generalize their dtypes and shapes. I am becoming more and more convinced that #563 is in the right direction. I should probably reopen that issue once I have a more concrete proposal.

[eliegoudout]

The original issue is #498, although we have discussed the details in some online meetings.

Thanks for the pointer!

The choice of stacking points is because we can vectorize operations this way, such as elementwise operations and per-function reduction operations (using ufunc.reduceat). That said, I am not opposed to hearing alternatives. If you have a better proposal, it would be a good time to say so, before the final decision is made.

I think it's hard to really defend one approach over the other. Imo, a relevant question is "how often do we end up splitting the long points/values arrays during implementation, and is it worth it?".
Here is a way to think about it for me:

• The slowdown from splitting can become relevant for let's say >100 or >1000 samples (to be refined I guess),
• The time gained from parallelization is relevant up to N(to identify) points, because after a certain number of points, there are no significant gains. Indeed, having 100 lists of 10_000 points to work with must be essentially the same as one long array of 1_000_000 points I guess (correct me if I'm wrong). Then, if N=64 for example (I have absolutely no idea what it is, but maybe you do?), one may start to consider that since samples will often have more than 64 points, then the slowdown from keeping lists of arrays is negligible.

I'm sorry if this is not very clear, I don't know how to better phrase this line of thought.

One final idea I have would be to make split points a cached_property with an appropriate deleter method, so that both views are often available for virtually no cost.

[eliegoudout]

I am becoming more and more convinced that #563 is in the right direction.

Having considered a few methods regarding FDataGrid and FDataIrregular, I do think it would avoid multiple dr = np.asarray(self.domain_range). The speedup would be negligible for sure, but the readability would improve while deleting a few lines of code.

[vnmabus]

I am toying around in branch https://github.com/GAA-UAM/scikit-fda/tree/feature/ndfunction with the idea of having non-necessarily rectangular domains with a bounding box (equivalent to domain_range, but as an ndarray).