[SQL] Operational Error: too many FROM clause terms, max 200

[antonylebechec]

Describe the bug
SQL Error where too many samples are selected

To Reproduce
Steps to reproduce the behavior, for example:

1. Select multiple samples in Validation plugin (or a big database?)
2. Apply a filter
3. See error:

"Operational Error: too many FROM clause terms, max 200"

Expected behavior
SQL query ok, no bug.
Probably LEFT/INNER JOIN problem.
Find a way to deal with multiple samples at the time.
See if using Sources solve the problem.

[dridk]

I suspect this comes from SQLite.
I would like to set a maximum field selection . It is unreadable to display 200 samples on the same line.

[antonylebechec]

I agree for a maximum number of fields, on variant view.
However, we will probably use CuteVariant to explore statistically, or with deep learning, and in that way, we will probably need to select a large number of information (fields) for a large number of samples (LEFT JOIN)

[antonylebechec]

Maybe by using the variable SQLITE_MAX_SRCLIST
https://github.com/sqlite/sqlite/blob/master/src/build.c
But I don't know how to change this... except by recompile SQLite...

[antonylebechec]

Dealing with more than 200 samples in a VQL query (e.g. WHERE samples.$any.gt >=0) is impossible.
But a plugin specific for statistical questions using specific SQL queries is possible.
An idea should be to reduce the number of considered samples to those on the "basket" (cf Validation plugin)