High request times for retrieval of statistics & data on forms with a large number of contributors and submissions

[DavisRayM]

Current situation

The statistics endpoint /api/v1/stats/submissions alongside the data endpoint /api/v1/data (when used with start and limit query params) and media count endpoints currently take a lot of time(Around 1 - 3 minute depending on server load) to compute the statistics of a given form... This leads to some unpleasant user interactions.

We should look into a way to decrease the amount of processing/query time we take for such requests in order to speed up the request times as the current wait time is too long IMO

Suggested approaches

• Calculate submission stats as submissions are made(Async task post submission perhaps?) & save the statistics to cache

[DavisRayM]

Screenshot of the queries currently run on the stats endpoint:

Results: 20.54 ms (16 queries including 12 similar and 10 duplicates ); Form had around 4k submissions

[alepietrobon]

keep me posted on this, guys. Clients are onto it. Thanks!

[alepietrobon]

Dunno if i should reopen this or create a new ticket, but this problem is not solved yet.
These stats are still taking many minutes to load

[DavisRayM]

Closing this issue; Kindly re-open if review fails