Search unusable for very large projects

[aajtodd]

Describe the bug
The drop down search is unresponsive for very large projects.

Expected behaviour
Search should work on all project sizes.

Possibly an option to disable search all together in gradle plugin if a solution can't be found/implemented.

Screenshots
N/A

To Reproduce
The AWS SDK for Kotlin docs demonstrate the issue. We have over 300 sub-projects. Try the search drop down and type in e.g. getObject, no search results are given (firefox + chrome).

Dokka configuration

https://github.com/awslabs/aws-sdk-kotlin/blob/main/build.gradle.kts

Installation

• Operating system: macOS
• Build tool: Gradle v7.2
• Dokka version: 1.7.10

Additional context
May be improved by being smarter about how indexing is done and breakup the contents to support search over large code bases.

[IgnatBeresnev]

To be honest, I'm quite embarrassed to admit it, but Dokka seems to be downloading a 318MB JSON index file needed for search on every page load. This one in particular:

https://sdk.amazonaws.com/kotlin/api/latest/scripts/pages.json

If you're using NGINX to host documentation, I'm pretty sure you can set Cache Control headers for this specific resource. This is not going to make the actual search faster (it's still slow to the point of freezing), but it's at least going to work and save users some CPU resources, as well as save you some traffic:

What probably happens is the user is trying to open search before the JSON has been downloaded, and it leads to bugs.

---

It's quite an unusual case as even for stdlib the JSON is only 5MB. Still, we'll think of how we can mitigate it, it's definitely a problem.