Search UI perfomance issues

[gbtb]

Hello.
I'm experiencing huge slowdown during first opening of search dropdown after a documentation page (re)load. I click inside search text edit, dropdown starts to open and then it hangs for about 4 to 8 seconds not reacting to keystrokes. After it unfreezes, buffered keystrokes gets displayed, but still its quite annoying, since as a newcomer I tend to open many tabs with docs and travel between them a lot.

I've suspected a slow network query, so I've looked into developer tools and looks like its not a query but some slow promise.

I'm using Firefox 69.0.1 (64-bit) with Arch Linux on an old but decent notebook with intel i5 and 8gb of ram.

[SeanTAllen]

What search are you referring to? Can you provide a link to an exemplar URL? The site that this repo is for has no search. I suspect you mean https://stdlib.ponylang.io/. If that is the case, the repo for that would be https://github.com/ponylang/mkdocs-theme.

[gbtb]

You are right, I was talking about search on https://stdlib.ponylang.io/ . Thanks for re-direction. I will open issue there.

[SeanTAllen]

@gtb no need to open an issue there! I can transfer this one.

[CandleCandle]

Chrome 77.0.3865.90
First search is quite slow, subsequent searches are fast.

[SeanTAllen]

On FF, this is super slow for me for the first search then pokey after that.

On MS Edge Dev, it is slow for first search and snappy after that.

[sylvanc]

MS Edge Dev: 1 sec delay on first search, then snappy.

[mfelsche]

The reason for the slowdown is that the search index to operate on is fetched when the search field is accessed the first time. The code responsible for this is here:

mkdocs-theme/src/assets/javascripts/components/Material/Search/Result.jsx

Lines 132 to 224 in d17f178

	/* Initialize index, if this has not be done yet */
	if (ev.type === "focus" && !this.index_) {
	/* Initialize index */
	const init = data => {
	/* Preprocess and index sections and documents */
	this.docs_ = data.reduce((docs, doc) => {
	const [path, hash] = doc.location.split("#")
	/* Associate section with parent document */
	if (hash) {
	doc.parent = docs.get(path)
	/* Override page title with document title if first section */
	if (doc.parent && !doc.parent.done) {
	doc.parent.title = doc.title
	doc.parent.text = doc.text
	doc.parent.done = true
	}
	}
	/* Some cleanup on the text */
	doc.text = doc.text
	.replace(/\n/g, " ") /* Remove newlines */
	.replace(/\s+/g, " ") /* Compact whitespace */
	.replace(/\s+([,.:;!?])/g, /* Correct punctuation */
	(_, char) => char)
	/* Index sections and documents, but skip top-level headline */
	if (!doc.parent || doc.parent.title !== doc.title)
	docs.set(doc.location, doc)
	return docs
	}, new Map)
	/* eslint-disable no-invalid-this */
	const docs = this.docs_,
	lang = this.lang_
	/* Create stack and index */
	this.stack_ = []
	this.index_ = lunr(function() {
	const filters = {
	"search.pipeline.trimmer": lunr.trimmer,
	"search.pipeline.stopwords": lunr.stopWordFilter
	}
	/* Disable stop words filter and trimmer, if desired */
	const pipeline = Object.keys(filters).reduce((result, name) => {
	if (!translate(name).match(/^false$/i))
	result.push(filters[name])
	return result
	}, [])
	/* Remove stemmer, as it cripples search experience */
	this.pipeline.reset()
	if (pipeline)
	this.pipeline.add(...pipeline)
	/* Set up alternate search languages */
	if (lang.length === 1 && lang[0] !== "en" && lunr[lang[0]]) {
	this.use(lunr[lang[0]])
	} else if (lang.length > 1) {
	this.use(lunr.multiLanguage(...lang))
	}
	/* Index fields */
	this.field("title", { boost: 10 })
	this.field("text")
	this.ref("location")
	/* Index documents */
	docs.forEach(doc => this.add(doc))
	})
	/* Register event handler for lazy rendering */
	const container = this.el_.parentNode
	if (!(container instanceof HTMLElement))
	throw new ReferenceError
	container.addEventListener("scroll", () => {
	while (this.stack_.length && container.scrollTop +
	container.offsetHeight >= container.scrollHeight - 16)
	this.stack_.splice(0, 10).forEach(render => render())
	})
	}
	/* eslint-enable no-invalid-this */
	/* Initialize index after short timeout to account for transition */
	setTimeout(() => {
	return typeof this.data_ === "function"
	? this.data_().then(init)
	: init(this.data_)
	}, 250)

https://github.com/ponylang/mkdocs-theme/blob/master/src/assets/javascripts/application.js#L362-L376

It currently fetches the 3.6MB index in json format, parses it and builds the index from it. Those operations are responsible for the delay.

I am not sure we can do much about the general operations necessary for building up the index.

One thing we should look at is to exclude the sources from the search index which should make it significantly smaller and will thus speed up things. And we should make it properly visible that the search is initializing, maybe some spinner or whatever UX element is best for that purpose.

[EpicEric]

Material for MkDocs 5.0.0 has been released. Among other things, it includes:

• Search UI does not freeze anymore (moved to web worker)
• Search index built only once when using instant loading
• Support for prebuilt search indexes

The first two items would probably be enough to solve this issue. If not, we could look into the last one.

This would probably need a squash-and-commit from the changes on upstream after 76d72da until abd2fc3, which would probably break a few things here. It would also break the current docgen -- since there were some breaking changes from version 2.7.1 of this fork up to upstream's 5.0.0 --, so ponyc will require some minor adjustments as well.

[SeanTAllen]

@EpicEric do you know what would need to change with ponyc's docgen?

[SeanTAllen]

A small thing we could also do in the meantime is not index the src directory.

[mfelsche]

This isnt actually a thing mkdocs supports. :(
But there is a plugin that might do what we want: https://github.com/chrieke/mkdocs-exclude-search

[EpicEric]

@SeanTAllen said:

@EpicEric do you know what would need to change with ponyc's docgen?

No, I don't recall what I meant by that.

[SeanTAllen]

@EpicEric does it only index directories that are in mkdocs.yml?

[SeanTAllen]

I prebuilt the index for https://ponylang.github.io/appdirs/ and it didn't seem to make a difference.

Is there something extra that needs to be done as far as you know @EpicEric ?