Search correct language for Docusaurus docsearch #744

JoelMarcey · 2018-06-09T01:06:29Z

Motivation

Only search the current version of the docs. Right now there are duplicates because next is also searched.

https://github.com/facebook/Docusaurus/blob/d28b864a59fabeea45add8c090a13de7d0530de5/lib/core/Site.js#L140 does the replacement

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

eyes

Related PRs

N/A

Only search the current version of the docs. Right now there are duplicates because `next` is also searched. https://github.com/facebook/Docusaurus/blob/d28b864a59fabeea45add8c090a13de7d0530de5/lib/core/Site.js#L140 does the replacement

docusaurus-bot · 2018-06-09T01:07:58Z

Deploy preview for docusaurus-preview ready!

Built with commit 0dfce4a

https://deploy-preview-744--docusaurus-preview.netlify.com

endiliey · 2018-06-09T01:36:55Z

Actually we should probably do it for languages too.

But the current state is that
https://github.com/algolia/docsearch-configs/blob/master/configs/docusaurus.json

Only has 'en' for lang and 'next' for version
So no matter which language or version we always only search on that lang & version.

I am currently away so I won't be able to submit PR for it.

Currently searching both next and latest. Only want latest. Ref: facebook/docusaurus#744

JoelMarcey · 2018-06-09T03:23:22Z

@endiliey I added a PR to Aloglia

…lter

endiliey

I've pushed a fix to your PR @JoelMarcey

Let's wait for algolia/docsearch-configs#446 to be merged & test it on netlify

endiliey · 2018-06-09T01:32:42Z

website/siteConfig.js

@@ -36,6 +36,7 @@ const siteConfig = {
  algolia: {
    apiKey: '3eb9507824b8be89e7a199ecaa1a9d2c',
    indexName: 'docusaurus',
+    facetFilters: [ "version:VERSION" ],


Shouldn't it be.

algoliaOptions: { facetFilters: [ "version:VERSION"] }

endiliey · 2018-06-09T03:37:39Z

website/siteConfig.js

@@ -36,6 +36,7 @@ const siteConfig = {
  algolia: {
    apiKey: '3eb9507824b8be89e7a199ecaa1a9d2c',
    indexName: 'docusaurus',
+    facetFilters: [ "lang:LANGUAGE", "version:VERSION" ],


Need algoliaOptions for this to work

https://docusaurus.io/docs/en/next/search.html#enabling-the-search-bar

algoliaOptions: { facetFilters: ["lang:LANGUAGE", "version:VERSION"] }

yangshun · 2018-06-11T01:31:44Z

Is this PR being blocked by algolia/docsearch-configs#446?

endiliey · 2018-06-11T01:34:49Z

@yangshun. Yes, tbh we need to wait ~24 hours after that PR got merged so algolia crawler had finished crawling & we can test this

…docusaurus#744 #438

endiliey

algolia/docsearch-configs#446 only allow latest version.

When we are at docusaurus non-latest version, the search will not work.

Solution

Change "version:VERSION" to "version:latest"
But this means whichever version we are at, we are searching on latest docs only. I think this is unnatural, let's say I'm at docusaurus version 1.05 page, the latest version is 1.2 and language 'en', it will search for the version 1.2 instead of 1.05
Change algolia config to include all docusaurus version. (Submit another PR at docsearch-configs)
So when we are at v1.05 and lang 'ko' it will search on that version & language

…g duplicates as #446 cc facebook/docusaurus#744 #438

s-pace · 2018-06-11T10:09:11Z

👋

I am currently trying to tailor you the best configuration in order to have something simple and easy to understand.

We are facing two majors issue regarding how we scrap your website:

There is no redirection from https://docusaurus.io/docs/<lang>/ to a document page like https://docusaurus.io/docs/<lang>/installation.
Could it be possible to have such redirection straight away?
Sitemap introduces pages with URL ending by ".html" for the english pages while the same page without this trailing ".html" are still available. This introduces some duplicates since we crawl pages with and without the trailing ".html" in order to discover non-english pages. Could it be possible to avoid these pages with the trailing ".html" from the sitemap?

An easier solution would be to extract the context of the page (version and lang) from the DOM itself. Could it be possible to add meta tags?

JoelMarcey · 2018-06-11T20:40:32Z

@s-pace Hi 👋 -- sorry this is causing so much trouble. But I am hoping if we solve this correctly for Docusaurus, then sites that use Docusaurus can benefit from what we learned.

An easier solution would be to extract the context of the page (version and lang) from the DOM itself. Could it be possible to add meta tags?

I think we can probably do this, maybe. Just any meta tag? e.g.

<meta docLanguage = ...>
<meta docVersion = ...>

If we use meta tags:

Can we avoid the redirection problems you raised?
Would the sitemap issue be fixed too?
Would we need to change anything on the Algolia config side to enable this?

Thanks!

s-pace · 2018-06-12T07:27:00Z

@JoelMarcey

I am hoping if we solve this correctly for Docusaurus, then sites that use Docusaurus can benefit from what we learned.

Makes sense definitely. Let's try to have something working with it.

Using meta tag is fine for us. It will introduce a bit more of complexity within your configuration but we can do it.

Can we avoid the redirection problems you raised?

This will not change the issue we are struggling with since we receive a 404 and thus we stop crawling your website. If this is not possible, we will have to write one start_url object to reach an available page and one for the root in order to define the allowed domain to crawl. Although this can be solved if you provide us a clean and complete sitemap (easiest solution).

Would the sitemap issue be fixed too?

Unfortunately this issue will not be fixed since pages are treated as different pages. The easiest solutions I see:

Remove the trailing .html from the sitemap. (You may want to have sitemap for other languages btw)
Choose to only provde pages with or without the trailing .html
Specify a canonical page which should be the one you want our Search UI to redirect to.

Would we need to change anything on the DocSearch config side to enable this?

Yes, we will need to change something from the config anyway. This can be quickly done.

endiliey · 2018-06-12T10:05:33Z

@s-pace

I see two solutions of this. Either
A. Redirect docusaurus.io/docs/<lang>/ to any first available docs
B. Change the docsearch config start_url from docusaurus.io/docs/<lang>/ to docusaurus.io/docs/<lang>/installation so that it wont be 404 & algolia can crawl the site

Which do you prefer ?

In new version of Docusaurus, we provide user two options
A. Pages only accessible with .html extension
B. Pages are accessible with and without .html extension

If my understanding is right the solution for this is to provide a complete sitemap that only provide pages with or without .html pages (choose only one)

So the fix is to generate Sitemap with only .html extension for option A and Sitemap without .html extension for option B

At https://docusaurus.io/sitemap.xml

ref facebook/docusaurus#744

JoelMarcey · 2018-06-12T16:33:07Z

There is no redirection from https://docusaurus.io/docs/<lang>/ to a document page like https://docusaurus.io/docs/<lang>/installation. Could it be possible to have such redirection straight away?

I created algolia/docsearch-configs#453 as a short-term fix to this.

JoelMarcey · 2018-06-12T18:15:39Z

@s-pace I have a PR up for Docusaurus that creates a sitemap depending on whether a config option is set to allow extension-less URLs. So the sitemap will have *.html extensions if that config option is not set. If it is set, then the sitemap will not have any *.html extensions.

yangshun · 2018-06-16T04:17:57Z

@JoelMarcey Is this good to merge?

JoelMarcey · 2018-06-16T04:21:43Z

I am not sure we need this now. Or at least the whole thing. Do you see duplicate search results when you search now?

_{Sent with GitHawk}

yangshun · 2018-06-16T04:26:31Z

Actually I never did see duplicate search results before. I'm not seeing it now also. Will let you verify and close it if it's no longer needed.

JoelMarcey · 2018-06-16T04:27:21Z

I look at it again over the weekend and close, update or merge as appropriate.

endiliey · 2018-06-16T04:31:44Z

Actually the current state is that whichever our language is at, we always search for english language docs.

And whichever version we are currently, we always search for 'latest version'

https://github.com/algolia/docsearch-configs/blob/master/configs/docusaurus.json

JoelMarcey · 2018-06-16T04:35:05Z

Right. I think latest is good. We just want the language support.

endiliey · 2018-06-16T04:44:00Z

@JoelMarcey
I'll patch your PR, test & merge it later. I am back today.

Edit:
Blocked by algolia/docsearch-configs#458
DO NOT MERGE 😄

…lter

s-pace · 2018-06-18T14:24:45Z

DocSearch PR merged:

<!-- at the end of the HEAD -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" />

<!-- at the end of the BODY -->
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script>
<script type="text/javascript"> docsearch({
  apiKey: '3eb9507824b8be89e7a199ecaa1a9d2c',
  indexName: 'docusaurus',
  inputSelector: '### REPLACE ME ####',
  algoliaOptions: { 'facetFilters': ["lang:$LANG", "tags:$TAGS"] },
  debug: false // Set debug to true if you want to inspect the dropdown
});
</script>

Add a search input in your page if you don't have any yet. Then update the inputSelector value in JS snippet to a CSS selector that targets your search input field.
Replace $LANG with the lang you want to search on.
The list of possible lang is hardcoded in the config.
So as of today you have: en, es-ES, ro, tr, zh-CN
Replace $TAGS with the tags you want to search on.
The list of possible tags is hardcoded in the config.
So as of today you have: blog, docs

endiliey · 2018-06-18T15:11:17Z

Search for chinese when on chinese

Search for espanol when on espanol

Remove duplicate search results

e47736e

Only search the current version of the docs. Right now there are duplicates because `next` is also searched. https://github.com/facebook/Docusaurus/blob/d28b864a59fabeea45add8c090a13de7d0530de5/lib/core/Site.js#L140 does the replacement

facebook-github-bot added the CLA Signed Signed Facebook CLA label Jun 9, 2018

JoelMarcey mentioned this pull request Jun 9, 2018

Update Docusaurus search configs for sitemap and start urls algolia/docsearch-configs#438

Merged

JoelMarcey added a commit to JoelMarcey/docsearch-configs that referenced this pull request Jun 9, 2018

Only search the latest docs. Add supported language search.

16fb001

Currently searching both next and latest. Only want latest. Ref: facebook/docusaurus#744

JoelMarcey mentioned this pull request Jun 9, 2018

Only search the latest docs. Add supported language search. algolia/docsearch-configs#446

Merged

Add language facet

87d5d5f

JoelMarcey changed the title ~~Remove duplicate search results~~ Remove duplicate search results and add languages Jun 9, 2018

endiliey added 2 commits June 9, 2018 14:52

Merge remote-tracking branch 'upstream/master' into alogolia-facet-fi…

2c21522

…lter

fix missing algoliaOptions

2496aef

endiliey approved these changes Jun 9, 2018

View reviewed changes

s-pace pushed a commit to algolia/docsearch-configs that referenced this pull request Jun 11, 2018

deploying a config that isn't showing duplicates as #446 cc facebook/…

fd128dd

…docusaurus#744 #438

endiliey suggested changes Jun 11, 2018

View reviewed changes

s-pace pushed a commit to algolia/docsearch-configs that referenced this pull request Jun 11, 2018

removing confusing unused keys & deploying a config that isn't showin…

69ac2c5

…g duplicates as #446 cc facebook/docusaurus#744 #438

JoelMarcey added a commit to JoelMarcey/docsearch-configs that referenced this pull request Jun 12, 2018

Add specific start URL

bd9b369

ref facebook/docusaurus#744

JoelMarcey mentioned this pull request Jun 12, 2018

Add specific start URL algolia/docsearch-configs#453

Merged

s-pace mentioned this pull request Jun 12, 2018

Revert "Add specific start URL" algolia/docsearch-configs#454

Merged

JoelMarcey mentioned this pull request Jun 12, 2018

Configure sitemap depending on whether clean URLs enabled #765

Merged

endiliey added ✅ Test Plan labels Jun 13, 2018

endiliey added 2 commits June 16, 2018 17:26

Merge remote-tracking branch 'upstream/master' into alogolia-facet-fi…

f6f792b

…lter

use only language facetfilter

0dfce4a

endiliey force-pushed the alogolia-facet-filter branch from ebfbb80 to 0dfce4a Compare June 16, 2018 09:27

endiliey mentioned this pull request Jun 16, 2018

add lang search for docusaurus algolia/docsearch-configs#458

Merged

endiliey added the status: don't merge yet This PR is completed, but we are still waiting for other things to be ready. label Jun 16, 2018

endiliey changed the title ~~Remove duplicate search results and add languages~~ search correct language for Docusaurus docsearch Jun 16, 2018

endiliey changed the title ~~search correct language for Docusaurus docsearch~~ Search correct language for Docusaurus docsearch Jun 16, 2018

endiliey merged commit df42926 into facebook:master Jun 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Search correct language for Docusaurus docsearch #744

Search correct language for Docusaurus docsearch #744

JoelMarcey commented Jun 9, 2018

docusaurus-bot commented Jun 9, 2018 •

edited

Loading

endiliey commented Jun 9, 2018

JoelMarcey commented Jun 9, 2018

endiliey left a comment •

edited

Loading

endiliey Jun 9, 2018

endiliey Jun 9, 2018

JoelMarcey Jun 9, 2018

yangshun commented Jun 11, 2018

endiliey commented Jun 11, 2018

endiliey left a comment •

edited

Loading

s-pace commented Jun 11, 2018

JoelMarcey commented Jun 11, 2018

s-pace commented Jun 12, 2018

endiliey commented Jun 12, 2018 •

edited

Loading

JoelMarcey commented Jun 12, 2018

JoelMarcey commented Jun 12, 2018

yangshun commented Jun 16, 2018

JoelMarcey commented Jun 16, 2018

yangshun commented Jun 16, 2018

JoelMarcey commented Jun 16, 2018

endiliey commented Jun 16, 2018

JoelMarcey commented Jun 16, 2018

endiliey commented Jun 16, 2018 •

edited

Loading

s-pace commented Jun 18, 2018

endiliey commented Jun 18, 2018

Search correct language for Docusaurus docsearch #744

Search correct language for Docusaurus docsearch #744

Conversation

JoelMarcey commented Jun 9, 2018

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related PRs

docusaurus-bot commented Jun 9, 2018 • edited Loading

endiliey commented Jun 9, 2018

JoelMarcey commented Jun 9, 2018

endiliey left a comment • edited Loading

Choose a reason for hiding this comment

endiliey Jun 9, 2018

Choose a reason for hiding this comment

endiliey Jun 9, 2018

Choose a reason for hiding this comment

JoelMarcey Jun 9, 2018

Choose a reason for hiding this comment

yangshun commented Jun 11, 2018

endiliey commented Jun 11, 2018

endiliey left a comment • edited Loading

Choose a reason for hiding this comment

s-pace commented Jun 11, 2018

JoelMarcey commented Jun 11, 2018

s-pace commented Jun 12, 2018

endiliey commented Jun 12, 2018 • edited Loading

JoelMarcey commented Jun 12, 2018

JoelMarcey commented Jun 12, 2018

yangshun commented Jun 16, 2018

JoelMarcey commented Jun 16, 2018

yangshun commented Jun 16, 2018

JoelMarcey commented Jun 16, 2018

endiliey commented Jun 16, 2018

JoelMarcey commented Jun 16, 2018

endiliey commented Jun 16, 2018 • edited Loading

s-pace commented Jun 18, 2018

endiliey commented Jun 18, 2018

docusaurus-bot commented Jun 9, 2018 •

edited

Loading

endiliey left a comment •

edited

Loading

endiliey left a comment •

edited

Loading

endiliey commented Jun 12, 2018 •

edited

Loading

endiliey commented Jun 16, 2018 •

edited

Loading