Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] very slow response when allowedSites = __all #3292

Open
netcoop opened this issue Jul 22, 2022 · 3 comments · May be fixed by #3294
Open

[BUG] very slow response when allowedSites = __all #3292

netcoop opened this issue Jul 22, 2022 · 3 comments · May be fixed by #3294
Assignees

Comments

@netcoop
Copy link
Contributor

netcoop commented Jul 22, 2022

Describe the bug
Using the setting search.query.allowedSites = __all in a multi-site setup with many sites makes the search response and the auto-suggest very, very slow. I have 1 TYPO3 with 33 sites, and response times are between 5 and 15 seconds when using '__all'. If I set the value to all 33 domains, the response time < 1s.

I have debugged the issue and found that generating the list of domains is what takes up way too much time, I measured 5 to 12 seconds for these 33 sites. Additional problem is that this is done for every request, including every AJAX request for the auto-suggest feature.

Looking into the code (SiteHashService, SiteRepository), I found that an enormous amount of work is done of which in the end only the list of domains is used. What is even worse is that the result gets cached, but in a runtime cache which is empty at the next request, and because this method is only called once for each request, this caching only generates more overhead.

The core SiteFinder class can produce the same list of domains in less than 1 ms. Unless I missed something crucial that's being done in the complex current process I would propose to replace the method in SiteHashService (and further clean up that class):

<?php

        ....

	protected function getDomainListOfAllSites(): string
	{
		$siteFinder = GeneralUtility::makeInstance(TYPO3\CMS\Core\Site\SiteFinder::class);
		$sites = $siteFinder->getAllSites();
		$domains = [];
		foreach ($sites as $site) {
			$domains[] = $site->getBase()->getHost();
		}

		return implode(',', $domains);
	}

To Reproduce
Steps to reproduce the behavior:

  1. Create a setup with many sites on different domains or subdomains
  2. set plugin.tx_solr.search.query.allowedSites = __all
  3. check how long it takes to for the search page or the auto-suggest to respond
  4. now try the same with all domains explicitly listed in allowedSites
    You'll find a remarkable speed improvement with step 4.

Expected behavior
Using '__all' should not slow down the search or auto-suggest.

Used versions (please complete the following information):

  • TYPO3 Version: [11.5.13]
  • EXT:solr Version: [11.5.0-rc-2]
  • Used Apache Solr Version: [8.11.1]
  • PHP Version: [7.4.30]
  • MariaDB Version: [10.3]

Possibly related to:
#2655

@dkd-kaehm
Copy link
Collaborator

@netcoop Thanks for reporting and nice explanation.
Would you like to provide a pull-request for this issue?


Targets:

  • main
  • release-11.5.x
  • release-11.2.x

@netcoop
Copy link
Contributor Author

netcoop commented Jul 22, 2022

I'll prepare a pull request

netcoop added a commit to netcoop/ext-solr that referenced this issue Jul 24, 2022
@netcoop netcoop linked a pull request Jul 24, 2022 that will close this issue
netcoop added a commit to netcoop/ext-solr that referenced this issue Jul 27, 2022
netcoop added a commit to netcoop/ext-solr that referenced this issue Jul 27, 2022
netcoop added a commit to netcoop/ext-solr that referenced this issue Jul 27, 2022
@netcoop
Copy link
Contributor Author

netcoop commented Jul 27, 2022

I tried to fix the tests, but 1 of the integration tests actually showed that my fix produces a slightly different result than the original code: mine returns all domains, the original code only those with solr enabled. Or are there even more constraints that I'm not aware of?

Anyway, a solution could be one of 3 options:

  1. adding the constraints to my solution using siteFinder (checking if solr is enabled should be possible from the siteConfiguration I guess, if that's the only thing that's missing)
  2. improving the performance of the current code, would be a good idea anyway because it is evaluated for each request, but I fear most time is consumed by making the solr connection (haven't been able to check that yet)
  3. caching the method, with more than a runtime cache

Suggestions for a preferred solution or other ideas that can help solve this?

dkd-kaehm pushed a commit to netcoop/ext-solr that referenced this issue Sep 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants