Disable cache for EXT:crawler indexing requests

[kraemer-igroup]

Hi @lochmueller, like Solr, indexing for indexed_search via EXT:crawler is not working properly in combination with the staticfilecache extension (see tomasnorre/crawler#1029).

For Solr this was fixed within these issues:
#200
#279

Could we also disable the cache for EXT:crawler indexing requests for indexed_search? I could provide a pull request.
EXT:crawler uses "X-T3Crawler" as header:
/~https://github.com/tomasnorre/crawler/blob/main/Classes/CrawlStrategy/GuzzleExecutionStrategy.php#L99

[lochmueller]

Hey @kraemer-igroup

thanks for the note. Could you describe the problem more detailed? You mean the reindex is not working, because EXT:crawler get the SFC Cache entry with a real http request? So you want to add the Crawler header to the htaccess configuration?

Feel free to send a PR.

Regards,
Tim

[kraemer-igroup]

Hey @lochmueller,

yes, sorry, the crawler request seems to get the SFC entry, which does not invoke the needed middlewares to generate the index entry. I tried do workaround it by adding this into my htaccess file:

# Disable cache for EXT:crawler indexing requests
RewriteCond %{HTTP:X-T3Crawler} .+
RewriteRule .* - [E=SFC_HOST:invalid-host]

But this only works, if the option useFallbackMiddleware is disabled. The pages/news are getting indexed again despite having a SFC entry. If I also do something like this with "X-T3Crawler" instead of "X-Tx-Solr-Iq", then the indexing works again, even with the useFallbackMiddleware option enabled: 756048b