fix: add another user-agent to robots.txt to prevent ICA crashes

acdh-oeaw · May 15, 2024 · 663c2ea · 663c2ea
1 parent 6370915
commit 663c2ea
Showing 1 changed file with 4 additions and 1 deletion.
diff --git a/robots_template/robots.txt b/robots_template/robots.txt
@@ -1,2 +1,5 @@
 User-agent: SemrushBot
-Disallow: /
+Disallow: /
+
+User-agent: Bytespider
+Disallow: /