Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document/fix robots.txt #255

Closed
ccorda opened this issue Feb 9, 2023 · 0 comments · Fixed by #263
Closed

Document/fix robots.txt #255

ccorda opened this issue Feb 9, 2023 · 0 comments · Fixed by #263
Assignees
@ccorda
Milestone
1.3

Comments

@ccorda
Copy link
Contributor

ccorda commented Feb 9, 2023

We've typically hosted the origin wordpress on a *.wpengine.com subdomain. This then gets the default robots.txt file they apply to keep content out of the index:

User-agent: *
Disallow: /

However the downside is that your images have no-index, which keeps twitter from using. We then use the META.proxyWordPressImages option to get around this.

A couple improvements:

  1. document this.
  2. Rather than assume the server will inject no-index, let's specify with a version controlled robots.txt for WP.
@ccorda ccorda added this to the 1.3 milestone Feb 9, 2023
@ccorda ccorda self-assigned this Feb 9, 2023
@ccorda ccorda mentioned this issue Feb 11, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ccorda ccorda

Labels
None yet
Projects
None yet
Milestone
1.3
Development

Successfully merging a pull request may close this issue.

feat: default robots.txt for wordpress patronage/bubs-next
1 participant
@ccorda