Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Request]: Follow up on robots.txt #683

Open
2 of 11 tasks
jduss4 opened this issue Aug 27, 2024 · 0 comments
Open
2 of 11 tasks

[Request]: Follow up on robots.txt #683

jduss4 opened this issue Aug 27, 2024 · 0 comments
Labels
Guides Initiative 2 Tracks work for "Improving the Maintainability of the 18F Guides and Methods"

Comments

@jduss4
Copy link
Contributor

jduss4 commented Aug 27, 2024

A description of the work

I created a robots.txt file for the guides (#682), but to be very thorough we need to address a few additional questions:

  • are there any user agents we want to have specific instructions?
  • are there any pages that we don't want crawled?

We could use DAP analytics to determine if there are any (friendly, instruction-receiving) bots that we would like to behave differently than they are currently.

Point of contact on this issue

jduss4

Reproduction steps (if necessary)

No response

Skills Needed

  • Any Human
  • Design
  • Content
  • Engineering
  • Acquisition
  • Product
  • Other

Does this need to happen in the next 2 weeks?

  • Yes
  • No

How much time do you anticipate this work taking?

4 hours

Acceptance Criteria

  • Additional directives have been added to the robots.txt file if there are changes to user-agents
  • Any URLs we do not wish to be crawled have been added to the robots.txt file for all bots
@jduss4 jduss4 added Initiative 2 Tracks work for "Improving the Maintainability of the 18F Guides and Methods" Guides labels Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Guides Initiative 2 Tracks work for "Improving the Maintainability of the 18F Guides and Methods"
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jduss4