Advertools/Advertools_Audit_robots_txt_and_xml_sitemap_issues.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5da81fdb-1607-4dcd-8945-78492d41c20c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"8%\" alt=\"Advertools.png\" src=\"https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/.github/assets/logos/Advertools.png\" style=\"border-radius: 15%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96049d01-11e9-4538-9012-57992898281e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# Advertools - Audit robots txt and xml sitemap issues\n",
    "<a href=\"https://bit.ly/3JyWIk6\">Give Feedback</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=Advertools+-+Audit+robots+txt+and+xml+sitemap+issues:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76b0bd6e-4a5c-472f-a473-a315e3738936",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #advertools #xml #sitemap #website #audit #seo #robots.txt #google"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1089c7e4-b7a9-48ab-8320-e17847ad3b30",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Elias Dabbas](https://www.linkedin.com/in/eliasdabbas/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3293028-a492-463c-9e9e-96b2f8dcfcc3",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Last update:** 2023-05-30 (Created: 2023-05-29)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d93cd7a5-1d4f-4bc6-94a2-b74461c7f538",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Description:** This notebook helps you check if there are any conflicts between robots.txt rules and your XML sitemap.\n",
    "\n",
    "* Are you disallowing URLs that you shouldn't?\n",
    "* Test and make sure you don't publish new pages with such conflicts.\n",
    "* Do this in bulk: for all URL/rule/user-agent combinations run all tests with one command."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d84a0cf-6ba8-456a-8b42-1bf63e3a17f2",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**References:**\n",
    "- [advertools robots.txt functions](https://advertools.readthedocs.io/en/master/advertools.robotstxt.html)\n",
    "- [Google's robots reference](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)\n",
    "- [advertools XML sitemaps](https://advertools.readthedocs.io/en/master/advertools.sitemaps.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75ef3a2-b261-45a8-bd64-52cb78b5f502",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "460f75f0-18d1-45a3-8695-31fc87c3a56d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Install libraries\n",
    "If running it on naas, run the code below to install the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8178986f-8dae-4f76-b16f-803be2287d77",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "#!pip install advertools adviz pandas==1.5.3 --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f530c66d-1a5b-47d1-8d80-01f21a83ac3e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1e8d108a-44de-44b4-941f-f12795838c5c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "import advertools as adv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d35a9e6f-71c4-4cb2-b9bc-32c23b18c4ba",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup Variables\n",
    "- `robotstxt_url`: URL of the robots.txt file to convert to a `DataFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cebcf1a5-23ce-49ca-aee9-aac7ff28a2e4",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "robotstxt_url = \"https://www.youtube.com/robots.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ad6e918-ae61-4a78-9b71-a6ad5f530f53",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dafe9992-9fcb-4849-a69e-ced13c995aff",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Analyze potential robots.txt and XML conflicts\n",
    "\n",
    "Getting the robots.txt file and converting it to a `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4949b9c4-f852-47f5-9889-3ccce1a0100a",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "robots_df = adv.robotstxt_to_df(robotstxt_url=robotstxt_url)\n",
    "robots_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f205baa-8d88-41a9-b860-b429b195077e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "Get XML sitemap(s) and convert to a `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "89f415ef-b5ec-42f8-932d-7d01cb2fea07",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "sitemap = adv.sitemap_to_df(\n",
    "    # the function will extract and combine all available sitemaps\n",
    "    # in the robots.txt file\n",
    "    robotstxt_url,\n",
    "    max_workers=8,\n",
    "    recursive=True)\n",
    "sitemap"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd817594-1b7f-4d02-a245-9e1dcd6ad2b6",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "#### Testing robots.txt\n",
    "For all URL/user-agent combinations check if the URL is blocked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9d47b183-7e9d-4013-8f64-f15568513676",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "user_agents = robots_df[robots_df['directive'].str.contains('user-agent', case=False)]['content']\n",
    "user_agents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd8071bb",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "Generate the robots.txt test report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a169f1a7-2e73-4a70-9de8-0bcd2ce321f1",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get users agent\n",
    "user_agents = robots_df[robots_df['directive'].str.contains('user-agent', case=False)]['content']\n",
    "print(user_agents)\n",
    "\n",
    "# Testing robots.txt\n",
    "robots_report = adv.robotstxt_test(\n",
    "    robotstxt_url=robotstxt_url,\n",
    "    user_agents=user_agents,\n",
    "    urls=sitemap['loc'].dropna()\n",
    ")\n",
    "\n",
    "print(\"Row fetched:\", len(robots_report))\n",
    "robots_report.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4f99c99-dd8b-4893-8d5e-7ce5bfa29d1d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "Does the website have URLs listed in the XML sitemap that are also disallowed by its robots.txt?\n",
    "\n",
    "(this is not necessarily a problem, because they might disallow it for some user-agents only), and it's good to check."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "550110a8-50b2-49a0-9752-5132256356b9",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output\n",
    "\n",
    "Get the URLs that cannot be fetched"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24df9dac-a42c-4b46-9344-4cbc48cd23ce",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Filter result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7b6a8e95-fd71-4602-9a29-ccb4bb52c9e2",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_report = robots_report[~robots_report['can_fetch']].reset_index(drop=True)\n",
    "print(\"Row fetched:\", len(df_report))\n",
    "df_report.head(5)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "naas": {
   "notebook_id": "2733269de8aa4a8a013af03ae7af3ac21d9d860849af4b06adb3b31fbfc13ad2",
   "notebook_path": "Advertools/Advertools_Audit_robots_txt_and_xml_sitemap_issues.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.4.0"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}