ir_and_boolean_model.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2+"
    },
    "colab": {
      "name": "02_boolean_model.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TurkuNLP/Text_Mining_Course/blob/master/ir_and_boolean_model.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q-V1vx62ntdg"
      },
      "source": [
        "# Information Retrieval\n",
        "\n",
        "* Text search engines\n",
        "* We will go through this topic in several steps\n",
        "  * Term-document matrices\n",
        "  * Dense vector representations\n",
        "  * Solr \"industry-grade\" search engine setup and use"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "o9TPr7snnRCH"
      },
      "source": [
        "**Disclaimer:** some of the material is borrowed with thanks from http://www.cis.lmu.de/~hs/teach/14s/ir/\n",
        "\n",
        "# Boolean model\n",
        "\n",
        "* The Boolean model is arguably the simplest model to\n",
        "  base an information retrieval system on.\n",
        "* Queries are Boolean expressions, e.g., *Caesar* and *Brutus*\n",
        "* The seach engine returns all documents that satisfy the Boolean expression, but there are exceptions:\n",
        "  * e.g. page contains variant of query words: morphology, spelling correction, synonyms\n",
        "\n",
        "## Term-document matrix\n",
        "\n",
        "* Rows: terms\n",
        "* Columns: document\n",
        "* $M_{ij}=1$ if term *i* present in document *j*\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "1ICwFimQnRCO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9011ef83-bea5-498a-9e9f-999fe261e861"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "documents=[\"This is a silly example\",\"A better example\",\"Nothing to see here\",\"This is a great and long example\"]\n",
        "cv=CountVectorizer(lowercase=False,binary=True)\n",
        "print(\"Term-document matrix:\\n\")\n",
        "td_matrix=cv.fit_transform(documents).todense().T   #.T transposes the matrix, sklearn maintains document-term\n",
        "print(td_matrix)\n",
        "print(\"\\nIDX -> terms mapping:\\n\")\n",
        "print(cv.get_feature_names())\n",
        "print(\"\\nterm -> IDX mapping:\\n\")\n",
        "print(cv.vocabulary_) # note the _ at the end\n"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Term-document matrix:\n",
            "\n",
            "[[0 0 1 0]\n",
            " [1 0 0 1]\n",
            " [0 0 0 1]\n",
            " [0 1 0 0]\n",
            " [1 1 0 1]\n",
            " [0 0 0 1]\n",
            " [0 0 1 0]\n",
            " [1 0 0 1]\n",
            " [0 0 0 1]\n",
            " [0 0 1 0]\n",
            " [1 0 0 0]\n",
            " [0 0 1 0]]\n",
            "\n",
            "IDX -> terms mapping:\n",
            "\n",
            "['Nothing', 'This', 'and', 'better', 'example', 'great', 'here', 'is', 'long', 'see', 'silly', 'to']\n",
            "\n",
            "term -> IDX mapping:\n",
            "\n",
            "{'This': 1, 'is': 7, 'silly': 10, 'example': 4, 'better': 3, 'Nothing': 0, 'to': 11, 'see': 9, 'here': 6, 'great': 5, 'and': 2, 'long': 8}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "x4KMWUGinRCR"
      },
      "source": [
        "* Every row is an *incidence vector* of a term - which documents the term appears in\n",
        "* Boolean retrieval in its simplest form:\n",
        "  * Boolean operations on incidence vectors"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "mc4nsEFGnRCS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "13eb4c25-9253-4bd3-cbb3-efc4e4724663"
      },
      "source": [
        "t2i=cv.vocabulary_\n",
        "print(\"example\")\n",
        "print(td_matrix[t2i[\"example\"]])\n",
        "print()\n",
        "print(\"example and great\")\n",
        "print(td_matrix[t2i[\"example\"]] & td_matrix[t2i[\"great\"]])\n",
        "print()\n",
        "print(\"not example\")\n",
        "print(1-td_matrix[t2i[\"example\"]]) #1-x does the negation in our case\n",
        "print()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "example\n",
            "[[1 1 0 1]]\n",
            "\n",
            "example and great\n",
            "[[0 0 0 1]]\n",
            "\n",
            "not example\n",
            "[[0 0 1 0]]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "MB3RwEa1nRCU"
      },
      "source": [
        "We can piece it together into an elementary search engine:\n",
        "\n",
        "* Accept queries like \"( not example or great ) and Nothing\"\n",
        "* Rewrite them into a Python expression\n",
        "* eval() that expression"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "eX2wF-RknRCX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f043d3dd-9ea2-492f-f952-eb128da6830f"
      },
      "source": [
        "# Operators and, or, not become &, |, 1 -\n",
        "# Parentheses are left untouched\n",
        "# Everything else interpreted as a term and fed through td_matrix[t2i[\"...\"]]\n",
        "\n",
        "d={\"and\":\"&\",\"or\":\"|\",\"not\":\"1 -\",\"(\":\"(\",\")\":\")\"} # operator replacements\n",
        "def rew_token(t):\n",
        "    return d.get(t,f'td_matrix[t2i[\"{t}\"]]') #if it is operator, replace, if not, then replace with td_matrix[t2i[\"{t}\"]]\n",
        "\n",
        "def rew_query(query): #rewrite every token in the query\n",
        "    return \" \".join(rew_token(t) for t in query.split())\n",
        "\n",
        "def test_query(query):\n",
        "    print(\"Query:\\\"\"+query+\"\\\"\")\n",
        "    print(\"Rewritten:\",rew_query(query))\n",
        "    print(\"Matching:\",eval(rew_query(query)))\n",
        "    print()\n",
        "\n",
        "test_query(\"example and not Nothing\")\n",
        "test_query(\"not example or great\")\n",
        "test_query(\"( not example or great ) and Nothing\")\n"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Query:\"example and not Nothing\"\n",
            "Rewritten: td_matrix[t2i[\"example\"]] & 1 - td_matrix[t2i[\"Nothing\"]]\n",
            "Matching: [[1 1 0 1]]\n",
            "\n",
            "Query:\"not example or great\"\n",
            "Rewritten: 1 - td_matrix[t2i[\"example\"]] | td_matrix[t2i[\"great\"]]\n",
            "Matching: [[0 0 1 1]]\n",
            "\n",
            "Query:\"( not example or great ) and Nothing\"\n",
            "Rewritten: ( 1 - td_matrix[t2i[\"example\"]] | td_matrix[t2i[\"great\"]] ) & td_matrix[t2i[\"Nothing\"]]\n",
            "Matching: [[0 0 1 0]]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "wJkanF3dnRCY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "247d78d2-4236-4b2d-d925-4f51012e92ec"
      },
      "source": [
        "# Should match all documents, but crashes instead\n",
        "# This one you will fix as an exercise\n",
        "import traceback\n",
        "try:\n",
        "    test_query(\"not awordwhichdoesnotexist\") #should match all documents\n",
        "except:\n",
        "    traceback.print_exc() #I only need to do this because of IPython"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Query:\"not awordwhichdoesnotexist\"\n",
            "Rewritten: 1 - td_matrix[t2i[\"awordwhichdoesnotexist\"]]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Traceback (most recent call last):\n",
            "  File \"<ipython-input-25-a02963e0a344>\", line 5, in <module>\n",
            "    test_query(\"not awordwhichdoesnotexist\") #should match all documents\n",
            "  File \"<ipython-input-24-ade8113ddedf>\", line 15, in test_query\n",
            "    print(\"Matching:\",eval(rew_query(query)))\n",
            "  File \"<string>\", line 1, in <module>\n",
            "KeyError: 'awordwhichdoesnotexist'\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "MNzTVUaynRCZ"
      },
      "source": [
        "* We now have a rudimentary boolean IR system\n",
        "* It is not that great but ready in < 10 lines of code in Python"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "Z81tk-KrnRCZ"
      },
      "source": [
        "# Size constraints\n",
        "\n",
        "* Term-document matrix is vocabulary size times document set size\n",
        "* 1M documents, each 1000 words\n",
        "* 1,000,000 x 1,000 = 1,000,000,000 words of running text\n",
        "* 6 bytes per word -> ~6GB in size\n",
        "* Assume 500,000 unique terms\n",
        "* Term-document matrix has 1,000,000 x 500,000 / 8 / 1024 / 1024 / 1024 -> ~60GB\n",
        "* 60GB of space to index a collection of 6GB of text!\n",
        "* 480GB if we were to use 1B integers to remember the 0/1 values\n",
        "* ...but most of this are zeros...\n",
        "\n",
        "## Sparse representation\n",
        "\n",
        "* Only remember the non-zero entries\n",
        "* For every term, remember a (usually sorted) list of documents in which it appears\n",
        "* This is the famous **inverted index**\n",
        "* Scipy sparse formats: https://docs.scipy.org/doc/scipy/reference/sparse.html\n",
        "  * *CSC* for every column remember the list of rows\n",
        "  * *CSR* for every row remember the list of columns\n",
        "  \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "scrolled": false,
        "id": "b-bC3vAFnRCa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f361f030-64b1-4afd-c947-3633edfa7459"
      },
      "source": [
        "documents=[\"This is a silly example\",\"A better example\",\"Nothing to see here\",\"This is a great and long example\"]\n",
        "cv=CountVectorizer(lowercase=False,binary=True)\n",
        "# Exact same code as above, but I removed the .todense()\n",
        "td_matrix=cv.fit_transform(documents)\n",
        "print(\"scikit's document-term matrix:\")\n",
        "print(td_matrix)\n",
        "print()\n",
        "print(\"Transposed: (note incorrect sort)\")\n",
        "print(td_matrix.T)\n",
        "print()\n",
        "print(\"Transposed, and in the correct sparse format:\")\n",
        "print(td_matrix.T.tocsr())\n",
        "td_matrix=td_matrix.T.tocsr()"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "scikit's document-term matrix:\n",
            "  (0, 1)\t1\n",
            "  (0, 7)\t1\n",
            "  (0, 10)\t1\n",
            "  (0, 4)\t1\n",
            "  (1, 4)\t1\n",
            "  (1, 3)\t1\n",
            "  (2, 0)\t1\n",
            "  (2, 11)\t1\n",
            "  (2, 9)\t1\n",
            "  (2, 6)\t1\n",
            "  (3, 1)\t1\n",
            "  (3, 7)\t1\n",
            "  (3, 4)\t1\n",
            "  (3, 5)\t1\n",
            "  (3, 2)\t1\n",
            "  (3, 8)\t1\n",
            "\n",
            "Transposed: (note incorrect sort)\n",
            "  (1, 0)\t1\n",
            "  (7, 0)\t1\n",
            "  (10, 0)\t1\n",
            "  (4, 0)\t1\n",
            "  (4, 1)\t1\n",
            "  (3, 1)\t1\n",
            "  (0, 2)\t1\n",
            "  (11, 2)\t1\n",
            "  (9, 2)\t1\n",
            "  (6, 2)\t1\n",
            "  (1, 3)\t1\n",
            "  (7, 3)\t1\n",
            "  (4, 3)\t1\n",
            "  (5, 3)\t1\n",
            "  (2, 3)\t1\n",
            "  (8, 3)\t1\n",
            "\n",
            "Transposed, and in the correct sparse format:\n",
            "  (0, 2)\t1\n",
            "  (1, 0)\t1\n",
            "  (1, 3)\t1\n",
            "  (2, 3)\t1\n",
            "  (3, 1)\t1\n",
            "  (4, 0)\t1\n",
            "  (4, 1)\t1\n",
            "  (4, 3)\t1\n",
            "  (5, 3)\t1\n",
            "  (6, 2)\t1\n",
            "  (7, 0)\t1\n",
            "  (7, 3)\t1\n",
            "  (8, 3)\t1\n",
            "  (9, 2)\t1\n",
            "  (10, 0)\t1\n",
            "  (11, 2)\t1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "-yxQVGd8nRCb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eb07f545-3a67-4cd4-d24b-6690f11cd806"
      },
      "source": [
        "# The sparse representations do not allow many of the necessary operations, so we need\n",
        "# to make the rows dense, once we retrieve them, not a huge deal for our toy examples\n",
        "def rew_token(t):\n",
        "    return d.get(t,f'td_matrix[t2i[\"{t}\"]].todense()')\n",
        "\n",
        "test_query(\"example and not Nothing\")\n",
        "test_query(\"( not example or great ) and Nothing\")"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Query:\"example and not Nothing\"\n",
            "Rewritten: td_matrix[t2i[\"example\"]].todense() & 1 - td_matrix[t2i[\"Nothing\"]].todense()\n",
            "Matching: [[1 1 0 1]]\n",
            "\n",
            "Query:\"( not example or great ) and Nothing\"\n",
            "Rewritten: ( 1 - td_matrix[t2i[\"example\"]].todense() | td_matrix[t2i[\"great\"]].todense() ) & td_matrix[t2i[\"Nothing\"]].todense()\n",
            "Matching: [[0 0 1 0]]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "34KqWic0nRCc"
      },
      "source": [
        "## Boolean retrieval (cont.)\n",
        "\n",
        "* The code above needs `.todense()` to perform the *and / or / not* arithmetics -> inefficient, but simple\n",
        "* Other option - make sure the lists of documents are sorted\n",
        "* AND - intersection of two sorted lists\n",
        "  * Walk the lists\n",
        "  * Linear complexity in terms of number of documents\n",
        "  * A good exercise - write a function which takes two lists like those above, and computes their intersection as a new list"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "fsGy8x1anRCc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc6264ad-1728-49ae-fff6-80f347a9b8b1"
      },
      "source": [
        "print(\"Documents for 'example'\")\n",
        "print(td_matrix[t2i[\"example\"]].nonzero()[1])\n",
        "print()\n",
        "print(\"Documents for 'great'\")\n",
        "print(td_matrix[t2i[\"great\"]].nonzero()[1])"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Documents for 'example'\n",
            "[0 1 3]\n",
            "\n",
            "Documents for 'great'\n",
            "[3]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "aDsdcUiznRCd"
      },
      "source": [
        "# Phrase and proximity queries\n",
        "\n",
        "* *\"Stanford university\"* - not the same thing as *Stanford AND university*\n",
        "* The basic inverted index of no help\n",
        "\n",
        "## biword index\n",
        "\n",
        "* Index all word bigrams\n",
        "* \"Stanford university\" becomes a term\n",
        "* \"to be or not to be\" -> \"to be\" and \"be or\" and \"or not\" and \"not to\" + postprocessing\n",
        "* Not a great solution:\n",
        "  * Massive waste of space: blows up the index size quadratically\n",
        "  * Needs postprocessing to weed out the false hits\n",
        "\n",
        "## positional index\n",
        "\n",
        "* For every term, and every document, remember a list of the positions, not just \"1\"\n",
        "* A modification of the sorted list intersection algorithm to answer the query\n",
        "* Also allows proximity queries \"X within N words from Y\"\n",
        "* Quite heavy computationally\n",
        "\n",
        "## combined index\n",
        "\n",
        "* Most common biwords indexed directly\n",
        "* Rest solved with positional indexing\n",
        "* Compromise between the two"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "l6bzYdw4nRCe"
      },
      "source": [
        "# Result ranking\n",
        "\n",
        "* Obviously useful for large document collections\n",
        "* Come up with a number describing the fit of a document to a query\n",
        "* Return top-N documents\n",
        "* Basic observations:\n",
        "  * The more query terms hit, the more relevant the document is\n",
        "  * The more times the query terms hit in the document, the more relevant the document is\n",
        "  * Rare terms are more informative than common terms\n",
        "* Firstly, we don't want to store 0/1, we want to store the count"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "scrolled": false,
        "id": "ToirASDGnRCf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1a92530c-f3bc-4a91-8d47-fb19aecf202f"
      },
      "source": [
        "documents=[\"This is a silly silly silly example\",\"A better example\",\"Nothing to see here nor here nor here\",\"This is a great example and a long example too\"]\n",
        "cv=CountVectorizer(lowercase=False)\n",
        "# Exact same code as above, but now I also removed binary=True\n",
        "td_matrix=cv.fit_transform(documents)\n",
        "t2i=cv.vocabulary_\n",
        "td_matrix=td_matrix.T.tocsr()\n",
        "print(td_matrix.todense())"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0 0 1 0]\n",
            " [1 0 0 1]\n",
            " [0 0 0 1]\n",
            " [0 1 0 0]\n",
            " [1 1 0 2]\n",
            " [0 0 0 1]\n",
            " [0 0 3 0]\n",
            " [1 0 0 1]\n",
            " [0 0 0 1]\n",
            " [0 0 2 0]\n",
            " [0 0 1 0]\n",
            " [3 0 0 0]\n",
            " [0 0 1 0]\n",
            " [0 0 0 1]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "dxDHkYA5nRCh"
      },
      "source": [
        "* Sum up the counts across query hits"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "u3zh17LsnRCh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1dc04d8c-5e47-4ce0-f640-3db1a824b1bc"
      },
      "source": [
        "print(\"Documents for 'example' or 'better'\")\n",
        "hits=td_matrix[t2i[\"example\"]]+td_matrix[t2i[\"better\"]]\n",
        "print(\"Hits:\",hits,sep=\"\\n\")\n",
        "print(\"Hits nonzero\", hits.nonzero())\n",
        "print(\"Documents:\", hits.nonzero()[1])\n",
        "print(\"Scores:\", hits[hits.nonzero()].A1) #A1 returns itself as flat array\n"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Documents for 'example' or 'better'\n",
            "Hits:\n",
            "  (0, 0)\t1\n",
            "  (0, 1)\t2\n",
            "  (0, 3)\t2\n",
            "Hits nonzero (array([0, 0, 0], dtype=int32), array([0, 1, 3], dtype=int32))\n",
            "Documents: [0 1 3]\n",
            "Scores: [1 2 2]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "RWz4EbZ8nRCh"
      },
      "source": [
        "* Now we can sort on scores and get an order on documents\n",
        "* Now a document with 2x example hits equally well as document with 1x example and 1x better\n",
        "* Maybe not optimal?\n",
        "* A document with a term occurring 10x is more important, but not ten times so\n",
        "* Usual solution: squeeze the counts through log or similar function (done already at index time)\n",
        "  * $1+log_{10}(tf)$ is a typical formula used"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "scrolled": false,
        "id": "iTVM0uMhnRCi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cda2dd5b-3d6c-43ea-92ac-4c469d22e6d7"
      },
      "source": [
        "# Switch from CountVectorizer to TfidfVectorizer\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "# Parameters with which TfidfVectorizer does same thing as CountVectorizer\n",
        "tfv=TfidfVectorizer(lowercase=False,sublinear_tf=False,use_idf=False,norm=None,binary=False)\n",
        "td_matrix=tfv.fit_transform(documents)\n",
        "t2i=tfv.vocabulary_\n",
        "td_matrix=td_matrix.T.tocsr()\n",
        "print(\"Just to check we get same numbers as with CountVectorizer:\")\n",
        "print(td_matrix.todense())\n",
        "\n",
        "# Turn log-squeeze on\n",
        "tfv2=TfidfVectorizer(lowercase=False,sublinear_tf=True,use_idf=False,norm=None)\n",
        "td_matrix2=tfv2.fit_transform(documents)\n",
        "t2i=tfv2.vocabulary_\n",
        "td_matrix2=td_matrix2.T.tocsr()\n",
        "print(td_matrix2.todense())\n"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Just to check we get same numbers as with CountVectorizer:\n",
            "[[0. 0. 1. 0.]\n",
            " [1. 0. 0. 1.]\n",
            " [0. 0. 0. 1.]\n",
            " [0. 1. 0. 0.]\n",
            " [1. 1. 0. 2.]\n",
            " [0. 0. 0. 1.]\n",
            " [0. 0. 3. 0.]\n",
            " [1. 0. 0. 1.]\n",
            " [0. 0. 0. 1.]\n",
            " [0. 0. 2. 0.]\n",
            " [0. 0. 1. 0.]\n",
            " [3. 0. 0. 0.]\n",
            " [0. 0. 1. 0.]\n",
            " [0. 0. 0. 1.]]\n",
            "[[0.         0.         1.         0.        ]\n",
            " [1.         0.         0.         1.        ]\n",
            " [0.         0.         0.         1.        ]\n",
            " [0.         1.         0.         0.        ]\n",
            " [1.         1.         0.         1.69314718]\n",
            " [0.         0.         0.         1.        ]\n",
            " [0.         0.         2.09861229 0.        ]\n",
            " [1.         0.         0.         1.        ]\n",
            " [0.         0.         0.         1.        ]\n",
            " [0.         0.         1.69314718 0.        ]\n",
            " [0.         0.         1.         0.        ]\n",
            " [2.09861229 0.         0.         0.        ]\n",
            " [0.         0.         1.         0.        ]\n",
            " [0.         0.         0.         1.        ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "wv-PVRMVnRCj"
      },
      "source": [
        "* and now the *example* and *better* document should rank above the *2x example* document"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "_KPEpYd0nRCk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ba8846d4-39e4-40bb-f2ed-6219d53c3eb6"
      },
      "source": [
        "print(\"Documents for 'example' and 'better'\")\n",
        "hits=td_matrix2[t2i[\"example\"]]+td_matrix2[t2i[\"better\"]]\n",
        "print(\"Hits:\",hits,sep=\"\\n\")\n",
        "print(\"Documents:\", hits.nonzero()[1])\n",
        "print(\"Scores:\", hits[hits.nonzero()].A1) #A1 returns itself as flat array"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Documents for 'example' and 'better'\n",
            "Hits:\n",
            "  (0, 0)\t1.0\n",
            "  (0, 1)\t2.0\n",
            "  (0, 3)\t1.6931471805599454\n",
            "Documents: [0 1 3]\n",
            "Scores: [1.         2.         1.69314718]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "Rr0qBQHRnRCl"
      },
      "source": [
        "## Informativeness of terms\n",
        "\n",
        "* Not all words are equally informative\n",
        "* Rare words are clearly much more informative than common ones\n",
        "* The query *\"jabberwocky movie cast\"* should put more weight on *jabberwocky*\n",
        "* What we want:\n",
        "  * High positive weight for rare terms\n",
        "  * Low positive weight for common terms\n",
        "* Typical way: IDF *inverse document frequency*\n",
        "  * df_t is the *document frequency* of *t* - number of documents *t* appear in\n",
        "  * $IDF_t=\\frac{N}{df_t}$ inverse document frequency of *t*\n",
        "  * Usually one would squeeze this through log so\n",
        "  * $IDF_t=log_{10}(\\frac{N}{df_t})$\n",
        "\n",
        "## Tf.Idf\n",
        "\n",
        "* An extremely common weighting scheme in IR\n",
        "* Product of term's tf (log-squeezed) with the term's idf (log-squeezed)\n",
        "* Gives a score of that term's hit in a given document\n",
        "  * $(1+log_{10}tf_t)\\cdot log_{10}\\frac{N}{df_t}$\n",
        "  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "m4DtGVkcnRCm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "56a74356-bda7-4d8d-a532-39aaab3566a5"
      },
      "source": [
        "# same as above, but use_idf=True\n",
        "tfv=TfidfVectorizer(lowercase=False,sublinear_tf=True,use_idf=True,norm=None)\n",
        "td_matrix=tfv.fit_transform(documents)\n",
        "t2i=cv.vocabulary_\n",
        "td_matrix=td_matrix.T.tocsr()\n",
        "print(td_matrix.todense())"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[0.         0.         1.91629073 0.        ]\n",
            " [1.51082562 0.         0.         1.51082562]\n",
            " [0.         0.         0.         1.91629073]\n",
            " [0.         1.91629073 0.         0.        ]\n",
            " [1.22314355 1.22314355 0.         2.07096206]\n",
            " [0.         0.         0.         1.91629073]\n",
            " [0.         0.         4.02155128 0.        ]\n",
            " [1.51082562 0.         0.         1.51082562]\n",
            " [0.         0.         0.         1.91629073]\n",
            " [0.         0.         3.24456225 0.        ]\n",
            " [0.         0.         1.91629073 0.        ]\n",
            " [4.02155128 0.         0.         0.        ]\n",
            " [0.         0.         1.91629073 0.        ]\n",
            " [0.         0.         0.         1.91629073]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "4_jZXxXmnRCn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2d047e5c-31f4-4833-b92f-a548bedac051"
      },
      "source": [
        "print(\"Documents for 'example' or 'better' with full tf.idf weighting\")\n",
        "hits=td_matrix[t2i[\"example\"]]+td_matrix[t2i[\"better\"]]\n",
        "print(\"Hits:\",hits,sep=\"\\n\")\n",
        "print(\"Documents:\", hits.nonzero()[1])\n",
        "print(\"Scores:\", hits[hits.nonzero()].A1) #A1 returns itself as flat array"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Documents for 'example' or 'better' with full tf.idf weighting\n",
            "Hits:\n",
            "  (0, 0)\t1.2231435513142097\n",
            "  (0, 1)\t3.139434283188365\n",
            "  (0, 3)\t2.0709620553277333\n",
            "Documents: [0 1 3]\n",
            "Scores: [1.22314355 3.13943428 2.07096206]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "18qJz18RnRCn"
      },
      "source": [
        "* Doesn't work on toy examples, really, let's index a bit more\n",
        "* http://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OkLcnreGLa4s",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "61b4e563-d17d-4e86-f80f-d08fc3530937"
      },
      "source": [
        "!wget -nc -O fiwiki.xml.bz2 https://www.dropbox.com/s/r82qnfdj1encx1z/fiwiki-20181001-corpus.xml.bz2?dl=1"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "File ‘fiwiki.xml.bz2’ already there; not retrieving.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tgsgczMJLqmP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "83b875e9-b221-47a0-9476-556d3147f91e"
      },
      "source": [
        "!wget -nc -O xml2txt.pl https://www.dropbox.com/s/p3ta9spzfviovk0/xml2txt.pl?dl=0\n",
        "!bzcat fiwiki.xml.bz2 | perl xml2txt.pl -articles /dev/stdin /dev/stdout  | head -n 100000 | tqdm | gzip > fiwiki.txt.gz"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "File ‘xml2txt.pl’ already there; not retrieving.\n",
            "preserving articles boundaries\n",
            "100000it [00:05, 18520.24it/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "scrolled": false,
        "id": "IqyrVBl5nRCn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "927d249e-9a9e-40d8-f796-75a153c5d9e0"
      },
      "source": [
        "! zcat fiwiki.txt.gz | head -n 40"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<article name=\"Amsterdam\">\n",
            "Amsterdam on Alankomaiden pääkaupunki. Amsterdam on väkiluvultaan Alankomaiden suurin kaupunki, huhtikuun alussa 2006 siellä asui 743 905 asukasta eli noin joka 20. hollantilainen asuu Amsterdamissa. Yhteensä Amsterdamissa ja sitä ympäröivällä kaupunkialueella asuu noin 1 450 000 ihmistä eli vajaa kymmenesosa Alankomaiden asukkaista. Amsterdam sijaitsee Amstelin suistossa IJsselmeerin rannalla Alankomaiden Pohjois-Hollannin provinssissa. Vaikka Amsterdam on Alankomaiden perustuslain mukaan maan pääkaupunki, sijaitsevat niin kuningashuone, hallitus, parlamentti kuin korkein oikeuskin Haagissa.Maantiede ja ilmasto\n",
            "Amsterdam sijaitsee tasaisella alankoalueella, ja osa siitä on merenpinnan tason alapuolella. Kaupunki on IJsselmeeriin kuuluvan IJ’n etelärannalla. Amsteljoki virtaa kaupungin läpi. Vauraus johti myös taiteen ja luonnontieteen kehitykseen. Rembrandt, Frans Hals, Vermeer ja Paulus Potter vaikuttivat siihen aikaan, ja heidän työnsä olivat erittäin kysyttyjä vauraamman väestönosan keskuudessa.\n",
            "1700-luvulla Amsterdamista tuli maailman suurin pankkikeskus, mutta kulta-aika oli loppumassa. Todellinen rappio alkoi Ison-Britannian tuhottua maan laivaston sodassa vuonna 1780. Myös Yhdistynyt Itä-Intian kauppakomppania (VOC) teki vararikon vuonna 1791. Neljä vuotta myöhemmin Napoleon valloitti maan ja se oli taas vieraan vallassa.\n",
            "1800-luvulla tilanne Amsterdamissa parani, kun Ranskan valta oli alkanut heiketä. Vuosisadan loppupuolella Amsterdamiin perustettiin suuria puistoja ja museoita, ja elintaso kohosi huomattavasti.\n",
            "Alankomaat pysyi ensimmäisen maailmansodan puolueettomana. 1900-luvun alussa ponnistelut keskittyivät lähinnä maankuivaukseen, jolla pyrittiin lisäämään maataloutta ja elintilaa. Toisessa maailmansodassa maa kuitenkin vallattiin vuonna 1940 ja Amsterdamin juutalaisia vietiin keskitysleireihin, joista he eivät ikinä palanneet. Väestö alkoi nähdä nälkää, kun elintarvikkeiden ja polttoaineen jakelu lopetettiin. Maa vapautettiin vasta vuonna 1945.\n",
            "Toisen maailmansodan jälkeen Amsterdamin suvaitsevaisuus houkutteli paljon hippejä ja muita vaihtoehtokulttuurien edustajia. Kun kaupungin vanhoja rakennuksia oltiin purkamassa, aiheutti tämä suuren vastalausemyrskyn. Näin ollen esimerkiksi Jordaanin alue säästyi. Sittemmin Amsterdamista on kehittynyt yksi maailman suosituimmista kaupunkimatkakohteista, johtuen varmasti historiallisesta keskustasta ja ainutlaatuisesta taidehistoriasta. Nykypäivänä amsterdamilaisten huolenaiheena on ilmaston lämpenemisestä johtuva vedenpinnan nousu, joka uhkaa nousta kriittiselle tasolle. Alankomaissa on tehtävä paljon töitä sen eteen.Hallinto\n",
            "Amsterdamia johtaa muiden Alankomaiden kuntien tapaan valtuusto. Amsterdamin kaupunginvaltuustoon valitaan edustajat neljän vuoden välein. Siihen kuuluu 45 jäsentä. Valtuusto puolestaan valitsee keskuudestaan 6–8 raatimiestä raatimieskollegioon, joka esittelee ja toimeenpanee valtuuston päätökset. Valtuusto voi kuitenkin myös hylätä kollegion päätökset. Pormestari on sekä kaupunginvaltuuston että raatimieskollegion puheenjohtaja, mutta hänellä ei ole äänioikeutta valtuustossa.\n",
            "Kaupunginosien määrää on Amsterdamissa vähennetty 1900-luvun lopulta alkaen. Vielä vuonna 1990 niitä oli 16, mutta vuoden 2010 liitoksen jälkeen enää seitsemän. Viimeisimpänä asukasluvultaan pieni Westpoort siirrettiin suoraan keskushallinnon alaisuuteen. Amsterdamin kaupunginosat ovat:\n",
            "Amsterdam-Centrum\n",
            "Amsterdam-Noord\n",
            "Amsterdam Nieuw-West\n",
            "Amsterdam-Oost\n",
            "Amsterdam-West\n",
            "Amsterdam-Zuid\n",
            "Amsterdam ZuidoostTalous\n",
            "Amsterdamin talous perustuu palveluihin, ja vain kymmenesosa työskentelee valmistusteollisuudesta. Kaikista työpaikoista noin viidennes on kiinni kansainvälisessä kaupassa ja liikenteessä. Muita merkittäviä palvelualoja ovat pankki- ja vakuutusala, terveys-, kulttuuri- ja yhteiskunnalliset palvelut sekä matkailu. Amsterdam on kansainvälisen rahatalouden keskus, ja kaupungissa on muun muassa pörssi ja useiden kansainvälisten pankkien toimistoja.\n",
            "Amsterdamin teollisuuslaitokset ovat monipuolisia, vaikka niiden osuus taloudesta on pienentynyt koko ajan. Amsterdamissa on muun muassa laivanrakennusta, öljynjalostusta, ruoan jalostusta ja timanttien hiontaa.Liikenne\n",
            "Amsterdam on yksi pyöräily-ystävällisimmistä suurkaupungeista koko maailmassa ja se on pyöräilykulttuurin keskus. Vuonna 2011 Amsterdamissa oli arviolta yli 900 000 polkupyörää. Pyörävarkaudet ovat yleisiä; Amsterdamissa varastettiin vuonna 2005 noin 54 000 polkupyörää. Laajan pyörätieverkon ja kaupungin pienen koon takia pyöräily on suurimmalle osalle kaupunkilaisista helpoin tapa liikkua paikasta toiseen. Autojen parkkimaksut kerätään parkkirahastoon ja kunnan liikennerahastoon, joita käytetään autoilun vähentämiseen ja autoilun päästöjen vähentämiseen.\n",
            "Julkinen liikenne Amsterdamissa koostuu lähinnä bussi- ja raitiovaunuverkosta. Jotkut vesikulkuneuvot ovat myös vuokrattavissa. Kaupungissa on myös useita ilmaisia lauttayhteyksiä Amstel-joen yli.\n",
            "Lentoasema\n",
            "Vajaan 18 kilometrin etäisyydellä Amsterdamista sijaitsee Schipholin lentoasema. Se on Hollannin suurin ja matkustajamäärillä mitattuna Euroopan viidenneksi suurin lentoasema. Sen kautta kulki vuonna 2007 yli 47 miljoonaa matkustajaa. Lentoasema on kotikenttä. Lentokenttä on seitsemän metriä merenpinnan alapuolella.Väestö\n",
            "Vuonna 2015 Amsterdamissa oli Alankomaiden tilastokeskuksen mukaan 821 752 asukasta. Suurkaupunkialueella asui 1 313 889 ihmistä. Kaupungin väkiluku on ollut kasvussa 1990-luvun alun jälkeen. Vuonna 1990 Amsterdamissa asui noin 695 000 ihmistä. Sitä ennen väkimäärä oli kuitenkin laskenut usean vuosikymmenen ajan, sillä vuonna 1960 kaupungissa asui noin 870 000 ihmistä.\n",
            "Amsterdamin väestönkasvu on seurausta syntyvyydestä ja maahanmuuttajista. Noin puolet kaupungin asukkaista on syntyperäisiä alankomaalaisia. Kaupunkiin on tullut paljon siirtolaisia Alankomaiden entisistä siirtomaista, kuten Indonesiasta, Surinamista ja entisen Alankomaiden Antillien saarilta. Lisäksi kaupungissa on paljon vierastyöläisiä sekä turvapaikanhakijoita. Asukkaista kolmasosa on tullut Euroopan ulkopuolelta.Kulttuuri\n",
            "Amsterdam on pienuudestaan huolimatta merkittävä kulttuurikaupunki. Arkkitehtuuri on kaupungissa ainutlaatuista kanaalien hallitessa kaupunkikuvaa. Kaupungissa on useita merkittäviä museoita, muun muassa useita Rembrandtin töitä sisältävä valtionmuseo Rijksmuseum, Vincent van Goghin tuotantoon keskittyvä Van Gogh Museum sekä juutalaisvainojen ajoista kertova Anne Frankin talo.\n",
            "Kaupungissa toimii yksi maailman kuuluisimmista orkestereista, Concertgebouw-orkesteri, joka esiintyy Museumpleinin kupeessa sijaitsevassa Concertgebouw-konserttitalossa. Bimhuis (lähellä keskusrautatieasemaa) keskittyy jazz- sekä improvisoituun musiikkiin. Leidseplein on yksi yöelämän keskuksista, ja sen ympäristössä sijaitsevat Melkweg ja Paradiso ovat areenoita monien eri alojen esiintyjille.\n",
            "Kaupunki on kuuluisa monia vuosisatoja vanhasta suvaitsevaisuuden perinteestä ja siitä onkin muodostunut eurooppalaisen vapaamielisyyden vertauskuva. Prostituutio on laillista, ja punaisten lyhtyjen alue ( ), kannabistuotteita myyvät coffee shopit ja monipuolinen yöelämä (baareja ja kahviloita yhteensä noin 1 500) houkuttelevat runsaasti turisteja. Alankomaat ovat kuitenkin päättäneet rajoittaa kannabiksen käyttöä kahviloiden ottaessa käyttöön niin sanotut klubikortit, joita voidaan myöntää vain täysi-ikäisille hollantilaisille.\n",
            "Monumentteja ja rakennuksia\n",
            "Amsterdamin maaperä on huokoista suota. Kovaa hiekkamaata on vasta noin 13 metrin syvyydessä. 1600-luvulta alkaen talot tuettiin paaluttamalla niiden perustukset hiekkamaahan asti. Osa näistä puisista paaluista on mädäntynyt, jolloin talo saattaa vähitellen vajota. Turistikin havaitsee, että vanhat talot voivat olla hiukan vinossa. Vinouden syyksi on esitetty myös tahallista vinoon rakentamista, sillä kanavaan päin kaltevaan taloon voitiin nostaa tavaraa vinssillä helpommin. Tähän viittaa se, että vuonna 1565 määrättiin, että talojen kaltevuus saa olla enintään 1:25.\n",
            "Merkittäviä rakennuksia ja monumentteja:\n",
            "Koninklijk Paleis, kuninkaanlinna\n",
            "Schreierstoren, kyyneltentorni, joka on kaupunginmuurin säilynyt osa\n",
            "tuulimylly De Gooyer\n",
            "Trippenhuis, asekauppiassuvun talo\n",
            "Tiede- ja tekniikkakeskus NEMO, joka esittelee tieteen saavutuksia maallikoille\n",
            "Homomonument, joka on maailman ensimmäinen homoseksuaaleille omistettu muistomerkki, pystytetty toisen maailmansodan homoseksuaalivainojen muistoksi\n",
            "Museoita\n",
            "Amsterdams Historisch Museum\n",
            "Anne Frankin talo\n",
            "Joods Historisch Museum\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "8ni2WaY_nRCo"
      },
      "source": [
        "import gzip\n",
        "def articles(gzipfile,max_articles=1000):\n",
        "    \"\"\"A function to yield documents, one at a time\"\"\"\n",
        "    with gzip.open(gzipfile,\"rt\") as f:\n",
        "        article=[]\n",
        "        for line in f:\n",
        "            line=line.strip()\n",
        "            article.append(line)\n",
        "            if line==\"</article>\":\n",
        "                yield \" \".join(article)\n",
        "                max_articles-=1\n",
        "                if max_articles==0:\n",
        "                    break\n",
        "                article=[]\n",
        "# Index 50K articles from Wikipedia, turn on lowercasing too\n",
        "tfv_wiki=TfidfVectorizer(lowercase=True,sublinear_tf=True,use_idf=True,norm=None)\n",
        "td_matrix_wiki=tfv_wiki.fit_transform(articles(\"fiwiki.txt.gz\",50000))"
      ],
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "xvhc9Il_nRCo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4bfd9478-045e-49a3-f5bb-d3c69286dfb1"
      },
      "source": [
        "import numpy as np\n",
        "# I will need access to which term is at which position\n",
        "vocab_list=[None]*len(tfv_wiki.vocabulary_)\n",
        "for term,idx in tfv_wiki.vocabulary_.items():\n",
        "    vocab_list[idx]=term\n",
        "\n",
        "# Indirect sort, returns array of indices\n",
        "idf_sorted=np.argsort(tfv_wiki.idf_)\n",
        "# top-30\n",
        "for t_idx in idf_sorted[:30]:\n",
        "    print(tfv_wiki.idf_[t_idx],\"    \",vocab_list[t_idx])"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.0      name\n",
            "1.0      article\n",
            "1.030622860213983      ja\n",
            "1.0672936357339917      on\n",
            "1.1367110455419023      myös\n",
            "1.3669314441059215      joka\n",
            "1.4162372050900756      oli\n",
            "1.4480247225269602      ei\n",
            "1.4693689100876446      mukaan\n",
            "1.4821404757671321      sen\n",
            "1.509503746508075      ovat\n",
            "1.509503746508075      tai\n",
            "1.5268259651124318      että\n",
            "1.5349231753450512      se\n",
            "1.538996500732687      sekä\n",
            "1.5430864859842122      mutta\n",
            "1.580669197133255      vuonna\n",
            "1.6527062110014346      vuoden\n",
            "1.6603573577369546      eli\n",
            "1.6820973443733602      jonka\n",
            "1.694735743245083      kun\n",
            "1.7011153502091223      kuin\n",
            "1.7172447321390059      jälkeen\n",
            "1.763842180117427      jotka\n",
            "1.781024733437424      jossa\n",
            "1.7862376036259573      katso\n",
            "1.7862376036259573      esimerkiksi\n",
            "1.8253189536684284      kuitenkin\n",
            "1.830768558435993      kanssa\n",
            "1.8362480242006185      ole\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "deletable": true,
        "editable": true,
        "id": "zbLtUWlLnRCp"
      },
      "source": [
        "# Vector-space model\n",
        "\n",
        "* So far we looked at term vectors (rows of the term-document matrix)\n",
        "* We could also think about document vectors (columns of the term-document matrix)\n",
        "* Documents are vectors in a high-dimensional space, terms are the dimensions\n",
        "* Document vectors very high-dimensional but very very sparse\n",
        "* Same for queries - queries can also be seen as vectors in a high-dimensional space\n",
        "* Search:\n",
        "  * similarity between query vector and document vector\n",
        "  * higher similarity means better hit (rank higher)\n",
        "  \n",
        "## Similarity measures\n",
        "\n",
        "* Similarity -> negative distance\n",
        "* Eucledian distance -> affected by vector length\n",
        "* Cosine similarity -> cosine of the angle between query and document vectors\n",
        "  * 1 for total similarity (angle 0)\n",
        "  * -1 for complete opposite\n",
        "  * we only have positive numbers in our vectors, so we are on the [0,1] scale not [-1,1]\n",
        "  * not sensitive to length\n",
        "  * incredibly easy to compute!\n",
        "  * $cos(u,v)=\\frac{u\\cdot v}{||u||\\cdot ||v||}=\\frac{\\sum_i v_i\\cdot u_i}{\\sqrt{\\sum_i u_i^2}\\sqrt{\\sum_i v_i^2}}$\n",
        "  * dot-product of normalized vectors - just multiply the numbers together, really\n",
        "  \n",
        "## Weighting schemes - document, query\n",
        "\n",
        "* Now we can apply different weighting to documents and queries\n",
        "* Typical choice:\n",
        "  * document: log tf, no idf, length-normalized - why no idf?\n",
        "  * query: log tf, log idf, length-normalized\n",
        "  * cosine similarity\n",
        "\n"
      ]
    }
  ]
}