notebooks/token_healing.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Token healing\n",
    "\n",
    "Guidance uses what we call \"token healing\" to fix tokenization artifacts that normally arise at the boundary between the end of a prompt and the beginning of a set of generated tokens. Note that token healing requires direct endpoint integration to run effciently, so it currently supported only for the `guidance.llms.Transformers` LLM backend.\n",
    "\n",
    "## Why token healing is needed\n",
    "Language models process tokens, which are chunks of text that often are similar to a word. This impacts how language models see text, and also how we can prompt them, since every prompt has to be a set of tokens. Encodings like [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) that are used by GPT-style models map all input bytes to token ids in a greedy manner. This works well during training, but can lead to some subtle issues during prompting and inference because the greedy token boundaries often don't line up with the end of the prompt if we consider the generated tokens that will also come next. Of course the end of a prompt will always align with a token boundary because the prompt is tokenized before being extended by the model, but if the first characters of the completion are part of a longer token that would span the prompt boundary, that longer token cannot be used (even though that is what the model would expect based on training data). \n",
    "\n",
    "To see why token healing is important consider the prompt \"This is a \", which is then completed with \"fine day.\" by the model, so resulting in the final string \"This is a fine day.\". If we tokenize the prompt \"This is a \" with GPT2 BPE we get `[1212, 318, 257, 220]`, and the tokenization of the extention \"fine day.\" is `[38125, 1110, 13]`. This results in a final prompt + generation token sequence of `[1212, 318, 257, 220, 38125, 1110, 13]`. If however we were to tokenize the whole string \"This is a fine day.\" jointly we instead get `[1212, 318, 257, 3734, 1110, 13]`. Which tokenization is correct? Well, the correct tokenization is the one that best communicates intent to the model. Since the model learned intent based on a greedy tokenization of the training text, that means the joint tokenization that also uses pure greedy matching will better align with how the model processed the training data, and so it is also likely to better communicate intent to the model. This is the reason why ending your prompt with a space is almost always a bad idea in GPT models since most word-based tokens have the space before the word, not after it.\n",
    "\n",
    "Note that another way to see that the \"standard\" prompt-boundary-based encoding is worse than the joint one we get with token healing is to observe that 38125 (the token id for \"fine\") is a large number, this means it is uncommon to see that token in the training data (since BPE encodings are built up greedily based on frequency). In contrast 3734 (the token id for \" fine\") is a much more common token and so more likely to clearly communicate intent to the model (since the model has seen it many times and hence had more opportunity to learn its meaning in many contexts).\n",
    "\n",
    "## How token healing works\n",
    "Guidance avoids the above tokenization artifacts automatically using a method we call \"token healing\" that backs up the generation process by one token before the end of the prompt, then constrains the first token generated to have a prefix that matches the last token in the prompt. This allows the generated text string to have the token encoding that the model would expect based on its training data, not an unusual alternative encoding forcing by the prompt boundary. Token healing allows you to express your prompts however you wish, without worrying about boundaries (which effect many tokens, not just space characters)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenization of `This is a `: [1212, 318, 257, 220]\n",
      "Tokenization of `fine day.`: [38125, 1110, 13]\n",
      "Tokenization of `This is a fine day.`: [1212, 318, 257, 3734, 1110, 13]\n"
     ]
    }
   ],
   "source": [
    "# compute the tokenizations of the example above\n",
    "import transformers\n",
    "tokenizer = transformers.AutoTokenizer.from_pretrained('gpt2')\n",
    "print(\"Tokenization of `This is a `:\", tokenizer.encode(\"This is a \"))\n",
    "print(\"Tokenization of `fine day.`:\", tokenizer.encode(\"fine day.\"))\n",
    "print(\"Tokenization of `This is a fine day.`:\", tokenizer.encode(\"This is a fine day.\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Token healing in action\n",
    "\n",
    "Below is a prompt that we run both with and without token healing to see how it can impact generation quality.\n",
    "\n",
    "### With token healing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-03ee8fa2-f453-4291-9ad8-e28ce784f74e\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-03ee8fa2-f453-4291-9ad8-e28ce784f74e\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>Tweak this proverb to apply to model instructions instead.\n",
       "\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='&amp;#123;&amp;#123;proverb&amp;#125;&amp;#125;'>Where there is no guidance, a people falls,\n",
       "but in an abundance of counselors there is safety.</span>\n",
       "- <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='&amp;#123;&amp;#123;book&amp;#125;&amp;#125;'>Proverbs</span> <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='&amp;#123;&amp;#123;chapter&amp;#125;&amp;#125;'>11</span>:<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='&amp;#123;&amp;#123;verse&amp;#125;&amp;#125;'>14</span>\n",
       "\n",
       "UPDATED\n",
       "Where there is no guidance<span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='&amp;#123;&amp;#123;gen &#x27;rewrite&#x27; stop=&quot;- &quot;&amp;#125;&amp;#125;'>, a people falls,but in an abundance of counselors there is safety.\n",
       "\n",
       "</span>\n",
       "- GPT Proverbs <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='&amp;#123;&amp;#123;gen &#x27;chapter&#x27; max_tokens=10&amp;#125;&amp;#125;'>11</span>:<span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='&amp;#123;&amp;#123;gen &#x27;verse&#x27; stop=&#x27;\\n&#x27;&amp;#125;&amp;#125;'>14</span></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"03ee8fa2-f453-4291-9ad8-e28ce784f74e\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import guidance\n",
    "llm = guidance.llms.Transformers(\"gpt2\")\n",
    "guidance.llms.Transformers.cache.clear()\n",
    "program = guidance(\"\"\"Tweak this proverb to apply to model instructions instead.\n",
    "\n",
    "{{proverb}}\n",
    "- {{book}} {{chapter}}:{{verse}}\n",
    "\n",
    "UPDATED\n",
    "Where there is no guidance{{gen 'rewrite' stop=\"- \"}}\n",
    "- GPT Proverbs {{gen 'chapter' max_tokens=10}}:{{gen 'verse' stop='\\\\n'}}\"\"\", llm=llm)\n",
    "\n",
    "# execute the program on a specific proverb\n",
    "program(\n",
    "    proverb=\"Where there is no guidance, a people falls,\\nbut in an abundance of counselors there is safety.\",\n",
    "    book=\"Proverbs\",\n",
    "    chapter=11,\n",
    "    verse=14\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Without token healing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-57c872b8-53a0-4b7b-8be9-64b45fd4bc0e\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-57c872b8-53a0-4b7b-8be9-64b45fd4bc0e\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>Tweak this proverb to apply to model instructions instead.\n",
       "\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='&amp;#123;&amp;#123;proverb&amp;#125;&amp;#125;'>Where there is no guidance, a people falls,\n",
       "but in an abundance of counselors there is safety.</span>\n",
       "- <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='&amp;#123;&amp;#123;book&amp;#125;&amp;#125;'>Proverbs</span> <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='&amp;#123;&amp;#123;chapter&amp;#125;&amp;#125;'>11</span>:<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='&amp;#123;&amp;#123;verse&amp;#125;&amp;#125;'>14</span>\n",
       "\n",
       "UPDATED\n",
       "Where there is no guidance<span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='&amp;#123;&amp;#123;gen &#x27;rewrite&#x27; stop=&quot;- &quot;&amp;#125;&amp;#125;'>, a people falls,but in an abundance of counselors there is safety.\n",
       "\n",
       "</span>\n",
       "- GPT Proverbs <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='&amp;#123;&amp;#123;gen &#x27;chapter&#x27; max_tokens=10&amp;#125;&amp;#125;'>__________________\n",
       "\n",
       "&quot;The Lord is the only</span>:<span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='&amp;#123;&amp;#123;gen &#x27;verse&#x27; stop=&#x27;\\n&#x27;&amp;#125;&amp;#125;'> the only is the one who is in the midst of the things.&quot;</span></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"57c872b8-53a0-4b7b-8be9-64b45fd4bc0e\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import guidance\n",
    "llm = guidance.llms.Transformers(\"gpt2\", token_healing=False)\n",
    "guidance.llms.Transformers.cache.clear()\n",
    "program = guidance(\"\"\"Tweak this proverb to apply to model instructions instead.\n",
    "\n",
    "{{proverb}}\n",
    "- {{book}} {{chapter}}:{{verse}}\n",
    "\n",
    "UPDATED\n",
    "Where there is no guidance{{gen 'rewrite' stop=\"- \"}}\n",
    "- GPT Proverbs {{gen 'chapter' max_tokens=10}}:{{gen 'verse' stop='\\\\n'}}\"\"\", llm=llm)\n",
    "\n",
    "# execute the program on a specific proverb\n",
    "program(\n",
    "    proverb=\"Where there is no guidance, a people falls,\\nbut in an abundance of counselors there is safety.\",\n",
    "    book=\"Proverbs\",\n",
    "    chapter=11,\n",
    "    verse=14\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While it may seems strange to us why GPT2 does not put a number after the space character (instead writing a bunch of underscores), it makes sense if we think about the tokens involved. If a number was likely to come after the space, then it would have been included in the token. By sending the token id 220 (a space) by itself to the model we are communicating that what comes next is not something that is in a token that starts with a space (since otherwise the greedy tokenization would have used it). So GPT2 picks a non-word token (which are not normally prefixed with a space). In this case it chooses 2602, which is a bunch of underscores \"________\"."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height: 1px; opacity: 0.5; border: none; background: #cccccc;\">\n",
    "<div style=\"text-align: center; opacity: 0.5\">Have an idea for more helpful examples? Pull requests that add to this documentation notebook are encouraged!</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "adatest",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}