docs: add privacy policy/guide/faq

- #2

README.org

@@ -147,10 +147,11 @@ Also check out my [[https://beepb00p.xyz/myinfra.html#promnesia][infrastructure
 :                                  └──────────────────────────────┘
 :                                                 ⇣⇣
 :                                  ┌─────────────────────────────────┐
-:                                  │ 🌐      web browser             ├───────────────────────────┐
-:                                  |     (promnesia extension)       ⇐  💾 local browser history |
-:                                  └─────────────────────────────────┴───────────────────────────┘
+:          ┌───────────────────────┤  🌐      web browser            ├────────────────────┐
+:          │  💾 browser bookmarks ⇒      (promnesia extension)      ⇐  💾 browser history |
+:          └───────────────────────┴─────────────────────────────────┴────────────────────┘
 
+# https://en.wikipedia.org/wiki/List_of_Unicode_characters#Box_Drawing
 # TODO would be really nice to have links here.. but not sure how without svg...
 
 * Install
@@ -161,7 +162,10 @@ Also check out my [[https://beepb00p.xyz/myinfra.html#promnesia][infrastructure
 - extension:
 
   - [[https://chrome.google.com/webstore/detail/promnesia/kdmegllpofldcpaclldkopnnjjljoiio][Chrome]]: desktop version. Unfortunately mobile Chrome doesn't support web extensions.
-  - [[https://addons.mozilla.org/en-US/firefox/addon/promnesia][Firefox]]: desktop and Android. Unfortunatly, iOS Firefox [[https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_for_iOS#Addons][doesn't support web extensions]].
+  - [[https://addons.mozilla.org/en-US/firefox/addon/promnesia][Firefox]]: desktop and Android
+    - note: web extensions on Android are mostly broken at the moment, see [[https://discourse.mozilla.org/t/add-on-support-in-new-firefox-for-android/53488][here]]
+      (unless you're using [[https://blog.mozilla.org/addons/2020/09/29/expanded-extension-support-in-firefox-for-android-nightly][Firefox Nightly]])
+    - unfortunately, iOS Firefox [[https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_for_iOS#Addons][doesn't support web extensions]].
   - you can also find 'unpacked' versions in [[https://github.com/karlicoss/promnesia/releases][Releases]]
 
     It can be useful because Chrome Web Store releases might take days to approve, but in general the store version if preferrable.
@@ -371,6 +375,7 @@ Now let's consider some *concrete* examples of different kinds of Visits:
   [[https://en.wiktionary.org/wiki/promnesia][Promnesia]] is coming from Ancient Greek and means "déjà vu".
   (ironically, promnesia project is kind of doing the opposite -- it replaces a vague feeling of seeing a page before with a reliable digital tool)
 
+#   TODO this belongs to features?
 - Which hotkeys/shortcuts does Promnesia register in my browser?
 
   Promnesia registers two shortcuts for: activating the extension and marking the already visited links on the current page.
@@ -391,22 +396,5 @@ I don't need money, but I understand it's often easier to give away than time, s
 * More links
 - [[file:doc/SOURCES.org][Documentation on the sources]]
 - [[file:doc/DEVELOPMENT.org][Developer's guide]]
-
-* Extension permissions
-- =activeTab=: getting current tab info and adding the sidebar
-- =webNavigation=: watching page state changes (to trigger the extension on page load)
-- =storage=: for settings
-- =contextMenus=: context menu
-- =notifications=: showing notifications
-
-There permissions are required at the moment, but there is an [[https://github.com/karlicoss/promnesia/issues/97][issue]] for work on possibly making them optional.
-
-- =tabs=: making the extension work without an explicit user action (the extension is meant to be a passive assistant)
-
-  The extension is still useful even with explicit action only, so worth making opt-in.
-- =history=: to use local browsing history
-
-  Local history isn't strictly required, so we could omit this if people prefer.
-- =file/http/https=: the extension is meant to work on any page, hence such a broad scope.
-
-  Migth be optional in the future, and requested on demand if people feel it's worth it
+- [[file:doc/TROUBLESHOOTING.org][Troubleshooting guide]]
+- [[file:doc/PRIVACY.org][Privacy policy]]

doc/DEVELOPMENT.org

@@ -1,7 +1,4 @@
-I'm in the process of putting up better developer's documentation, but you should be able to figure it out from the CI pipelines:
-
-- [[file:../.github/workflows/main.yml][Github Actions]]
-- [[file:../.circleci/config.yml][CircleCI]].
+I'm in the process of putting up better developer's documentation, but you should be able to figure it out from the [[file:../.github/workflows/main.yml][Github Actions pipeline]].
 
 * Run backend
    During development, the easiest way to run =promnesia= as =scripts/promnesia=. Alternatively, you can use =pip install -e .=.
@@ -17,16 +14,19 @@ I'm in the process of putting up better developer's documentation, but you shoul
    :         --publish # [optional], release to the Chrome Web Store/Mozilla addons
 
    You'll find the result in =dist/=. After that, you can load it in your browser and develop.
-   For firefox, note that mozilla intentionally complicates this in several ways. On android, only firefox nightly will let you install arbitrary extensions. On desktop, you have a choice between running nightly, re-installing your unpacked extension after every firefox restart, or setting up signing through web-ext. For a quick start into the extension development, on desktop, chrome is therefore recommended (and is the author's primary browser, i believe?). On mobile, firefox nightly is your only option (?).
+
+   - on Firefox, the temporary loaded extensions only persist until the browser restart. Chrome doesn't have that issue.
+   - on Firefox for Android, web extensions on Android are mostly broken at the moment, see [[https://discourse.mozilla.org/t/add-on-support-in-new-firefox-for-android/53488][here]] (unless you're using [[https://blog.mozilla.org/addons/2020/09/29/expanded-extension-support-in-firefox-for-android-nightly][Firefox Nightly]])
 
 * Run end-to-end tests
 
-   You need:
+End-to-end test go throught the full cycle from indexing & backend to automated extension tests, and are very helpful to make sure you don't break things.
+
+You need:
 
-   - firefox dev edition (standard wouldn't let you install unsigned extensions) and =geckodriver=
-   - TODO chromedriver?
-   - =pip3 install pyautogui= for keyboard automation
+- firefox + =geckodriver=, or chrome + =chromedriver=
+- =pip3 install pyautogui= for keyboard automation
 
-   Example of running a test:
+Example of running a test:
 
-   : PYTHONPATH=src python3 -m pytest -s 'tests/end2end_test.py::test_add_to_blacklist[chrome]'
+: PYTHONPATH=src python3 -m pytest -s 'tests/end2end_test.py::test_add_to_blacklist[chrome]'

doc/GUIDE.org

@@ -0,0 +1,23 @@
+# TODO fill with guide on features etc?
+
+# TODO mention hook?
+
+* FAQ
+** TODO sources
+** how is my data used?
+See [[file:PRIVACY.org][privacy/security policy]]
+** is Windows supported?
+Sort of, some people have successfully run Promnesia under Windows, but I don't have a Windows computer around so it's hard for me to ensure it works.
+See [[https://github.com/karlicoss/promnesia/issues/91][issues/91]].
+
+not displaying https://github.com/karlicoss/promnesia/issues/171
+trigger the hotkey
+
+# TODO disable debug logs..
+# TODO mark visited (like in ublock), suggest element zapper (maybe just point at FEATURES?)
+** I expect to see a specific visit, but it doesn't appear
+You can check the
+
+indexing is periodic at the moment and only runs regularly. Real-time indexing is harder to implement, there is an outstanding issue [[https://github.com/karlicoss/promnesia/issues/20][here]]
+
+** TODO customize style

doc/PRIVACY.org

@@ -0,0 +1,58 @@
+**TLDR**: Promnesia isn't collecting any data and isn't transmitting your data to any external site. Not telemetry is collected or sent either.
+
+* How Promnesia works
+See [[file:../README.org::#how-does-it-work]["How does it work"]] for the longer explaination, but in summary:
+
+- promnesia indexer runs against the files on your computer, and stores the results in the intermediate database (also on your computer)
+- promnesia server runs on your computer and exposes a local port (=13131= by default), so it can communicate with the browser extension
+- promnesia extension runs in your browser and requests various data from the promnesia server
+
+**By default this all happens only within your system**. The extension works in local-first/offline mode, which is one of the main goals.
+(However, if you want, you can set it up on an external domain, with the reverse proxy or something like that).
+
+* What data is requested from the promnesia server
+
+There are just a few requests to the Promnesia server the extesion is doing at the moment.
+
+- ~search/search_around~: sends whatever you typed in the search box
+- ~visits~: sends the current tab URL to the server (unless it's excluded)
+- ~visited~: sends all URLs on the current page to the server (except the excluded ones)
+
+None of the information above is kept by the server, it **only reads the data from the database**.
+It is however possible that some of it is retained in the console logs, that would depend on how exactly you're running the server and the log retention policy of your system.
+
+# FIXME mention excludelist
+# TODO if you want to backup your browser history and feed in promnesia (e.g. to overcome the 90 days limit etc)
+
+* Other external requests
+
+# TODO link to the excludelist feature
+- if you have external excludelists URLs in the extension settings, they will be updated now and then.
+
+
+* Extension permissions
+- =activeTab=: getting current tab info and adding the sidebar
+- =webNavigation=: watching page state changes (to trigger the extension on page load)
+- =storage=: for settings
+- =contextMenus=: context menu
+- =notifications=: showing notifications
+
+There permissions are required at the moment, but there is an [[https://github.com/karlicoss/promnesia/issues/97][issue]] for work on possibly making them optional.
+
+- =tabs=: making the extension work without an explicit user action (the extension is meant to be a passive assistant)
+
+  The extension is still useful even with explicit action only, so worth making opt-in.
+- =history=: to use local browsing history
+
+  Local history isn't strictly required, so we could omit this if people prefer.
+- =bookmarks=: used as one of the sources
+
+  It can already be toggled in the settings, so the permission could be dynamic too
+- =file/http/https=: the extension is meant to work on any page, hence such a broad scope.
+
+  Might be optional in the future, and requested on demand if people feel it's worth it
+
+* Security
+While I have some reasonable understanding of security, I'm no expert, so would be very grateful if you flag potential issues or [[https://github.com/karlicoss/promnesia/issues/14][go through the code]] (especially extension).
+
+There is some ongoing work on Dockerizing: [[promnesia][issues/55]].

doc/SOURCES.org

@@ -7,12 +7,13 @@ Some sources require additional dependencies, which you can install if needed.
 print('\n') # fix github org-mode issue with drawers
 
 import setup
-for (name, description), _ in setup.DEPS_SOURCES.items():
+for (name, description), vals in setup.DEPS_SOURCES.items():
     # fuck org-ruby. promnesia[name] should be in quotes, but then it doesn't render as code. ugh.
     # https://github.com/wallyqs/org-ruby/issues/45
     print(f"- ~pip3 install --user promnesia[{name}]~")
     print(f'   ')
-    print(f'   {description}')
+    vals = [v.split('>')[0] for v in vals]
+    print(f'   {description}: {", ".join(vals)}')
 #+end_src
 
 #+RESULTS:
@@ -21,22 +22,22 @@ for (name, description), _ in setup.DEPS_SOURCES.items():
 
 - ~pip3 install --user promnesia[optional]~
 
-   dependencies that bring some bells & whistles
+   dependencies that bring some bells & whistles: logzero, python-magic
 - ~pip3 install --user promnesia[HPI]~
 
-   dependencies for [[https://github.com/karlicoss/HPI][HPI]]
+   dependencies for [[https://github.com/karlicoss/HPI][HPI]]: HPI
 - ~pip3 install --user promnesia[html]~
 
-   dependencies for sources.html
+   dependencies for sources.html: beautifulsoup4, lxml
 - ~pip3 install --user promnesia[markdown]~
 
-   dependencies for sources.markdown
+   dependencies for sources.markdown: mistletoe
 - ~pip3 install --user promnesia[org]~
 
-   dependencies for sources.org
+   dependencies for sources.org: orgparse
 - ~pip3 install --user promnesia[telegram]~
 
-   dependencies for sources.telegram
+   dependencies for sources.telegram: dataset
 :end:
 
 Alternatively, you can just install all of them in bulk: ~pip3 install --user promnesia[all]~.

doc/TROUBLESHOOTING.org

@@ -1,8 +1,7 @@
-Here I'm going to keep some sort of FAQ updated as people encounter common issues.
-
-In the meantime:
+General helpful notes:
 
 - your browsing history may contain *sensitive data*. If you're posting or sending me error logs *make sure you strip away all private information* (ideally only attach the relevant bits of the log, containing errors)
+
 - make sure you [[file:../README.org::#install][installed promnesia]] properly, and [[file:SOURCES.org][extra dependencies]], if necessary
 - make sure [[file:../README.org::#try-it-out][demo]] works as expected
 
@@ -14,3 +13,35 @@ In the meantime:
 - run =promnesia index= in your terminal, and pay attention to the error messages and warnings
 - run =promnesia serve= in your terminal (on a different port if necessary). Similarly, pay attention to the errors.
 - in your browser: try [[https://developer.chrome.com/extensions/tut_debugging][opening the debugger]] and checking for the errors in the extension logs
+
+- check [[https://github.com/karlicoss/promnesia/issues][issues]] in case someone else had the same issue too
+- if you're posting an issue with the extension UI, post the page you're having problem with and ideally attach a screenshot (if you think it might help)
+
+* Known bugs/issues
+** TODO issue with url normalisation
+Might happen if the site uses weird way of addressing content, e.g. in the query parameters or fragment (TODO example)
+if you want to fix it for yourself, check out cannon.py (and tests)
+*** TODO hmm need to display normalised url somewhere in the interface?
+
+** extension is too slow
+I've made efforts to make sure everything is responsive and asynchronous, but it is possible in theory that extension slows the page down. Usually that happens if there are too many visits corresponding to the page, e.g. main pages like ~twitter.com~ or ~reddit.com~.
+The easiest fix is to add the exact page (e.g. ~https://twitter.com/home~) to the excludelist. (FIXME link)
+** highlights: the highlight matching algorithm is very simple, so it's possible that it highlights too much or can't detect a hihghight
+See [[https://github.com/karlicoss/promnesia/issues/30][issues/30]]
+** sidebar: might be glitchy when the search tab is open: [[https://github.com/karlicoss/promnesia/issues/109][issues/109]]
+** sidebar: looks weird/truncated on some sites, see [[https://github.com/karlicoss/promnesia/issues/102][issues/102]]
+** mark visited: general glitches
+
+Usually, the more popular the website is, the more dynamic bullshit is on it (e.g. youtube/twitter), which might make things more glitchy and slow.
+
+If the page is too dynamic, it might load more content after the page body (DOM) is loaded (e.g. infinite scroll features).
+It's trickier to detect such changes (see [[https://github.com/karlicoss/promnesia/issues/171][issues/171]]).
+
+In the meantime the easiest is just to toggle 'mark/unmark visited' twice to redraw the marks.
+
+That said, I've spend effort into making sure the marks aren't moving the page elements: the marks can display an outline/border, but it should never disturb and shift the existing page elements, since it's very annoying.
+
+** markdown: no line numbers in the link locators
+FIXME uninstall mistletoe, it will be indexed as
+** TODO indexed too much
+filterlist/hook

doc/config.py

@@ -125,3 +125,6 @@
     # you can use regexes too!
     'redditmedia.com.*.(jpg|png|gif)',
 ]
+
+
+# TODO add HOOK?