update docs for preview tools branch #6919

Lots of edits were made previously in this branch.

doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv

@@ -1,5 +1,6 @@
+Tool	Type	Scope	Description
 TwoRavens	explore	file	A system of interlocking statistical tools for data exploration, analysis, and meta-analysis: http://2ra.vn. See the :doc:`/user/data-exploration/tworavens` section of the User Guide for more information on TwoRavens from the user perspective and the :doc:`/installation/r-rapache-tworavens` section of the Installation Guide. 
 Data Explorer	explore	file	A GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Explorer for the instructions on adding Data Explorer to your Dataverse; and the :doc:`/installation/prerequisites` section of the Installation Guide for the instructions on how to set up **basic R configuration required** (specifically, Dataverse uses R to generate .prep metadata files that are needed to run Data Explorer). 
 Whole Tale	explore	dataset	A platform for the creation of reproducible research packages that allows users to launch containerized interactive analysis environments based on popular tools such as Jupyter and RStudio. Using this integration, Dataverse users can launch Jupyter and RStudio environments to analyze published datasets. For more information, see the `Whole Tale User Guide <https://wholetale.readthedocs.io/en/stable/users_guide/integration.html>`_.
-File Previewers	preview, explore	file	A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, text, video, tabular data, and spreadsheets - allowing them to be viewed without downloading. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/GlobalDataverseCommunityConsortium/dataverse-previewers
+File Previewers	explore	file	A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, text, video, tabular data, and spreadsheets - allowing them to be viewed without downloading. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/GlobalDataverseCommunityConsortium/dataverse-previewers
 Data Curation Tool	configure	file	A GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Curation-Tool for the installation instructions.

doc/sphinx-guides/source/_static/installation/files/root/external-tools/fabulousFileTool.json

@@ -4,7 +4,6 @@
   "toolName": "fabulous",
   "scope": "file",
   "type": "explore",
-  "hasPreviewMode": "false",
   "toolUrl": "https://fabulousfiletool.com",
   "contentType": "text/tab-separated-values",
   "toolParameters": {

doc/sphinx-guides/source/admin/external-tools.rst

@@ -1,7 +1,7 @@
 External Tools
 ==============
 
-External tools can provide additional features that are not part of Dataverse itself, such as data file previews and exploration.
+External tools can provide additional features that are not part of Dataverse itself, such as data file previews, visualization, and curation.
 
 .. contents:: |toctitle|
   :local:
@@ -12,7 +12,7 @@ Inventory of External Tools
 ---------------------------
 
 .. csv-table:: 
-   :header: "Tool", "Type", "Scope", "Description"
+   :header-rows: 1
    :widths: 20, 10, 5, 65
    :delim: tab
    :file: ../_static/admin/dataverse-external-tools.tsv
@@ -25,20 +25,18 @@ Managing External Tools
 Adding External Tools to Dataverse
 +++++++++++++++++++++++++++++++++++
 
-To add an external tool to your Dataverse installation, configure the JSON manifest file for that tool. Here is an example manifest for a sample explore tool:
+To add an external tool to your installation of Dataverse you must first download a JSON file for that tool, which we refer to as a "manifest". It should look something like this:
 
 .. literalinclude:: ../_static/installation/files/root/external-tools/fabulousFileTool.json
 
-Download the JSON manifest file from :ref:`inventory-of-external-tools` for the tools you wish to install.
+Go to :ref:`inventory-of-external-tools` and download a JSON manifest for one of the tools by following links in the description to installation instructions.
 
 Configure the tool with the curl command below, making sure to replace the ``fabulousFileTool.json`` placeholder for name of the JSON manifest file you downloaded.
 
 .. code-block:: bash
 
   curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools --upload-file fabulousFileTool.json 
 
-Note that some tools will provide a preview mode, which provides an embedded, simplified view of the tool on the file pages of your installation. This is controlled by the ``hasPreviewMode`` parameter. 
-
 Listing All External Tools in Dataverse
 +++++++++++++++++++++++++++++++++++++++
 
@@ -75,15 +73,28 @@ Testing External Tools
 
 Once you have added an external tool to your installation of Dataverse, you will probably want to test it to make sure it is functioning properly.
 
+File Level vs. Dataset Level
+++++++++++++++++++++++++++++
+
+File level tools are specific to the file type (content type or MIME type). For example, a tool may work with PDFs, which have a content type of "application/pdf". 
+
+In contrast, dataset level tools are always available no matter what file types are within the dataset.
+
 File Level Explore Tools
 ++++++++++++++++++++++++
 
-File level explore tools are specific to the file type (content type or MIME type). For example, the Data Explorer tool can be configured to explore tabular data files.
+File level explore tools provide a variety of features from data visualization to statistical analysis.
+
+For each supported file type, file level explore tools appear in the file listing of the dataset page as well as under the "Access" button on each file page.
 
 File Level Preview Tools
 ++++++++++++++++++++++++
 
-File level preview tools allow the user to see a preview of the file contents without having to download it. Explore tools can also use the ``hasPreviewMode`` parameter to display a preview, which is a simplified view of an explore tool designed specifically for embedding in the file page.
+File level preview tools allow the user to see a preview of the file contents without having to download it.
+
+When a file has a preview available, a preview icon will appear next to that file in the file listing on the dataset page. On the file page itself, the preview will appear in a Preview tab either immediately or once a guestbook has been filled in or terms, if any, have been agreed to.
+
+File level explore tools can also show previews if they have been designed to show a simplified view for embedding in the file page and if the ``hasPreviewMode`` parameter is set to ``true``.
 
 File Level Configure Tools
 ++++++++++++++++++++++++++

doc/sphinx-guides/source/api/apps.rst

@@ -68,7 +68,7 @@ https://github.com/CenterForOpenScience/osf.io/tree/develop/addons/dataverse
 Geoconnect
 ~~~~~~~~~~
 
-Geoconnect configures Dataverse files to be visualized and explored on `WorldMap <http://worldmap.harvard.edu>`_. Read more about it in the :doc:`/user/data-exploration/worldmap` section of the User Guide.
+Geoconnect allows Dataverse files to be visualized and explored on `WorldMap <http://worldmap.harvard.edu>`_. Read more about it in the :doc:`/user/data-exploration/worldmap` section of the User Guide.
 
 https://github.com/IQSS/geoconnect
 

doc/sphinx-guides/source/api/external-tools.rst

@@ -9,30 +9,38 @@ External tools can provide additional features that are not part of Dataverse it
 Introduction
 ------------
 
-External tools are additional applications the user can open from Dataverse to preview or explore data files. The term "external" is used to indicate that the user has left the Dataverse web interface. For example, the user looking at a dataset on ``demo.dataverse.org`` can open an explore tool hosted on the domain ``fabulousfiletool.com``.
+External tools are additional applications the user can access or open from Dataverse to preview, explore, and manipulate data files and datasets. The term "external" is used to indicate that the tool is not part of the main Dataverse application.
 
-The "other website" (fabulousfiletool.com in the example above) is probably part of the same ecosystem of scholarly publishing that Dataverse itself participates in. Sometimes the other website runs entirely in the browser. Sometimes the other website is a full blown server side web application like Dataverse itself.
+Once you have created the external tool itself (which is most of the work!), you need to teach Dataverse how to construct URLs that your tool needs to operate. For example, if you've deployed your tool to fabulousfiletool.com your tool might want the ID of a file and the siteUrl of the Dataverse installation like this: https://fabulousfiletool.com?fileId=42&siteUrl=http://demo.dataverse.org
 
-The possibilities for external tools are endless. Let's look at some examples to get your creative juices flowing.
+In short, you will be creating a manifest in JSON format that describes not only how to construct URLs for your tool, but also what types of files your tool operates on, where it should appear in the Dataverse web interfaces, etc. 
+
+The possibilities for external tools are endless. Let's look at some examples to get your creative juices flowing. Then we'll look at a complete list of parameters you can use when creating the manifest file for your tool.
+
+If you're still looking for more information on external tools, you can also watch a video introduction called `Background on the External Tool Framework`_ (slides_) from the 2020 Dataverse Community Meeting.
+
+.. _Background on the External Tool Framework: https://youtu.be/YH4I_kldmGI?t=159
+
+.. _slides: https://osf.io/xjdfw/
 
 Examples of External Tools
 --------------------------
 
 Note: This is the same list that appears in the :doc:`/admin/external-tools` section of the Admin Guide.
 
 .. csv-table:: 
-   :header: "Tool", "Type", "Scope", "Description"
+   :header-rows: 1
    :widths: 20, 10, 5, 65
    :delim: tab
    :file: ../_static/admin/dataverse-external-tools.tsv
 
 How External Tools Are Presented to Users
 -----------------------------------------
 
-An external tool can appear in Dataverse in one of three ways:
+An external tool can appear in Dataverse in a variety of ways:
 
-- as an "Explore" or "Preview" option for a file or dataset
-- as an "Configure" option for a file
+- as an explore, preview, or configure option for a file
+- as an explore option for a dataset
 - as an embedded preview on the file landing page
 
 See also the :ref:`testing-external-tools` section of the Admin Guide for some perspective on how installations of Dataverse will expect to test your tool before announcing it to their users.

doc/sphinx-guides/source/developers/geospatial.rst

@@ -10,7 +10,7 @@ Geoconnect
 
 Geoconnect works as a middle layer, allowing geospatial data files in Dataverse to be visualized with Harvard WorldMap. To set up a Geoconnect development environment, you can follow the steps outlined in the `local_setup.md <https://github.com/IQSS/geoconnect/blob/master/local_setup.md>`_ guide. You will need Python and a few other prerequisites.
 
-As mentioned under the :ref:`architecture` section of Preparation in the Installation Guide, Geoconnect is an optional external tool for Dataverse, so this section is only necessary to follow it you are working on an issue related to this feature.
+As mentioned under the :ref:`architecture` section of Preparation in the Installation Guide, Geoconnect is an optional component of Dataverse, so this section is only necessary to follow it you are working on an issue related to this feature.
 
 How Dataverse Ingests Shapefiles
 --------------------------------

doc/sphinx-guides/source/installation/prep.rst

@@ -52,19 +52,19 @@ Required Components
 When planning your installation you should be aware of the following components of the Dataverse architecture:
 
 - Linux: RHEL/CentOS is highly recommended since all development and QA happens on this distribution.
-- App server: Payara is the recommended Jakarta EE application server
+- App server: Payara is the recommended Jakarta EE application server.
 - PostgreSQL: a relational database.
 - Solr: a search engine. A Dataverse-specific schema is provided.
 - SMTP server: for sending mail for password resets and other notifications.
 - Persistent identifier service: DOI and Handle support are provided. Production use requires a registered DOI or Handle.net authority.
+- Rserve: runs as a daemon to execute R code.
 
 Optional Components
 +++++++++++++++++++
 
 There are a number of optional components you may choose to install or configure, including:
 
 - External Tools: Third party tools for data exploration can be added to Dataverse by following the instructions in the :doc:`/admin/external-tools` section of the Admin Guide.
-- R, rApache, Zelig, and TwoRavens: Statistical tools for data exploration, analysis, and meta-analysis. See the :doc:`r-rapache-tworavens` section of the Installation Guide, as well as :doc:`/user/data-exploration/tworavens` section of the User Guide.
 - Dropbox integration :ref:`dataverse.dropbox.key`: for uploading files from the Dropbox API.
 - Apache: a web server that can "reverse proxy" Jakarta EE applications (like Dataverse) and rewrite HTTP traffic.
 - Shibboleth: an authentication system described in :doc:`shibboleth`. Its use with Dataverse requires Apache.

doc/sphinx-guides/source/user/data-exploration/worldmap.rst

@@ -16,7 +16,7 @@ Note: WorldMap hosts their own `user guide <http://worldmap.harvard.edu/static/d
 What is Geoconnect?
 ===================
 
-Geoconnect is a platform that integrates Dataverse and WorldMap, allowing researchers to visualize their geospatial data. Geoconnect can be used to create maps of shapefiles or of tabular files containing geospatial information. Geoconnect is an optional configure external tool for Dataverse, so if you are interested in this feature but don't see it in the installation of Dataverse you are using, contact support for that installation to request they enable Geoconnect.
+Geoconnect is a platform that integrates Dataverse and WorldMap, allowing researchers to visualize their geospatial data. Geoconnect can be used to create maps of shapefiles or of tabular files containing geospatial information. Geoconnect is an optional component of Dataverse, so if you are interested in this feature but don't see it in the installation of Dataverse you are using, contact support for that installation to request they enable Geoconnect.
 
 Once the map has been created using Geoconnect, can be viewed by choosing the WorldMap explore tool option for that data file.
 

doc/sphinx-guides/source/user/dataset-management.rst

@@ -163,12 +163,12 @@ Certain file types in Dataverse are supported by additional functionality, which
 File Previews
 -------------
 
-Installations of Dataverse can install previewers for common file types uploaded by their research communities. The previews appear on the file page. If a preview tool for a specific file type is available, the preview will be created and will display automatically. File previews are not available for restricted files unless they are being accessed using a Private URL. See also :ref:`privateurl`.
+Installations of Dataverse can install previewers for common file types uploaded by their research communities. The previews appear on the file page. If a preview tool for a specific file type is available, the preview will be created and will display automatically, after terms have been agreed to or a guestbook entry has been made, if necessary. File previews are not available for restricted files unless they are being accessed using a Private URL. See also :ref:`privateurl`.
 
 Tabular Data Files
 ------------------
 
-Files in certain formats - Stata, SPSS, R, Excel(xlsx) and CSV - may be ingested as tabular data (see :doc:`/user/tabulardataingest/index` section of the User Guide for details). Tabular data files can be further explored and manipulated with :doc:`/admin/external-tools` if they have been enabled in the installation of Dataverse you are using.
+Files in certain formats - Stata, SPSS, R, Excel (xlsx), CSV and TSV - may be ingested as tabular data (see :doc:`/user/tabulardataingest/index` section of the User Guide for details). Tabular data files can be further explored and manipulated with :doc:`/admin/external-tools` if they have been enabled in the installation of Dataverse you are using.
 
 Additional download options available for tabular data (found in the same drop-down menu under the "Download" button): 
 
@@ -184,7 +184,7 @@ See also :ref:`restricted-tabular-files`.
 Geospatial
 ----------
 
-Geospatial `shapefiles <http://en.wikipedia.org/wiki/Shapefile>`_ can be further explored and manipulated through an optional installation configuration with `WorldMap <../user/data-exploration/worldmap.html>`_, a geospatial data visualization and analysis tool developed by the `Center for Geographic Analysis <http://gis.harvard.edu/>`_ at Harvard University. A shapefile is a set of files, often uploaded/transferred in .zip format.  This set may contain up to 15 files.  A minimum of 3 specific files (.shp, .shx, .dbf) are needed to be a valid shapefile and a 4th file (.prj) is required for WorldMap--or any type of meaningful visualization.
+Geospatial `shapefiles <http://en.wikipedia.org/wiki/Shapefile>`_ can be further explored and manipulated through an optional integration with `WorldMap <../user/data-exploration/worldmap.html>`_, a geospatial data visualization and analysis tool developed by the `Center for Geographic Analysis <http://gis.harvard.edu/>`_ at Harvard University. A shapefile is a set of files, often uploaded/transferred in .zip format.  This set may contain up to 15 files.  A minimum of 3 specific files (.shp, .shx, .dbf) are needed to be a valid shapefile and a 4th file (.prj) is required for WorldMap--or any type of meaningful visualization.
 
 For ingest into Dataverse and connecting to WorldMap, these 4 files are the minimum required:
 

doc/sphinx-guides/source/user/find-use-data.rst

@@ -104,8 +104,6 @@ If you'd like to download a single file or some subset of the dataset's files, y
 
 You may also download a file from its file page by clicking the Download button in the upper right corner of the page, or by :ref:`url_download` under the Metadata tab on the lower half of the page.
 
-Tabular data files offer additional options such as explore using any data exploration or visualization :doc:`/admin/external-tools`, or choose from a number of tabular-data-specific download options.
-
 Tabular Data
 ^^^^^^^^^^^^
 
@@ -115,7 +113,7 @@ Ingested files can be downloaded in several different ways.
 
 - The original file, which may be in a proprietary format which requires special software
 
-- Rdata format if the instalation has configured this
+- RData format if the installation has configured this
 
 - The variable metadata for the file in DDI format
 
@@ -151,7 +149,7 @@ After you've downloaded the Dataverse Package, you may want to double-check that
 Explore Data
 ------------
 
-Please see the :doc:`/user/data-exploration/index` to get started.
+Some file types and datasets offer data exploration options if external tools have been installed. The tools are described in the :doc:`/admin/external-tools` section of the Admin Guide.
 
 .. |image-file-tree-view| image:: ./img/file-tree-view.png
    :class: img-responsive