(core) add initial support for special shares

Summary:
This gives a mechanism for controlling access control within a document that is distinct from (though implemented with the same machinery as) granular access rules.

It was hard to find a good way to insert this that didn't dissolve in a soup of complications, so here's what I went with:
 * When reading rules, if there are shares, extra rules are added.
 * If there are shares, all rules are made conditional on a "ShareRef" user property.
 * "ShareRef" is null when a doc is accessed in normal way, and the row id of a share when accessed via a share.

There's no UI for controlling shares (George is working on it for forms), but you can do it by editing a `_grist_Shares` table in a document. Suppose you make a fresh document with a single page/table/widget, then to create an empty share you can do:

```
gristDocPageModel.gristDoc.get().docData.sendAction(['AddRecord', '_grist_Shares', null, {linkId: 'xyz', options: '{"publish": true}'}])
```

If you look at the home db now there should be something in the `shares` table:

```
$ sqlite3 -table landing.db "select * from shares"
+----+------------------------+------------------------+--------------+---------+
| id |          key           |         doc_id         |   link_id    | options |
+----+------------------------+------------------------+--------------+---------+
| 1  | gSL4g38PsyautLHnjmXh2K | 4qYuace1xP2CTcPunFdtan | xyz | ...      |
+----+------------------------+------------------------+--------------+---------+
```

If you take the key from that (gSL4g38PsyautLHnjmXh2K in this case) and replace the document's urlId in its URL with `s.<key>` (in this case `s.gSL4g38PsyautLHnjmXh2K` then you can use the regular document landing page (it will be quite blank initially) or API endpoint via the share.

E.g. for me `http://localhost:8080/o/docs/s0gSL4g38PsyautLHnjmXh2K/share-inter-3` accesses the doc.

To actually share some material - useful commands:

```
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Views_section').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Views_section', 1, {shareOptions: '{"publish": true, "form": true}'}])
gristDocPageModel.gristDoc.get().docData.getMetaTable('_grist_Pages').getRecords()
gristDocPageModel.gristDoc.get().docData.sendAction(['UpdateRecord', '_grist_Pages', 1, {shareRef: 1}])
```

For a share to be effective, at least one page needs to have its shareRef set to the rowId of the share, and at least one widget on one of those pages needs to have its shareOptions set to {"publish": "true", "form": "true"} (meaning turn on sharing, and include form sharing), and the share itself needs {"publish": true} on its options.

I think special shares are kind of incompatible with public sharing, since by their nature (allowing access to all endpoints) they easily expose the docId, and changing that would be hard.

Test Plan: tests added

Reviewers: dsagal, georgegevoian

Reviewed By: dsagal, georgegevoian

Subscribers: jarek, dsagal

Differential Revision: https://phab.getgrist.com/D4144

app/client/components/Comm.ts

@@ -27,7 +27,7 @@ import * as dispose from 'app/client/lib/dispose';
 import * as log from 'app/client/lib/log';
 import {CommRequest, CommResponse, CommResponseBase, CommResponseError, ValidEvent} from 'app/common/CommTypes';
 import {UserAction} from 'app/common/DocActions';
-import {DocListAPI, OpenLocalDocResult} from 'app/common/DocListAPI';
+import {DocListAPI, OpenDocOptions, OpenLocalDocResult} from 'app/common/DocListAPI';
 import {GristServerAPI} from 'app/common/GristServerAPI';
 import {getInitialDocAssignment} from 'app/common/urlUtils';
 import {Events as BackboneEvents} from 'backbone';
@@ -149,9 +149,8 @@ export class Comm extends dispose.Disposable implements GristServerAPI, DocListA
    * committed to a document that is called in hosted Grist - all other methods
    * are called via DocComm.
    */
-  public async openDoc(docName: string, mode?: string,
-                       linkParameters?: Record<string, string>): Promise<OpenLocalDocResult> {
-    return this._makeRequest(null, docName, 'openDoc', docName, mode, linkParameters);
+  public async openDoc(docName: string, options?: OpenDocOptions): Promise<OpenLocalDocResult> {
+    return this._makeRequest(null, docName, 'openDoc', docName, options);
   }
 
   /**

app/client/models/DocPageModel.ts

@@ -17,7 +17,7 @@ import {menu, menuDivider, menuIcon, menuItem, menuText} from 'app/client/ui2018
 import {confirmModal} from 'app/client/ui2018/modals';
 import {AsyncFlow, CancelledError, FlowRunner} from 'app/common/AsyncFlow';
 import {delay} from 'app/common/delay';
-import {OpenDocMode, UserOverride} from 'app/common/DocListAPI';
+import {OpenDocMode, OpenDocOptions, UserOverride} from 'app/common/DocListAPI';
 import {FilteredDocUsageSummary} from 'app/common/DocUsage';
 import {Product} from 'app/common/Features';
 import {buildUrlId, IGristUrlState, parseUrlId, UrlIdParts} from 'app/common/gristUrls';
@@ -182,8 +182,13 @@ export class DocPageModelImpl extends Disposable implements DocPageModel {
         if (!urlId) {
           this._openerHolder.clear();
         } else {
-          FlowRunner.create(this._openerHolder,
-            (flow: AsyncFlow) => this._openDoc(flow, urlId, urlOpenMode, state.params?.compare, linkParameters)
+          FlowRunner.create(
+            this._openerHolder,
+            (flow: AsyncFlow) => this._openDoc(flow, urlId, {
+              openMode: urlOpenMode,
+              linkParameters,
+              originalUrlId: state.doc,
+            }, state.params?.compare)
           )
           .resultPromise.catch(err => this._onOpenError(err));
         }
@@ -325,9 +330,9 @@ It also disables formulas. [{{error}}]", {error: err.message})
     this.offerRecovery(err);
   }
 
-  private async _openDoc(flow: AsyncFlow, urlId: string, urlOpenMode: OpenDocMode | undefined,
-                         comparisonUrlId: string | undefined,
-                         linkParameters: Record<string, string> | undefined): Promise<void> {
+  private async _openDoc(flow: AsyncFlow, urlId: string, options: OpenDocOptions,
+                         comparisonUrlId: string | undefined): Promise<void> {
+    const {openMode: urlOpenMode, linkParameters} = options;
     console.log(`DocPageModel _openDoc starting for ${urlId} (mode ${urlOpenMode})` +
                 (comparisonUrlId ? ` (compare ${comparisonUrlId})` : ''));
     const gristDocModulePromise = loadGristDoc();
@@ -383,7 +388,11 @@ It also disables formulas. [{{error}}]", {error: err.message})
     comm.useDocConnection(doc.id);
     flow.onDispose(() => comm.releaseDocConnection(doc.id));
 
-    const openDocResponse = await comm.openDoc(doc.id, doc.openMode, linkParameters);
+    const openDocResponse = await comm.openDoc(doc.id, {
+      openMode: doc.openMode,
+      linkParameters,
+      originalUrlId: options.originalUrlId,
+    });
     if (openDocResponse.recoveryMode || openDocResponse.userOverride) {
       doc.isRecoveryMode = Boolean(openDocResponse.recoveryMode);
       doc.userOverride = openDocResponse.userOverride || null;
@@ -493,7 +502,6 @@ function buildDocInfo(doc: Document, mode: OpenDocMode | undefined): DocInfo {
   const isPreFork = openMode === 'fork';
   const isTemplate = doc.type === 'template' && (isFork || isPreFork);
   const isEditable = !isSnapshot && (canEdit(doc.access) || isPreFork);
-
   return {
     ...doc,
     isFork,

app/common/ACLRuleCollection.ts

@@ -1,8 +1,10 @@
 import {parsePermissions, permissionSetToText, splitSchemaEditPermissionSet} from 'app/common/ACLPermissions';
+import {ACLShareRules, TableWithOverlay} from 'app/common/ACLShareRules';
 import {AclRuleProblem} from 'app/common/ActiveDocAPI';
 import {DocData} from 'app/common/DocData';
 import {AclMatchFunc, ParsedAclFormula, RulePart, RuleSet, UserAttributeRule} from 'app/common/GranularAccessClause';
 import {getSetMapValue, isNonNullish} from 'app/common/gutil';
+import {ShareOptions} from 'app/common/ShareOptions';
 import {MetaRowRecord} from 'app/common/TableData';
 import {decodeObject} from 'app/plugin/objtypes';
 import sortBy = require('lodash/sortBy');
@@ -347,7 +349,9 @@ export class ACLRuleCollection {
     const names: string[] = [];
     for (const rule of this.getUserAttributeRules().values()) {
       const tableRef = tablesTable.findRow('tableId', rule.tableId);
-      const colRef = columnsTable.findMatchingRowId({parentId: tableRef, colId: rule.lookupColId});
+      const colRef = columnsTable.findMatchingRowId({
+        parentId: tableRef, colId: rule.lookupColId,
+      });
       if (!colRef) {
         invalidUAColumns.push(`${rule.tableId}.${rule.lookupColId}`);
         names.push(rule.name);
@@ -379,11 +383,13 @@ export class ACLRuleCollection {
 export interface ReadAclOptions {
   log: ILogger;     // For logging warnings during rule processing.
   compile?: (parsed: ParsedAclFormula) => AclMatchFunc;
-  // If true, call addHelperCols to add helper columns of restricted columns to rule sets.
-  // Used in the server for extra filtering, but not in the client, because:
-  // 1. They would show in the UI
-  // 2. They would be saved back after editing, causing them to accumulate
-  includeHelperCols?: boolean;
+  // If true, add and modify access rules in some special ways.
+  // Specifically, call addHelperCols to add helper columns of restricted columns to rule sets,
+  // and use ACLShareRules to implement any special shares as access rules.
+  // Used in the server, but not in the client, because of at least the following:
+  // 1. Rules would show in the UI
+  // 2. Rules would be saved back after editing, causing them to accumulate
+  enrichRulesForImplementation?: boolean;
 
   // If true, rules with 'schemaEdit' permission are moved out of the '*:*' resource into a
   // fictitious '*SPECIAL:SchemaEdit' resource. This is used only on the client, to present
@@ -455,13 +461,32 @@ function getHelperCols(docData: DocData, tableId: string, colIds: string[], log:
  * Parse all ACL rules in the document from DocData into a list of RuleSets and of
  * UserAttributeRules. This is used by both client-side code and server-side.
  */
-function readAclRules(docData: DocData, {log, compile, includeHelperCols}: ReadAclOptions): ReadAclResults {
-  const resourcesTable = docData.getMetaTable('_grist_ACLResources');
-  const rulesTable = docData.getMetaTable('_grist_ACLRules');
+function readAclRules(docData: DocData, {log, compile, enrichRulesForImplementation}: ReadAclOptions): ReadAclResults {
+  // Wrap resources and rules tables so we can have "virtual" rules
+  // to implement special shares.
+  const resourcesTable = new TableWithOverlay(docData.getMetaTable('_grist_ACLResources'));
+  const rulesTable = new TableWithOverlay(docData.getMetaTable('_grist_ACLRules'));
+  const sharesTable = docData.getMetaTable('_grist_Shares');
 
   const ruleSets: RuleSet[] = [];
   const userAttributes: UserAttributeRule[] = [];
 
+  let hasShares: boolean = false;
+  const shares = sharesTable.getRecords();
+  // ACLShareRules is used to edit resourcesTable and rulesTable in place.
+  const shareRules = new ACLShareRules(docData, resourcesTable, rulesTable);
+  // Add virtual rules to implement shares, if there are any.
+  // Add the virtual rules only when implementing/interpreting them, as
+  // opposed to accessing them for presentation or manipulation in the UI.
+  if (enrichRulesForImplementation && shares.length > 0) {
+    for (const share of shares) {
+      const options: ShareOptions = JSON.parse(share.options || '{}');
+      shareRules.addRulesForShare(share.id, options);
+    }
+    shareRules.addDefaultRulesForShares();
+    hasShares = true;
+  }
+
   // Group rules by resource first, ordering by rulePos. Each group will become a RuleSet.
   const rulesByResource = new Map<number, Array<MetaRowRecord<'_grist_ACLRules'>>>();
   for (const ruleRecord of sortBy(rulesTable.getRecords(), 'rulePos')) {
@@ -472,7 +497,6 @@ function readAclRules(docData: DocData, {log, compile, includeHelperCols}: ReadA
     const resourceRec = resourcesTable.getRecord(resourceId);
     if (!resourceRec) {
       throw new Error(`ACLRule ${rules[0].id} refers to an invalid ACLResource ${resourceId}`);
-      continue;
     }
     if (!resourceRec.tableId || !resourceRec.colIds) {
       // This should only be the case for the old-style default rule/resource, which we
@@ -482,7 +506,7 @@ function readAclRules(docData: DocData, {log, compile, includeHelperCols}: ReadA
     const tableId = resourceRec.tableId;
     const colIds = resourceRec.colIds === '*' ? '*' : resourceRec.colIds.split(',');
 
-    if (includeHelperCols && Array.isArray(colIds)) {
+    if (enrichRulesForImplementation && Array.isArray(colIds)) {
       colIds.push(...getHelperCols(docData, tableId, colIds, log));
     }
 
@@ -506,7 +530,13 @@ function readAclRules(docData: DocData, {log, compile, includeHelperCols}: ReadA
       } else if (rule.aclFormula && !rule.aclFormulaParsed) {
         throw new Error(`ACLRule ${rule.id} invalid because missing its parsed formula`);
       } else {
-        const aclFormulaParsed = rule.aclFormula && JSON.parse(String(rule.aclFormulaParsed));
+        let aclFormulaParsed = rule.aclFormula && JSON.parse(String(rule.aclFormulaParsed));
+        // If we have "virtual" rules to implement shares, then regular
+        // rules need to be tweaked so that they don't apply when the
+        // share is active.
+        if (hasShares && rule.id >= 0) {
+          aclFormulaParsed = shareRules.transformNonShareRules({rule, aclFormulaParsed});
+        }
         body.push({
           origRecord: rule,
           aclFormula: String(rule.aclFormula),