UTRS Appeal transfer

[stwalkerster]

Re-hashing #26...

Thoughts have again resurfaced about transferring UTRS requests over to ACC. UTRS could send us email, provided IP, webserver IP, UA, block info, and the generic appeal reason. The initial thoughts we've had revolve around a HTTP API between ACC and UTRS, where UTRS will transfer the relevant appeal data to ACC, ACC returns a token to UTRS, then UTRS can redirect the user back to ACC with the token for ACC to take over handling of the request.

Below is a quick sequence diagram for what I initially expect the process to look like:

sequenceDiagram
  
actor BlockedUser
actor Admin
participant UTRS
participant ACC

note over BlockedUser,UTRS: Standard UTRS process happens first


Admin->>UTRS: Approval of transfer
UTRS->>ACC:Send transferral request (with all necessary appeal data + appeal id)
ACC-->>UTRS:Token issued on receipt of data
UTRS-->>Admin: Notification of successful transfer

UTRS->>BlockedUser: Notification (email) containing link to ACC with token
UTRS->>UTRS: Close request as transferred
BlockedUser->>ACC:User clicks link to ACC with token
ACC-->>BlockedUser:ACC presents form for remaining data
BlockedUser->>ACC:Submission of missing data

ACC->>UTRS: Notification of ACC request ID (using token)
UTRS-->>ACC: Acknowledgement

ACC->>ACC:Invalidate token for new requests

note over ACC:ACC team handles request

ACC->>BlockedUser: Notification of account creation


ACC->>UTRS: Notification of ACC request status (using token)
UTRS-->>ACC: Acknowledgement

Loading

The HTTP API will be secured via a HMAC-based authentication system. We'll agree some way of organising the fields and hashing them with a shared-secret key. The result will be transmitted with the request, and the recipient will perform the same calculation. If the results match, then the sender must have the same key as the recipient.

This is still a work-in-progress to figure out exactly what and how we're going to attempt this.

---

API details

Two APIs need to be written, one ACC's side and one on UTRS' side.

These APIs are essentially both the same design and use the same format.

ACC "data transfer" API

ACC will present an API (eg: /internal.php/api/utrsinbound) which will accept a JSON document. This API is to be used to receive appeal data from UTRS as the first part of the transfer process.

---

Data Transfer API definition

The JSON document received by ACC should consist of a single object containing the following keys:

Field	Type	Description
nonce	string	Single-use token to prevent replay attacks. This must be different on each API call, even if all the other data is identical. ACC should validate that is is unique
utrsId	integer	ID of the UTRS appeal
email	string	Email address of the requester
emailConfirmed	bool	Whether the email address is already validated by UTRS.
networkIdentity	object[]	Network information stored for the appeal (see below)
blockInfo	object[]	Information about the block. TODO: figure out what info is available and from that what is useful
queueId	int? (nullable int)	ACC queue to defer request into
domain	string	ACC domain short name (eg enwiki)
comments	object[]	Comments to automatically create on the ACC request

comment is an array of objects. each object is:

Field	Type	Description
comment	string	Content of comment
origin	string	Who created the comment "utrs" or "admin" or "user"
username	string?	If origin==admin, the username of the admin, null otherwise

blockInfo is an array of objects. each object is:

Field	Type	Description
comment	string	Block reason
admin	string	Who set the block

networkIdentity should be an array of objects, with each object having the following information:

Field	Type	Description
ipv4	string?	IPv4 address
ipv6	string?	IPv6 address
ua	string?	User agent + client hints
trusted	bool	Is this data trusted (captured/validated by UTRS, or user-provided?)

---

This must be delivered as a HTTP POST request, with the following HTTP headers set:

Content-Type: application/json
Cache-Control: no-store
Authorization: Bearer 123456abcdef
X-ACC-API-Version: 1

The Authorization header value must be the word Bearer followed by a SHA-384 HMAC of the entire body content as a string. This can be generated in PHP with the hash_hmac('sha384', $data, $key) function. We will agree on a shared key.

If the data format of the request or response body needs to change at a later date, we will bump the X-ACC-API-Version header as required.

All other headers should be sent with the exact values shown above.

---

Example

{
    "nonce": "EsczJwh9cfY8hXrL5tDdlh6onOhYGpFBjKbhipdbhpM"
    "utrsId":2345,
    "email":"[email protected]",
    "emailConfirmed":true,
    "networkIdentity":[
        {
            "ip":"127.0.0.15",
            "ch":null,
            "trusted":true,    // checkuser data
        },
        {
            "ip":"1.2.3.4",
            "ua":null,
            "trusted":false,     // appeal for
        }
    ],
    "blockInfo":[
        {
           "comment":"for the lolz",
           "admin":"AmandaNP"
        },
        {
           "comment":"no u",
           "admin":"Stwalkerster"
        }
    ],
    "domain":  "enwiki"
    "queueId": null,
    "comments": [
       {
           "comment": "I like json",
           "origin": "admin"
       },
       {
           "comment": "other info",
           "origin": "utrs"
       }
    ]
}

---

ACC will then respond with a HTTP response. For example, the request above may produce the below response.

Headers:

Content-Type: application/json
Cache-Control: no-store

Body:

{
    "status":"OK",
    "error":null,
    "utrsId":2345,
    "token":"123456abcdef",
    "url":"https://accounts.wmflabs.org/index.php/utrs?t=123456abcdef"
}

---

Data Transfer API response definition

Field	Type	Description
status	string	Either "OK" or some other value. Only "OK" indicates success.
error	string or null	If status is "OK", then this field will be null. Otherwise, this field will contain an error message.
utrsId	integer	ID of the UTRS appeal
token	string	The token representing the received data at ACC. This token must be provided to ACC to create an account request linked to this UTRS appeal.
url	string	The URL to direct the user to in order to complete their appeal. This field should be an opaque URL to UTRS and provided as-is to the appellant. This URL will include the token value above

---

UTRS should check the status field is "OK", and then pass the token and/or URL back to the user. If the status field is not "OK", a relevant error message will be placed in the "error" field of the response.

UTRS request notification

UTRS will present an API which will accept a JSON document. The JSON document should consist of a single object containing the following keys:

---

Request status request definition

Field	Type	Description
nonce	string	Single-use token to prevent replay attacks. This must be different on each API call, even if all the other data is identical. UTRS should validate that it is unique.
utrsId	integer	ID of the UTRS appeal
token	string	The token representing the received data at ACC.
accId	integer	ID of the ACC request
status	string	Current status of the request. Will be either open or closed
closureType	string?	Further information about the request closure reason

---

This must be delivered as a HTTP POST request, with the following HTTP headers set:

Content-Type: application/json
Cache-Control: no-store
Authorization: Bearer 123456abcdef
X-UTRS-API-Version: 1

See notes in the ACC section about the meaning of the headers.

Example:

{
    "nonce": "onOSnXh81TNvhNHf7m_IRhPAQcORtoig5Qf1pLx2bK0"
    "utrsId": 2345,
    "token": "123456abcdef",
    "accId": 345678,
    "status": "open"
}

UTRS will then respond with a HTTP response. For example, the request above may produce the below response.

Headers:

Content-Type: application/json
Cache-Control: no-store

Body:

{
    "status":"OK",
    "error":null
}

---

Request status response definition

Field	Type	Description
status	string	Either "OK" or some other value. Only "OK" indicates success.
error	string or null	If status is "OK", then this field will be null. Otherwise, this field will contain an error message.

---

[dqwiki]

@stwalkerster After reviewing and rethinking today, additionally, I would actually prefer a report back with the result of the appeal transfer (created, not created, potentially templated reason - not the actual text). That said, this can very much be a down-the-road request and does not have to be implemented on release.

Motivation

[stwalkerster]

I would actually prefer a report back with the result of the appeal transfer (created, not created, potentially templated reason - not the actual text).

Understood, I'll try to factor that into the design. Bear in mind that it's possible you may get multiple notifications from ACC as it's possible for a request to be closed, reopened, and closed again. What I'll probably do is send UTRS every status update, and leave it to UTRS to decide what to do with that data. (Remember that status update does not include queue changes, it'll just be open/close with reason)

What I might actually do is use the UTRS-side API as a general status reporting system so it's used for both telling you about the request ID, and updating you on the request's status. I need to think a little more about this though.

---

I'm also going to make some more tweaks to the API definition. I want to split out ip in networkIdentity into ipv4 and ipv6, though if this causes problems for you I'm not too attached to it.

[stwalkerster]

OK, the other change I've made is to remove the validation step for the HTTP response of requests. I've also done some layout tweaks to the description above, putting some stuff in collapsible sections, and moving the diagram from an image to GitHub's native diagramming

[dqwiki]

@stwalkerster
A response can contain up to 4 items:

Field name	Meaning	Comments
status	----	As requested
code?	HTTP error code	Will be zero if none
api_error_code	An internal "track this error down" code for UTRS	0 is no error, otherwise errors start at 1000
message	Error message	String of the error

• I will need clarity on how we want the single use token generated, stored and checked.
• I am creating a master token-generating interface at its included partially in native features - so you can regenerate it as you see fit
• The only real thing remaining is setting up the security - the response is already as expected otherwise

[stwalkerster]

A response can contain up to 4 items:

Is this for the request status response?

I will need clarity on how we want the single use token generated, stored and checked.

How does something like this sound for the token?

time() . "." . base64_encode(random_bytes(24));

When it's received, split it on the . and store the two parts in a database table as two separate columns. If the first part compared to the current time is more than say 10 minutes, treat it as invalid. If the second part is in the database table already, treat it as invalid. Then you can have a regular cleanup job on the database table for old entries as things outside the time window will always be invalid.