Checking if we have the up to date document _rev

[dsebastien]

What is the simplest way to verify if the revision of a document that we currently hold is the latest one in Couch?

Background:
I have a simple Web application that sends partial update requests to a back-end that exposes a RESTful API. The details don't matter too much, but when that back-end receives the update request, it first needs to check if the user had the latest version of the document. If that's not the case then the API will return a 412 error back to the client (conditional request with If-Match for the rev); instructing it to refresh the data.

[jjrodrig]

Yo can use the same mechanism interacting with CouchDB. Use HEAD operation for a document

HEAD /db/doc
HEADER If-None-Match="<rev>"

If you have the same revision than the one is the current winning revision, it will respond 304 Not Modified

You'll find the same behaviour with the GET /db/doc operation

[wohali]

One slightly academic note, CouchDB does the same work for a HEAD as a GET on the backend. Therefore unless you need to save the amount of traffic going over the network, you can just use a GET and throw away everything other than the _rev value.

[janl]

This is answered already, but it got me thinking: which is faster: reading out a doc, and discarding the body before returning it (HEAD), or just looking up the doc metadata in _all_docs?key=doc_id and getting the result that way.

I got myself various databases with varying doc sizes (see below), all databases have 1M docs, that are generated with benchbulk.sh.

I used these contraptions to get the timing between the last byte sent to the server and first response byte in curl -v --trace-time`:

Get HEAD timing: curl -sIv --trace-time http://admin:[email protected]:5984/benchbulk/00000000000000009318 2>&1 | grep -A2 Accept | tail -2 | awk '{print $1}' | sed -e 's/[:.]//g' | awk 'p{print $0-p}{p=$0}'

Get _all_docs timing: curl -sv --trace-time http://admin:[email protected]:5984/benchbulk/_all_docs?key=\"00000000000000009318\" 2>&1 | grep -A2 Accept | tail -2 | awk '{print $1}' | sed -e 's/[:.]//g' | awk 'p{print $0-p}{p=$0}'

HEAD timing:

RESP=`curl -sIv --trace-time http://admin:[email protected]:5984/benchbulk-1/00000000000000500000 2>&1`; SENT_TS=$(echo "$RESP" | grep -Eo \(.\{15\}\)\ \>\ Accept | sed -e 's/ > Accept//' | sed -e 's/[:.]//g'); END_TS=$(echo "$RESP" | grep -Eo \(.\{17\}\)\ ?\ Connection | sed -e 's/\ \*\ Connection//' | sed -e 's/[.:]//g'); echo $(($END_TS-$SENT_TS))

_all_docs timing:

RESP=`curl -sv --trace-time http://admin:[email protected]:5984/benchbulk-1/_all_docs?key=\"00000000000000500000\" 2>&1`; SENT_TS=$(echo "$RESP" | grep -Eo \(.\{15\}\)\ \>\ Accept | sed -e 's/ > Accept//' | sed -e 's/[:.]//g'); END_TS=$(echo "$RESP" | grep -Eo \(.\{17\}\)\ ?\ Connection | sed -e 's/\ \*\ Connection//' | sed -e 's/[.:]//g'); echo $(($END_TS-$SENT_TS))

With 1 byte docs, there is no difference detectable, responses come in at 900-1400 microseconds.
With 100 byte docs, there is no difference detectable, responses come in at 900-1400 microseconds.
With 1000 byte docs, there is no difference detectable, responses come in at 900-1400 microseconds.
With 10000 byte docs, there is a slight difference, responses come in at 900-1400 microseconds with _all_docs, no surprise here, but HEAD requests to docs go to the 1200-1700 microsecond range.

UPDATE:

h/t @garrensmith, no measuring until the end of the request, to make it a fairer comparison, AND to account for hate act that CouchDB will send the view result metadata before doing the actual lookup:

1 byte docs: HEAD: 1100-1600 microseconds, _all_docs: 1300-1700 microseconds
10 byte docs: HEAD: 1100-1600 microseconds, _all_docs: 1300-1700 microseconds
100 byte docs: HEAD: 1100-1600 microseconds, _all_docs: 1300-1700 microseconds
1000 byte docs: HEAD: 1200-1600 microseconds, _all_docs: 1300-1700 microseconds
10000 byte docs: HEAD: 1600-1900 microseconds, _all_docs: 1300-1700 microseconds

The stock _all_docs response body with one row is 164 bytes, for comparison, with headers, it is 803, and doc headers for HEAD requests are ~928 bytes.

I stopped after this ;)

In conclusion: there is one more way to do this, if you don’t need the If-None-Match feature, but it isn’t any faster, unless you start hitting doc sizes that you might wanna avoid anyway, if if the response body handling is not in your favour.

Note that this is a rather speed SSD system here (recent Mac mini) and fast CPUs.

I’m running the tests many times to avoid caching artefacts, but that means all access should be cached. So you’d likely see more differences on slower disk and for uncached requests.

HEAD doc example trace:

curl -sIv --trace-time http://admin:[email protected]:5984/benchbulk/00000000000000500000 
19:56:35.553043 *   Trying 127.0.0.1...
19:56:35.553309 * TCP_NODELAY set
19:56:35.553408 * Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
19:56:35.553425 * Server auth using Basic with user 'admin'
19:56:35.553448 > HEAD /benchbulk/00000000000000500000 HTTP/1.1
19:56:35.553448 > Host: 127.0.0.1:5984
19:56:35.553448 > Authorization: Basic YWRtaW46YWRtaW4=
19:56:35.553448 > User-Agent: curl/7.54.0
19:56:35.553448 > Accept: */*
19:56:35.553448 > 
19:56:35.555484 < HTTP/1.1 200 OK
HTTP/1.1 200 OK
19:56:35.555510 < Cache-Control: must-revalidate
Cache-Control: must-revalidate
19:56:35.555525 < Content-Length: 100085
Content-Length: 100085
19:56:35.555539 < Content-Type: application/json
Content-Type: application/json
19:56:35.555553 < Date: Sat, 11 Jul 2020 17:56:34 GMT
Date: Sat, 11 Jul 2020 17:56:34 GMT
19:56:35.555568 < ETag: "1-0367f07906b3fe66f3d987e2f7dd9f0d"
ETag: "1-0367f07906b3fe66f3d987e2f7dd9f0d"
19:56:35.555585 < Server: CouchDB/2.3.1 (Erlang OTP/21)
Server: CouchDB/2.3.1 (Erlang OTP/21)
19:56:35.555600 < X-Couch-Request-ID: 851a910ec8
X-Couch-Request-ID: 851a910ec8
19:56:35.555614 < X-CouchDB-Body-Time: 0
X-CouchDB-Body-Time: 0

19:56:35.555628 < 
19:56:35.555644 * Connection #0 to host 127.0.0.1 left intact

_all_docs example trace:

curl -sv --trace-time http://admin:[email protected]:5984/benchbulk/_all_docs?key=\"00000000000000500000\"
19:57:12.729367 *   Trying 127.0.0.1...
19:57:12.729595 * TCP_NODELAY set
19:57:12.729731 * Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
19:57:12.729765 * Server auth using Basic with user 'admin'
19:57:12.729794 > GET /benchbulk/_all_docs?key="00000000000000500000" HTTP/1.1
19:57:12.729794 > Host: 127.0.0.1:5984
19:57:12.729794 > Authorization: Basic YWRtaW46YWRtaW4=
19:57:12.729794 > User-Agent: curl/7.54.0
19:57:12.729794 > Accept: */*
19:57:12.729794 > 
19:57:12.737763 < HTTP/1.1 200 OK
19:57:12.737789 < Cache-Control: must-revalidate
19:57:12.737802 < Content-Type: application/json
19:57:12.737815 < Date: Sat, 11 Jul 2020 17:57:12 GMT
19:57:12.737827 < Server: CouchDB/2.3.1 (Erlang OTP/21)
19:57:12.737840 < Transfer-Encoding: chunked
19:57:12.737851 < X-Couch-Request-ID: e523e76cd6
19:57:12.737863 < X-CouchDB-Body-Time: 0
19:57:12.737875 < 
{"total_rows":1000000,"offset":500000,"rows":[
{"id":"00000000000000500000","key":"00000000000000500000","value":{"rev":"1-0367f07906b3fe66f3d987e2f7dd9f0d"}}
]}
19:57:12.737918 * Connection #0 to host 127.0.0.1 left intact

[dsebastien]

Awesome, thanks for the detailed info ;-)