Fix #459: reduce timestamptz precision

[sadiqkhoja]

Closes #459

Changes:

• Reduce precision of timestamptz columns to milliseconds across the database

What has been done to verify that this works as intended?

• Added integration test

Why is this the best possible solution? Were any other approaches considered?

• see the discussion on the issue OData filter: date/time fields more precise than millisecond #459

How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?

NA

Does this change require updates to the API documentation? If so, please update docs/api.md as part of this PR.

NA

Before submitting this PR, please make sure you have:

• run make test-full and confirmed all checks still pass OR confirm CircleCI build passes
• verified that any code from external sources are properly credited in comments

[sadiqkhoja]

There is still a separate but related issue: Only using submissionDate > xxx filter criterion can skip submision(s) in batch processing. Possible solution is to recommend users to use submissionDate > xxx and submissionDate < xxx in ETL and similar workflows

Scenario:
Assume user has ETL proceduce where they are fetching new submissions periodically by using filter submissionDate greater than submissionDate of last known submission. Then consider following sequence of events:

t0: lastTimestamp is t0
t1: new submission(1)
t1: get submissions with submissionDate > t0 -- submission(1) is returned and lastTimestamp is t1
t1: new submission(2)
t2: get submissions with submissionDate > t1 -- no submission is returned

This can always happen no matter what the database timestamp precision is. Following test captures this scenario.

it.only('should not skip submission', testService(async (service, { run }) => {
  const asAlice = await service.login('alice', identity);
  
  let lastTimestamp = '2022-01-01T09:00:00.015Z';
  
  // ****************************************************************************************
  // Simulate concurrent requests
  // Assume following three requests are executed at 2022-01-01T09:00:00.020Z
  // But there is an order at higher precision i.e. microsecond or picosecond
  // ****************************************************************************************
  {
    await asAlice.post('/v1/projects/1/forms/withrepeat/submissions')
    .send(testData.instances.withrepeat.one)
    .set('Content-Type', 'text/xml')
    .expect(200)
    .then(() => run(sql`update submissions set "createdAt"='2022-01-01T09:00:00.020Z'`));
  
    lastTimestamp = await asAlice.get(`/v1/projects/1/forms/withrepeat.svc/Submissions?$filter=__system/submissionDate gt ${lastTimestamp}`)
      .expect(200)
      .then(({ body }) => {
        body.value.length.should.be.eql(1);
        body.value[0].__id.should.be.eql('rone');
        return  body.value[0].__system.submissionDate;
      });
  
    await asAlice.post('/v1/projects/1/forms/withrepeat/submissions')
      .send(testData.instances.withrepeat.two)
      .set('Content-Type', 'text/xml')
      .expect(200)
      .then(() => run(sql`update submissions set "createdAt"='2022-01-01T09:00:00.020Z'`));
  }
  // ****************************************************************************************
  
  
  await asAlice.get(`/v1/projects/1/forms/withrepeat.svc/Submissions?$filter=__system/submissionDate gt ${lastTimestamp}`)
    .expect(200)
    .then(({ body }) => {
      body.value.length.should.be.eql(1);
      body.value[0].__id.should.be.eql('rtwo');
    });
}));

[ktuite]

It looks good, I just feel bad for all those microseconds we'll be saying goodbye to!

We have a way of writing tests for individual migrations and I was trying to think of a useful before/after migration test but I can't think of one right now. The test in this PR covers the main timestamp precision scenario we want to fix.