Dev mode for indexer as well

[vividn]

Problem

The dev mode for the sandbox is a nice feature that allows for rapid transactions to be made during development and testing. However, the indexer does not keep up with algod in this mode. In fact, it takes longer for the indexer to recognize new transactions and add them to the index, than it does in "normal" mode for the transaction to confirm and then be added to the index. Because of this we cannot use the dev mode because we need to get data from the indexer as part of our process.

Steps to reproduce:

const indexerPort = 8980;
const baseServer = "http://localhost";
const token = { 'X-Algo-API-Token': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' };
const port = 4001;
const algosdk = require('algosdk');
const indexer = new algosdk.Indexer(token, baseServer, indexerPort);
const algod = new algosdk.Algodv2(token, baseServer, port); kmdToken = { 'X-KMD-API-Token': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' };
const account = algosdk.mnemonicToSecretKey('knock olive prevent shield poem ranch card movie sudden jeans wrist december stock manual lucky umbrella twist wet output surround venture limit jar abstract jelly');
// fill up RDCLUJUWYJP77SQAG5HSTTW3X7WWMKMG6VLGE3S7IZR6LKMYDJR3YZ5BPY with ALGOs

function sleep(seconds) {
  const ms = seconds * 1000;
  return new Promise((resolve) => setTimeout(resolve, ms));
}

async function waitForTx(txId) {
  while (true) {
  try {
    await indexer.lookupTransactionByID(txId).do()
    break;
  } catch (e) {
    await sleep(0.5)
  }
}
}

async function timeIndexer() {
  const params = await algod.getTransactionParams().do();

  const txn = {
    ...params,
    to: account.addr,
    from: account.addr,
    amount: 0
  }

  const signed = algosdk.signTransaction(txn, account.sk);

  // Start timer
  console.time("indexer confirmed in")

  const sent = await algod.sendRawTransaction(signed.blob).do();
  const txId = sent.txId;
  await waitForTx(txId);

  console.timeEnd("indexer confirmed in")
}

with dev mode:

> await timeIndexer(); await timeIndexer(); await timeIndexer(); await timeIndexer(); await timeIndexer();
indexer confirmed in: 24.345s
indexer confirmed in: 1:00.348 (m:ss.mmm)
indexer confirmed in: 59.795s
indexer confirmed in: 1:00.264 (m:ss.mmm)
indexer confirmed in: 59.758s

consecutive calls of await timeIndexer() show that the indexer only checks for new transactions every minute, and therefore cause delays in tests by up to one minute, making dev mode unusable with indexer based things

with standard sandbox:

> await timeIndexer(); await timeIndexer(); await timeIndexer(); await timeIndexer(); await timeIndexer();
indexer confirmed in: 11.207s
indexer confirmed in: 12.182s
indexer confirmed in: 12.145s
indexer confirmed in: 12.178s
indexer confirmed in: 11.637s

Solution

Have dev mode algod ping the indexer somehow to try to get new transactions whenever a block is created.
Or at minimum reduce the dev mode indexer to check for transactions more often (every few seconds or so)

Dependencies

Urgency

The dev mode is unusable for us (and I'm sure many other groups as well) without rapid indexing.

[reischapa]

@vividn

Ran into issues related to this today as well. +1 for this feature.

[robdmoore]

+1

[robdmoore]

More context here. The Reach Algorand dev container has some patches they apply to get this kind of experience and on an automated test I wrote it went from 37s to 7s when switching from Algorand Sandbox to the reach version.

The only problem is the reach sandbox doesn't expose Kmd so I can't automate getting the default wallet (although I can hardcode the address they encoded so there is a workaround for that) and is on an old version of Algorand sandbox so I can't use it since it doesn't have the latest features and my smart contract doesn't work :(

[DamianB-BitFlipper]

@robdmoore Do you mind linking to this Reach Algorand dev container patches. I am wondering if it would not be too difficult to port over these patches to the Python/TEAL indexer.

I too am having this behavior with ./sandbox up dev not registering recently sent transactions immediately. It is always 1 block behind algod.

[robdmoore]

I can do you one better.

We've published images to docker hub with the patch applied, but the latest version of Algorand and exposing Kmd (unlike the Reach container): https://github.com/MakerXStudio/algorand-sandbox-dev

[winder]

This should be working in the next release. The off by one error causing this trouble is fixed here: algorand/indexer#920