Benchmarking

[goodmami]

It would be very useful to have a benchmarking system in place, as some changes to the queries or to the schema have had big effects on performance. Recently I tried adding an index to a table to speed up the removing of lexicons, but adding a lexicon slowed to a crawl.

The benchmark suite should test a range of actions:

• Adding a lexicon (insert performance)
• Removing a lexicon
• Constrained queries (tests query efficiency, indexes)
• Unconstrained queries (tests Python data structure creation)
• Path queries

It should be easy to run these benchmarks across several versions, to see timing deltas.

The pytest-benchmark package might be useful for this. Or, as it's not critical functionality, we could limit it to Linux/macOS and use some shell scripts around the time command or Python's timeit. For time, the following alias is helpful as it gets memory usage (rss) and the exit code, too:

alias bnch='/usr/bin/time -f "\
  real\t%es
  user\t%Us
  sys\t%Ss
  rss\t%MK
  exit\t%x" "$@"'

I'm labeling this issue as help wanted as it doesn't require much knowledge of Wn's internals and, however nice it would be, is not the highest priority at the moment.

[dpalmasan]

I can help with this, but it might take me some time as I need to become familiar with the code :)

[goodmami]

That would be great, thanks! Some notes:

• This should not result in any code in the wn/ subdirectory.
• If you go with shell scripts, I prefer Linux/macOS as I don't have a Windows system to test on, and the scripts should go in a subdirectory like bench/ or benchmark/ or something, with a README.md explaining how to run them.
• Or if you go with pytest-benchmark or similar, there would be a new testing dependency and the benchmarks would be implemented under the tests/ subdirectory.