From 14e2649f3437c24624175f7baf498da06d7d8d92 Mon Sep 17 00:00:00 2001 From: "Karl M. Davis" Date: Sat, 26 Jun 2021 22:54:05 -0400 Subject: [PATCH] Drafted plan and stories for Round 1 release. --- dev/plans/0001-round-1.md | 54 +++++++++++++++++++ ...-synthetic-data.md => 0003-sample-data.md} | 2 +- dev/stories/0010-hapi-post-org-timeouts.md | 12 +++++ .../0011-operation-failure-debugging.md | 25 +++++++++ dev/stories/0012-ibm-fhir.md | 7 +++ dev/stories/0013-refactor-dockerfiles.md | 9 ++++ dev/stories/0014-cache-sample-data-in-s3.md | 5 ++ dev/stories/0015-tracing.md | 15 ++++++ .../0016-more-organization-operations.md | 12 +++++ .../0017-simplify-adding-operations.md | 5 ++ dev/stories/0018-analyze-synthea-output.md | 15 ++++++ dev/stories/0019-timeseries-data.md | 17 ++++++ dev/stories/0020-automate-runs-in-cloud.md | 11 ++++ dev/stories/0021-ci-tlc.md | 10 ++++ 14 files changed, 198 insertions(+), 1 deletion(-) create mode 100644 dev/plans/0001-round-1.md rename dev/stories/{0003-synthetic-data.md => 0003-sample-data.md} (98%) create mode 100644 dev/stories/0010-hapi-post-org-timeouts.md create mode 100644 dev/stories/0011-operation-failure-debugging.md create mode 100644 dev/stories/0012-ibm-fhir.md create mode 100644 dev/stories/0013-refactor-dockerfiles.md create mode 100644 dev/stories/0014-cache-sample-data-in-s3.md create mode 100644 dev/stories/0015-tracing.md create mode 100644 dev/stories/0016-more-organization-operations.md create mode 100644 dev/stories/0017-simplify-adding-operations.md create mode 100644 dev/stories/0018-analyze-synthea-output.md create mode 100644 dev/stories/0019-timeseries-data.md create mode 100644 dev/stories/0020-automate-runs-in-cloud.md create mode 100644 dev/stories/0021-ci-tlc.md diff --git a/dev/plans/0001-round-1.md b/dev/plans/0001-round-1.md new file mode 100644 index 0000000..20a9c07 --- /dev/null +++ b/dev/plans/0001-round-1.md @@ -0,0 +1,54 @@ +# Round 1 Release Plan + +This will be the first official release and also + the publishing of the first permanent benchmark results. +In general, the goals here are: + +1. Provide enough data to help FHIR API server users + make better-informed decisions than they otherwise could, + without going so overboard that the release never happens. +2. Provide enough data to help FHIR API server implementors + understand how their performance stands relative to other implementations + and also enough information for them to work on improving their performance, + if they'd like to. +3. Ensure that the results are reasonably stable / repeatable. +4. Make it reasonably simple to add new benchmarks. + +It's worth pointing out some goals that are neat & cool & stuff + but are nevertheless explicitly not in scope for this first release + (in the interests of ensuring that there _is_ a first release sometime this century): + +* Make it easy for implementors to incorporate these benchmarks + into their build processes. +* Add support for SaaS-only FHIR API server implementations. +* Calculate the cost per request served for each FHIR API server. + + +## Detailed Plan + +The following user stories are currently planned to be in scope for this release: + +* [x] [Compare Performance of FHIR Servers](../stories/0001-perf-compare.md) +* [x] [Continuous Integration](../stories/0002-ci.md) +* [x] [Sample Data](../stories/0003-sample-data.md) +* [x] [Support Firely Spark](../stories/0005-firely-spark.md) +* [x] [Increase Detail in the Application's Errors](../stories/0007-error-details.md) +* [x] [GitHub README](../stories/0008-readme.md) +* [x] [Publish Results to a Website](../stories/0009-publish-results.md) +* [ ] [IBM FHIR](../stories/0012-ibm-fhir.md) +* [ ] [Improve Management of FHIR Server Dockerfiles](../stories/0013-refactor-dockerfiles.md) +* [ ] [Cache Sample Data in S3](../stories/0014-cache-sample-data-in-s3.md) +* [ ] [Tracing](../stories/0015-tracing.md) +* [ ] [Support More `Organization` Operations](../stories/0016-more-organization-operations.md) +* [ ] [Make it Simpler to Add Benchmark Operations](../stories/0017-simplify-adding-operations.md) +* [ ] [Analyze Synthea Output](../stories/0018-analyze-synthea-output.md) +* [ ] [Support `Patient` Resource Operations](../stories/0004-patient-ops.md) +* [ ] [Timeseries Data: Latency, Operation Count, Request Size](../stories/0019-timeseries-data.md) +* [ ] [Improve Debugging of Operation Failures](../stories/0011-operation-failure-debugging.md) +* [ ] [Automate Benchmark Runs in Cloud](../stories/0020-automate-runs-in-cloud.md) +* [ ] [Give the CI Some TLC](../stories/0021-ci-tlc.md) + +In addition, the following bugs are currently planned to be fixed: + +* [x] [HAPI Failures After Launch](../stories/0006-hapi-startup-wait.md) +* [ ] [HAPI 'POST /Organization' Failures With Timeouts](../stories/0010-hapi-post-org-timeouts.md) \ No newline at end of file diff --git a/dev/stories/0003-synthetic-data.md b/dev/stories/0003-sample-data.md similarity index 98% rename from dev/stories/0003-synthetic-data.md rename to dev/stories/0003-sample-data.md index 88acbed..b54b1ca 100644 --- a/dev/stories/0003-synthetic-data.md +++ b/dev/stories/0003-sample-data.md @@ -1,4 +1,4 @@ -# Synthetic Data +# User Story: Sample Data As a reader/consumer of benchmarks, I need the data used by the FHIR servers when they're being genchmarked to be realistic, diff --git a/dev/stories/0010-hapi-post-org-timeouts.md b/dev/stories/0010-hapi-post-org-timeouts.md new file mode 100644 index 0000000..774826a --- /dev/null +++ b/dev/stories/0010-hapi-post-org-timeouts.md @@ -0,0 +1,12 @@ +# Bug: HAPI 'POST /Organization' Failures With Timeouts + +Getting a lot of operation failures with the benchmarks logging this: + +``` +{ + "msg": "Operation 'POST /Organization' failed: 'ServerOperationIterationState { _inner: ServerOperationIterationFailed { completed: ServerOperationIterationCompleted { start: ServerOperationIterationStarting { started: 2021-05-31T18:22:55.066093905Z }, completed: 2021-05-31T18:23:05.066276646Z }, error: Operation timed out: 'future has timed out' } }", + "level": "WARN", + "ts": "2021-05-31T18:23:45.09314212000:00" +} + +These consistently pop up with more concurrency, e.g. when running on `eddings` about a quarter of requests at `concurrent_users: 10` are failing due to this. \ No newline at end of file diff --git a/dev/stories/0011-operation-failure-debugging.md b/dev/stories/0011-operation-failure-debugging.md new file mode 100644 index 0000000..008be16 --- /dev/null +++ b/dev/stories/0011-operation-failure-debugging.md @@ -0,0 +1,25 @@ +# User Story: Improve Debugging of Operation Failures + +At higher concurrency levels right now, + I'm seeing a lot of operation timeout failures for both HAPI and Spark. + +However, the only way to debug those failures + is to stare at the log output during a benchmark run, + wait for failures to get logged, + and then quickly try to run `docker logs ...` for the container. +As debugging experiences go, + this is bad. + +Instead, I think we need to start collecting logs for the FHIR servers, + writing those logs to disk, + and then referencing all of the log file locations in the JSON output. + +## Details + +* For larger benchmark runs, these log files are liable to eat a lot of disk space, + perhaps even enough to exhaust the benchmark system's free space. + I may want to compress them upfront, to help mitigate this. +* Haven't spent some time trying out the goofy manual debugging procedure above for HAPI timeouts, + it's not clear to me that these logs + will actually provide enough information to diagnose the problems. + Nevertheless, this seems like a necessary first step towards improving debugging in general. \ No newline at end of file diff --git a/dev/stories/0012-ibm-fhir.md b/dev/stories/0012-ibm-fhir.md new file mode 100644 index 0000000..fea616a --- /dev/null +++ b/dev/stories/0012-ibm-fhir.md @@ -0,0 +1,7 @@ +# User Story: Support [IBM FHIR Server](https://github.com/IBM/FHIR) + +As a benchmark user, + I'd like to see the benchmarks include + [IBM FHIR Server](https://github.com/IBM/FHIR), + so that I can understand its performance, + relative to other FHIR server implementations. \ No newline at end of file diff --git a/dev/stories/0013-refactor-dockerfiles.md b/dev/stories/0013-refactor-dockerfiles.md new file mode 100644 index 0000000..6bffd16 --- /dev/null +++ b/dev/stories/0013-refactor-dockerfiles.md @@ -0,0 +1,9 @@ +# User Story: Improve Management of FHIR Server Dockerfiles + +As a benchmark contributor, + I'd like the various `Dockerfile`s to be better managed + so that it's easy to understand what version things are pinned to + and so that it's easy to update that. +I'd also like to either stop using `git submodule`s for those + or at least to better document how to work with them + (because I can never remember). \ No newline at end of file diff --git a/dev/stories/0014-cache-sample-data-in-s3.md b/dev/stories/0014-cache-sample-data-in-s3.md new file mode 100644 index 0000000..42ab9d3 --- /dev/null +++ b/dev/stories/0014-cache-sample-data-in-s3.md @@ -0,0 +1,5 @@ +# User Story: Cache Sample Data + +As a user of the benchmark application, + I would like it to cache the sample data it generates in S3, + so that I can run benchmarks more quickly. \ No newline at end of file diff --git a/dev/stories/0015-tracing.md b/dev/stories/0015-tracing.md new file mode 100644 index 0000000..dd8ac0a --- /dev/null +++ b/dev/stories/0015-tracing.md @@ -0,0 +1,15 @@ +# User Story: Switch to [Tracing](https://lib.rs/crates/tracing) for Logs + +As a user of the benchmark application, + I would like the logs to provide more information on causality, + so that I'm better able to diagnose issues when they're encountered. + + +# Details + +* I'll be honest: I don't have a real good use case or burning need for this right now + but, rather, I've been watching Tracing for a while now and I'm intrigued by it. + I mostly just want to try it out and see if it works well. +* In addition, I think it's probably time to move away from NDJSON log output, + as it's mostly just making the log output less useful right now. + * And Tracing also supports NDJSON output, if it turns out to be needed in the future. \ No newline at end of file diff --git a/dev/stories/0016-more-organization-operations.md b/dev/stories/0016-more-organization-operations.md new file mode 100644 index 0000000..0b98ebd --- /dev/null +++ b/dev/stories/0016-more-organization-operations.md @@ -0,0 +1,12 @@ +# User Story: Support More `Organization` Operations + +As a benchmark consumer, + I would like the benchmarks to cover more operations on the `Organization` resource, + so that I can see how reads, searches, etc. perform. + + +## Details + +* The `Organization` resource isn't particularly compelling, + but is instead selected here as we already have support for its `POST` operation, + so adding support for more operations should be less of a lift. \ No newline at end of file diff --git a/dev/stories/0017-simplify-adding-operations.md b/dev/stories/0017-simplify-adding-operations.md new file mode 100644 index 0000000..ff7d284 --- /dev/null +++ b/dev/stories/0017-simplify-adding-operations.md @@ -0,0 +1,5 @@ +# User Story: Make it Simpler to Add Benchmark Operations + +As a benchmark contributor, + I would like the benchmark operations to be refactored and better documented, + such that adding new benchmark operations is more straightforward. \ No newline at end of file diff --git a/dev/stories/0018-analyze-synthea-output.md b/dev/stories/0018-analyze-synthea-output.md new file mode 100644 index 0000000..9f2f21f --- /dev/null +++ b/dev/stories/0018-analyze-synthea-output.md @@ -0,0 +1,15 @@ +# User Story: Analyze Synthea Output + +As a benchmark contributor, + I would like the logs to report on how many resources Synthea produced, + broken out by resource type and total storage size, + so that I have a better idea which operations I might want to add support for next. + + +## Details + +* This will help to determine which resource's operations + will most stress servers in terms of data volume. +* I suspect that `Patient` resources are actually not the most common, + and perhaps there are far more `Encounter` resources, + but it's really just a guess. \ No newline at end of file diff --git a/dev/stories/0019-timeseries-data.md b/dev/stories/0019-timeseries-data.md new file mode 100644 index 0000000..a65a09c --- /dev/null +++ b/dev/stories/0019-timeseries-data.md @@ -0,0 +1,17 @@ +# User Story: Timeseries Data: Latency, Operation Count, Request Size + +As a benchmark contributor and/or FHIR server implementer, + I would like the benchmark application to output its request data + to a timeseries database and analysis suite, + such as [InfluxDB](https://www.influxdata.com/products/influxdb/) + plus [Grafana](https://grafana.com/), + so that I can analyze the performance of my FHIR server + over time during each benchmark operation. + + +## Details + +* This is something Wind Tunnel provides. +* I've heard from FHIR server implementers that this is an important feature for them. +* For some FHIR servers, they might have native support for InfluxDB, + and it'd be interesting to turn that on during benchmarking. \ No newline at end of file diff --git a/dev/stories/0020-automate-runs-in-cloud.md b/dev/stories/0020-automate-runs-in-cloud.md new file mode 100644 index 0000000..cbf49b1 --- /dev/null +++ b/dev/stories/0020-automate-runs-in-cloud.md @@ -0,0 +1,11 @@ +# User Story: Automate Benchmark Runs in Cloud + +As a benchmark contributor, + I would like the official benchmark runs (in the cloud) to be fully automated, + so that I can reliably reproduce their results. + + +## Details + +* Until we support SaaS/cloud-service FHIR server implementations, + all official benchmark runs should be performed on EC2 metal instances. \ No newline at end of file diff --git a/dev/stories/0021-ci-tlc.md b/dev/stories/0021-ci-tlc.md new file mode 100644 index 0000000..fc5735c --- /dev/null +++ b/dev/stories/0021-ci-tlc.md @@ -0,0 +1,10 @@ +# User Story: Timeseries Data: Latency, Operation Count, Request Size + +As a benchmark contributor, + I would like the CI setup for the benchmark application to get some TLC, + so that its runs are as reliable and fast as is reasonable. + + +## Details + +* I don't have any specific problems to resolve yet; I just expect that I will, eventually. \ No newline at end of file