entity-system-benchmarks

Benchmarks description:

Tested frameworks

• artemis-odb 0.4.0, 0.9.0, 1.0.1, 1.5.0 (prel.)
• ashley 1.6.0
• gdx-artemis 0.5.0
• retanizer 0.2.0

Benchmarks explained

• All benchmarks measure throughput; higher score is better.
• Attempt to be as similar as possible across frameworks.
• Run with entityCount: 1024, 4096, 16384 and 65536.
• Share the same random seed.
• Raw, unprocessed results, here: https://github.com/junkdog/entity-system-benchmarks/tree/master/results
• Additional data:
  • OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu1)
  • Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz

Component iteration benchmarks:

• Have 3 entity systems updating a position component.
• An Entity Deleter system: each 100th tick an entity is deleted and a new entity is created the next tick.
• There are 3 types of iteration benchmarks:
  • baseline: position isn't updated, not included in charts.
  • plain: position component is the default component type.
  • pooled: pooling all components.

Insert/remove benchmarks:

• Chooses from up to 144 different entity compositions.
• Each tick, removes/inserts 1/4 of the total entityCount.
• When creating entities, a random composition is selected.
  • The composition is randomly chosen, but shared across all frameworks/benchmarks.

Add/remove components benchmarks:

• Have 3 BaselinePositionSystems.
• Each tick, toggles the Position component for 1/4 all entities.
  • Entities are chosen from a pre-shuffled list of entity ids.
• Two types of benchmarks:
  • entity_edit: Toggling achieve by calling entity.edit().create/remove
  • transmuter: Toggles using the EntityTransmuter class.

Benchmark results

As usual, writing microbenchmarks is tricky business - take the numbers with a grain of salt. These benchmarks should not be interpreted as benchmarking each framework as a whole, though they aim to give a hint of each framework's performance.

Things that might concern

• Overall, ashley-realted benchmarks may be improved; I'm not very familiar with the framework.
• I never got pooling working with Ashley-1.2.0. Ashley-1.0.1:s pooling benchmarks had an error, but considering it's now so old, I'd rather fix the benchmark for the most recent version.
• artemis-odb's PackedComponent benchmarks only have a single component implementation, skewing the benchmark results in its favor.

artemis-odb_fast benchmarks

Artemis-odb-0.7.0 added support for automatically rewriting the generated class files of EntitySystems in order to achieve higher performance. The source code itself remains identical. More about it on the artemis-odb wiki: Hotspot Optimization.

Benchmarks: 4096 entites

ECS	baseline	plain	pooled	insert/remove	edit	transmute
artemis-odb-0.4.0	10609.16	6673.70		774.12
artemis-odb-0.9.0	11776.13	10949.46	11476.16	3219.62	5435.99	6912.18
artemis-odb-0.9.0_fast	13440.43	11988.66	12502.18	3245.35	5566.49	7144.00
artemis-odb-1.0.1	12395.24	10373.89	11563.60	3060.74	7827.03	9362.37
artemis-odb-1.0.1_fast	15000.39	11546.26	12953.15	3172.14	7995.48	10208.75
artemis-odb-2.0.0-RC1	12442.58	11098.03	12209.97	6217.78	8585.09	10495.03
artemis-odb-2.0.0-RC1_fast	15010.39	11920.99	13146.79	6225.45	9274.83	11296.83
artemis-odb-2.0.0-RC2	12426.67	11093.07	12232.79	6101.39	8692.16	10723.12
artemis-odb-2.0.0-RC2_fast	15061.05	12022.86	13247.54	6300.21	9326.63	11568.29
ashley-1.6.0	9621.74	4992.19		1710.13
gdx-artemis-0.5.0	9974.36		7545.31	1545.58
retinazer-0.2.0	19965.98	10485.84		8589.60	21339.54

Benchmarks: 16384 entites

ECS	baseline	plain	pooled	insert/remove	edit	transmute
artemis-odb-0.4.0	2613.93	2027.73		156.16
artemis-odb-0.9.0	2945.44	2867.71	2871.12	519.20	1248.99	1523.36
artemis-odb-0.9.0_fast	3361.41	3115.78	3098.69	521.04	1273.50	1578.94
artemis-odb-1.0.1	3111.22	2915.28	2950.20	536.95	1831.17	2424.73
artemis-odb-1.0.1_fast	3762.55	3199.09	3292.84	541.52	2012.89	2614.00
artemis-odb-2.0.0-RC1	3052.38	3036.99	3088.56	1495.83	2109.25	2620.57
artemis-odb-2.0.0-RC1_fast	3768.35	3256.18	3282.56	1513.29	2299.33	2805.50
artemis-odb-2.0.0-RC2	3063.18	3025.16	3088.51	1495.83	2201.62	2581.02
artemis-odb-2.0.0-RC2_fast	3770.17	3280.94	3280.77	1543.03	2349.88	2817.17
ashley-1.6.0	2357.24	747.36		157.02
gdx-artemis-0.5.0	2518.07		2257.00	375.60
retinazer-0.2.0	4989.42	2779.12		1832.26	5237.94

Benchmarks: 65536 entites

ECS	baseline	plain	pooled	insert/remove	edit	transmute
artemis-odb-0.4.0	321.93	231.24		23.06
artemis-odb-0.9.0	643.59	715.33	641.39	94.49	257.62	342.33
artemis-odb-0.9.0_fast	837.87	778.02	779.91	93.59	266.31	353.99
artemis-odb-1.0.1	777.32	721.85	722.76	96.40	433.83	587.22
artemis-odb-1.0.1_fast	940.23	804.48	803.49	95.85	440.08	630.28
artemis-odb-2.0.0-RC1	754.81	761.89	764.03	324.07	531.41	623.48
artemis-odb-2.0.0-RC1_fast	942.25	798.78	818.32	334.10	570.92	688.43
artemis-odb-2.0.0-RC2	755.53	743.70	747.61	325.39	546.60	642.08
artemis-odb-2.0.0-RC2_fast	941.18	815.75	818.53	336.52	582.39	694.83
ashley-1.6.0	272.72	74.12		12.58
gdx-artemis-0.5.0	341.71		374.17	67.38
retinazer-0.2.0	1136.29	670.11		279.42	1262.34

Benchmarks: 262144 entites

ECS	baseline	plain	pooled	insert/remove	edit	transmute
artemis-odb-0.4.0	71.62	55.36		2.24
artemis-odb-0.9.0	160.87	132.16	132.82	12.82	32.40	54.80
artemis-odb-0.9.0_fast	209.12	159.65	153.97	13.03	31.91	55.51
artemis-odb-1.0.1	175.67	137.73	136.02	14.48	88.26	135.43
artemis-odb-1.0.1_fast	235.38	156.37	156.06	14.56	91.62	148.32
artemis-odb-2.0.0-RC1	175.42	137.92	137.77	58.08	123.32	153.13
artemis-odb-2.0.0-RC1_fast	235.92	145.90	145.43	62.89	124.02	159.13
artemis-odb-2.0.0-RC2	177.21	138.00	136.71	63.99	98.55	151.73
artemis-odb-2.0.0-RC2_fast	232.51	147.59	146.88	62.76	134.52	163.33
ashley-1.6.0	47.45	15.78		0.85
gdx-artemis-0.5.0	96.69		75.71	12.85
retinazer-0.2.0	311.30	159.43		26.61	282.48