forked from wiredtiger/wiredtiger
-
Notifications
You must be signed in to change notification settings - Fork 0
/
NEWS
2178 lines (1450 loc) · 80.8 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
WiredTiger release 2.5.2, 2015-03-23
------------------------------------
The WiredTiger 2.5.2 release contains important bug fixes.
API changes:
* Allow memory_page_max to be at most a quater of the cache size not half.
This avoids operations getting stalled due to the cache being filled with
one or two pages.
Bug fixes and other important changes:
* When skipping a dirty page during a checkpoint, make sure the tree is marked
dirty.
refs SUPPORT-1248, SERVER-17319, SERVER-17506, #1404, #1643, #1721, #1735
* Fix a bug in range truncate where we could remove the wrong records.
refs SERVER-17345
* Fix a bug in LSM management where we could let the cache get full - leading
to a operations being blocked.
refs #1720
* Fix several bugs in the checkpoint implementation that could lead to a tree
being marked clean when it had updates in memory. If shutdown occurred at
a specific time those updates would be discarded without being written.
refs SUPPORT-1248
* Fix some bugs in logging - where system crashes could leave empty files that
would stop recovery working on re-start.
refs #1717, #1719, SERVER-17451
* Fix a bug in recovery. Force recovery instead of returning an error if the
LSN given doesn't exist.
refs #1700, #1704
* Move writing into log worker thread to avoid latency in application threads.
refs #1683
* Fix a bug in the reconfigure API related to adhering to shared cache quotas.
refs #1712, #1713
* Fix a bug in WiredTiger statistics where we weren't recording overflow
record statistics.
refs #1520, #1703, #1711
* Several enhancements to eviction of large pages including:
- Don't do forced eviction of a page if it is the current walk point.
- Don't update the read generation on page in if it's set to oldest.
- Clear the walk positions before the eviction server sleeps.
- Reverse the direction of the LRU walk regularly.
- Add all pages that would block to the eviction queue.
- If evicting dirty pages use the worker threads not the server.
refs #1706
* Use raw mode when dumping indices.
refs #1709
* Fix a bug where we could race opening files while a WT_CONNECTION::close is
in progress.
refs SERVER-17319
* Fix a bug in LSM where snapshot transaction updates could have the wrong
visibility check applied. Leading to invalid updates.
refs #1641, #1701, #1702
* Fix a bug in checkpoint where it could get an EBUSY return unnecessarily.
refs #1404, #1589, #1705
* Fix a bug when writing a page from memory to disk (reconciling). We could
overwrite the end of a temporary buffer in some cases.
refs #1697, #1699
* Sometimes we would choose a sub-optimal layout for on disk pages when
writing them out from memory.
refs #1699
* Improve the performance of in-memory lookups by making the content of the
page structure more cache friendly.
WiredTiger release 2.5.1, 2015-03-07
------------------------------------
The WiredTiger 2.5.1 release contains new features, minor API changes and many
bug fixes.
New features and API changes:
* Add a new "log=(recover=on)" option to ::wiredtiger_open. The default value
is "on", if set to "error", recovery won't be run on startup. An error will be
returned if recovery is needed but disabled. This option is mainly to support
the WiredTiger command line utility.
refs #1651
* Add a new WT_CURSOR::equals method that returns when the cursors are equal,
intended as a fast-path for cursor comparison.
* Change how statistics work when there are checkpoints for an object. We
used to aggregate statistics for all open checkpoints and the current
handle. We now only return statistics for the object being queried. See
upgrading documentation for further information.
* Add a mode to LSM that allows us to limit the size of a tree by dropping
old chunks. Enabled via "lsm=(chunk_count_limit=0)", default to 0 which
disables the functionality. Note that enabling this feature discards old
data automatically.
refs #1652
* Update the WiredTiger printlog command line utility to generate JSON that
can be parsed by third party tools.
Refs #1438
* Change how we track memory allocation overhead. We used to apply a fixed
size for each allocation (which was difficult to track). The overhead
can be specified via a new configuration option to ::wiredtiger_open using
"cache_overhead=8". The value is a percentage and the default is 8.
refs #1564 #1565
* Major enhancements to the wtstats.py tool that translates WiredTiger
statistics into an HTML graph. The tool now generates interactive graphs
and no longer requires third party Python libraries to be installed.
* Add a new WT_CURSOR::reconfigure method for cursor configuration.
See API documentation for more information.
refs #1381
* Add a new WT_SESSION.strerror method, a thread-safe alternative to
::wiredtiger_strerror.
refs #1516
Bug fixes for bugs that could cause data inconsistency:
* Fix a bug in recovery where we could lose track of file identifiers and
apply updates to the wrong file.
Refs SERVER-17142 SERVER-17131
* Fix several bugs in data consistency that could cause corruption when
restarting after a hard crash, including:
- A bug in table create that could cause recovery to fail.
refs SERVER-17204
Other significant changes:
* Significant tuning enhancements for the WiredTiger cache, including:
- Avoiding stalls due to evicting large pages
- Better algorithms for getting application threads to help with cache
management without interfering with operation latencies.
- Better algorithms for maintaining the cache when there are a few very hot
pages (e.g: append workloads). SERVER-17344
- Freeing obsolete references more aggressively, which saves space and
reduces traversal overhead when there are lots of updates or deletes.
refs SERVER-17195, #1647
- Fix a bug that could cause a data loss if we split a large page into
multiple smaller pages and attempted to evict the page at the same time.
refs #1583 #1563 SERVER-16868
* Significant improvements to the cursor truncate implementation,
especially for workloads that periodically truncate from the start of
the file.
refs SERVER-17141
* Fix a bug in eviction where reconfiguring the number of eviction threads
could result in a segfault.
refs SERVER-17293
* Significant bug fixes and performance enhancements to the logging subsystem
including:
- Avoid yielding excessively to avoid CPU overhead when there are many
active sessions.
refs #1610
- The log close thread needs to wait for outstanding writes. #1571
- Create a new utility thread to close and fsync log files. #1560
- Ensure that log files are closed in sequence. #1555
- Ensure that log file create appears atomic. #1482
- Fix a race between connection close and switching log files that could
lead to a new log file being in an undefined state. #1480
* Added support for advanced options to the WiredTiger verify command line
tool.
* Fix a bug in our conflict detection algorithm, where we were failing to
detect some write-write conflicts in no-overwrite cursors.
refs SERVER-16351
* Significant bug fixes when writing pages to disk, including:
- Stop double count the on-disk header when choosing split points. Refs #1655
- Fill the first and second pages as much as possible when splitting.
refs #1282
- Improve the algorithm for fitting large items onto pages when splitting.
refs #1630 #1631
- Fix a bug when using raw compression where we could overflow allocated
memory.
refs SERVER-16664
* Fix several cases where WT_SESSION::verify and WT_SESSION::salvage could
return EBUSY unnecessarily.
Refs #1404 SERVER-16457
* Fix a bug where racing between discarding and updating a tree returned an
error to the application.
refs #1618 SERVER-17048
* Fix a bug where opening a statistics cursor in parallel with a checkpoint
could lead to a deadlock.
refs #1575 SERVER-16738
* Change the shared cache implementation to use cache read pressure rather
than write pressure to determine how best to share memory (as checkpoints
skew write pressure as a metric).
refs #1569
* Fix a bug where a deadlock could occur if a checkpoint starts in parallel to
compact operations.
refs #1589 SERVER-16967
* Fix a bug in how we parse Huffman-encoding configuration settings during
WT_SESSION::create.
refs #1417 #1536
* Fix a bug where the custom extractor terminate callback was being made twice.
refs #1503
* Add a new mode to the eviction server where it writes out some pages even
though the eviction triggers have not yet been reached.
* Fix several issues reported by COVERITY static analysis tool.
WiredTiger release 2.5.0, 2014-12-24
------------------------------------
The WiredTiger 2.5.0 release contains significant new features, API changes
and many bug fixes.
Now that WiredTiger is part of MongoDB, we are tracking issues related to
MongoDB usage of WiredTiger in the MongoDB JIRA system. Some entries in
the changelog now reference JIRA tickets that can be found at:
http://jira.mongodb.org
New features and API changes:
* Add support for storing large values on-page in a btree rather than in
an overflow item. This is useful for workloads that want to keep large
items in cache - since WiredTiger overflow items are never cached.
It is configured via new `leaf_value_max` configuration setting. This
enhancement led to the deprecation of internal_item_max and leaf_item_max
configuration settings, see upgrading documentation for further
information. [#1282]
* Add support for compressing log files. When configured each compressable
log record will be compressed. This is configure with the
`log=(compressor=X)` configuration setting. See upgrading documentation
for further information. [#1359]
* No longer return EBUSY when opening a bulk cursor, verifying or salvaging
a database if a checkpoint is currently running. This allows
applications to do these exclusive operations without shutting down the
checkpoint server thread. [#1397] [#1404] SERVER-16236 SERVER-16457
* Add support for immutable indexes. [#1344]
* Added several new statistics, improved accuracy for some statistics
tracking and simplified mechanism for querying a particular statistic.
[#1505]
* Have the eviction server write out unnecessary pages prior to the cache
reaching the configured eviction trigger size. This can reduce the
amount of eviction application threads do when configured with a large
cache.
* Several enhancements to managing how long we keep files open
* Revert a change in the 2.4.1 release that caused the WT_ROLLBACK (and
deprecated WT_DEADLOCK) error return to map to different numeric values.
Applications should ensure they are compiling against the same version
of the wiredtiger.h header file as the library they link against,
otherwise odd behavior will be experienced.
* Support setting configuration strings to "none" as being equivalent to an
empty string in most cases. [#1417]
* Enhance the hot backup implementation to allow recovery to be run between
incremental backups. [#1183]
Other significant changes:
* Improve performance of cursor open when there are many tables in a database.
[#1391] [#1443]
* Reduce the impact checkpoints have on concurrent operations. This was
done by changing how we lock tables. [#1391] [#1392]
* Improve performance when scanning a table that has many deleted items.
SERVER-16247
* Fix a bug in checkpoint, where the metadata (turtle) file wasn't being
synced on checkpoint. [#1383]
* Fix a bug where WiredTiger could accumulate memory during page splits and
never free it. SERVER-16546
* Many enhancements and bug fixes for Windows.
* Fix a bug where a custom extractor terminate was being called twice.
[#1503]
* Fix a bug where a race between closing a handle and checkpointing could
lead to errors. [#1495] [#1497]
* Validate the block header checksum before we clear it - if the checksum
field had been corrupted, we didn't notice. SERVER-16457
* Fix a bug in write conflict detection. Cursors configured with
no-overwrite could sometimes not see update conflicts for deleted
records. SERVER-16351
* Several bug fixes and performance improvements in LSM including:
- Add support for custom collators in LSM trees. [#1361]
- Fix a bug in LSM search_near, where it returned a deleted item.
BF-694, BF-700
- Improve background maintenance operations so that the cache does
not get full unnecessarily.
- Fix a bug that could lead to updates being written into old chunks.
[#1432] [#1418]
- Fix a bug in background merge that could skip updates. SERVER-16123
* Fix a bug when maintaining the cache, that could cause checkpoints to
skip writing an update that should have been included. [#1419] SERVER-16336
* Fix a bug in WT_SESSION::drop, where failures generated error output even
when force was specified. [#1436]
* Fix a bug in WiredTiger integer packing code when figuring out how much
space is required for a value (it can shrink as numbers grow). SERVER-16118
* Several enhancements to the wtstats.py tool that generates graphs from
standard WiredTiger statistics logs. [#1365]
* Fix a bug on OS X where fsync isn't sufficient to flush a file, use
fcntl(F_FULLFSYNC) instead.
* Work around a bug in clang 3.5.0 compare and swap primitive. The
__sync_bool_compare_and_swap version of the API in clang produces bad
code for us with -O3 optimization enabled.
WiredTiger release 2.4.1, 2014-11-06
------------------------------------
The WiredTiger 2.4.1 release contains several new features, many bug fixes
and performance enhancements.
New features and API changes:
* Add new custom extractor functionality to WiredTiger indexes. Allowing an
application to define mutated and/or multiple keys for indexes. [#1199]
* Add a new WT_SESSION::transaction_pinned_range method that allows users
to identify when a session is keeping a transaction ID pinned for a long
time. [#1314]
* Enhance statistics output so that keys are more clearly categorized. [#1313]
* Rename WT_DEADLOCK error return to WT_ROLLBACK. WiredTiger uses the return
in cases other than traditional application deadlock. The old value is
retained as an alias to maintain backward compatability. [#1204]
* Increase the maximum configurable cache size to 100GB.
Other significant changes:
* Improve support for building on Windows platforms. [#1342]
* Fix a bug where WiredTiger could race closing handles. [#1336]
* Enhance performance when hot pages in cache are growing rapidly. [#1317]
* Fix a bug in recovery, where log files that are zero extended could
result in some log records being skipped. [#1334]
* Updates to the Java API to improve documentation and exception handling.
[#1295]
* Improve support for building on Oracle Solaris platform [#1329]
* Fix a bug where closing a handle could leave the tree in an inconsistent
state on failure. [#1316]
* Several bug fixes and improvements to LSM including:
- Improving algorithm for switching the in-memory chunk.
- Fixing a bug related to dropping obsolete chunks. [#1304]
* Fix a bug in schema level operations (table create, drop, etc). If there
was an explicit transaction running when the operation was performed that
was subsequently rolled back the object could be left in an inconsistent
state.
* Several enhancements to cache management when there are long running
transactions present.
WiredTiger release 2.4.0, 2014-10-15
------------------------------------
The WiredTiger 2.4.0 release contains significant new features, API changes
and many bug fixes.
New features and API changes:
* Cursors keep their position across transaction boundaries. That is
WT_SESSION::begin_transaction and WT_SESSION::commit_transaction no longer
reset cursors. [#1181]
* Change cursor behavior so that when an operation returns WT_NOTFOUND, the
cursor is now left pointing to the original key/value pair. [#1209]
* Initial support for building WiredTiger on Windows.
* Add ability to customize a collator for specific data sources or with
application managed metadata. See upgrading documentation for more
information. [#1165]
* Enhance extension mechanism in WiredTiger to support loading extensions from
the application binary - not just a separate library. [#1174]
* Replace WT_SESSION::create "lsm=(merge_threads)" configuration option with
::wiredtiger_open "lsm_manager=(worker_thread_max)". See upgrading documentation
for more information.
* Enhancements to the WiredTiger Python API build process. [#1188]
* Add ability to dump and load WiredTiger databases in JSON format. [#1154]
* Add ability to automatically checkpoint based on the volume of log records
generated since the last checkpoint. This is enabled using the
::wiredtiger_open configuration option "checkpoint=(log_size=size)" [#1170]
* Enhance functionality allowing users to write content into the WiredTiger
transaction log. [#1171][#1175]
* Enhance the WiredTiger HyperLevelDB implementation to support log replay.
[#1106][#1155]
Other significant changes:
* Fix several bugs in the shared cache implementation. [#1180][#1176]
* Fix a bug where the public URI field in a cursor did not match the string
passed to WT_SESSION::open_cursor. [#1235]
* Fix several bugs in salvage. [#1222][#1169]
* Several bug fixes and enhancements for WT_CONNECTION::reconfigure.
[#1214][#1172]
* Fix several bugs in raw compression implementation, particularly for data
that compresses extremely well. [#1191]
* Several bug fixes and enhancements to WiredTiger LevelDB interface.
* Switch default build from using adaptive pthread mutexes to default pthread
mutexes.
WiredTiger release 2.3.1, 2014-08-14
------------------------------------
The WiredTiger 2.3.1 release contains mainly performance enhancements and bug
fixes.
Changes to the WiredTiger API:
* Fix a bug in WT_CURSOR::set_value that could lead to undefined behavior with
some value formats.
* Make the asynchronous API generally available [#1139]
* Add log cursors for replay and verification. Make generated log record and
operation types public. [#1106]
* Allow eviction worker threads to be started and stopped dynamically.
Applications that use the `eviction_workers` configuration should see the
upgrading documentation on how to use this feature.
[#1116, #1143, #1158]
Other significant changes:
* Improve performance and reduce latency during checkpoints and LSM merges.
Remove uses of the checkpoint lock other than serializing checkpoints:
compact holds the schema lock, so it doesn't need to hold the checkpoint
lock, the new WT_BTREE handle close lock prevents checkpoints from colliding
with handle close, so LSM doesn't need the checkpoint lock either.
* Some minor cleanups, setting the internal session's name in a few places.
[#1073]
* Grab the live lock when loading a checkpoint in diagnostic mode: that could
race with a read. [#1102]
* Instead of keeping a list of file URIs for checkpoint to flush, open a handle
and stash it. [#1114]
* Add a new OS-layer function __wt_fsync_async to flush a file without waiting
for the results, call it from the Btree flush-leaves code so pages start
flushing while we're working the rest of the checkpoint. [#1136, #1152]
* Wait for the handle flush lock when writing the leaf pages instead of
returning EBUSY. [#1136]
* Add a wtperf page to the documentation, describe how to simulate workloads
and view statistics. [#1147]
* Flag new structures not listed in PREDEFINE. [#1148]
* Return EBUSY if no async handles available and fix ex_async to look for it.
[#1153]
* Fix some problems with navigation in the reference guide.
* Bump the number of slots for internal sessions: we have a lot more than 2
now. Add a test for `session_max` settings, make sure we add enough to
account for at least the default internal sessions.
* Remove tcbench: we're no longer maintaining it.
WiredTiger release 2.3.0, 2014-07-29
------------------------------------
The WiredTiger 2.3.0 release contains significant new features, performance
enhancements and bug fixes. Significant changes are described below.
Changes to the WiredTiger API (see upgrading documentation for details):
* Add a LevelDB API implementation for WiredTiger. This includes support for
stock LevelDB as well as Basho, HyperLevelDB and RocksDB versions of the API.
To build the LevelDB API include --enable-leveldb in the configure command,
to specify compatability with an alternative LevelDB API use
--enable-leveldb=[basho,hyper,rocksdb]. [#1028]
* Add ability to build some common extensions into the WiredTiger library.
This means that the libraries for those extensions don't need to be
dynamically loaded at runtime. Currently supported extensions are Snappy
compression and zlib compression. The option can be enabled by passing
--with-builtins=[snappy,zlib] to the configure command line.
* Add a new configuration to wiredtiger_open: statistics_log=(on_close=true),
that causes a set of statistics to be logged on WT_CONNECTION::close. [#1086]
* Add a new configuration to wiredtiger_open: exclusive, that causes the open
to fail if the database already exists.
Other significant changes:
* Performance improvement for high throughput workloads using multiple
eviction threads. Performance of some workloads improves by over 15% [#1087]
* Significant performance optimizations for queries, giving up to 20%
throughput improvement for in-memory query workloads.
https://github.com/wiredtiger/wiredtiger/wiki/Query-throughput
* Fix an off-by-one bug that could lead to ENOMEM during commit with logging.
[#1104][#1121]
* Allow bulk loads to multiple files to complete in parallel. [#1114][#1126]
WiredTiger release 2.2.1, 2014-06-24
------------------------------------
The WiredTiger 2.2.1 release contains mainly performance enhancements and bug
fixes. Significant changes are described below.
Changes to the WiredTiger API (see upgrading documentation for details):
* Change the order in which configuration setting mechanisms are applied by
wiredtiger_open. [#1010][#1034]
* Split the global transaction_sync configuration into two parts: a sync method
(dsync, fsync or none), and an enabled flag (false by default). [#1074]
* Add ability to sync with per transaction granularity. [#1074]
* Update WiredTiger Java API to throw WiredTigerException consistently. [#1011]
* Add ability to dump and load databases using JSON format. [#740][#1049]
Other significant changes:
* Various performance improvements to the main cursor search routine including
reductions in how often we need to copy data and profiling based optimizations
for tight search loops. [#1050][#1070]
* Fix a bug in recovery with missing files (e.g., after a hotbackup that raced
with file creation). [#1042][#1045]
* Several bug fixes and performance enhancements related to LSM trees and
snapshot isolation transactions. [#1057][#1060][#1075]
* Several performance tuning enhancements to LSM trees around locking,
throttling and switching chunks. [#1051]
* Algorithmic improvements to LSM tree compact operation. It is now faster
and more reliable. [#1063]
* Create a separate thread to manage open file handles - which means that:
- Application threads are less likely to be responsible for closing handles
- Multi threaded workloads don't open/close handles more often than necessary
[#1018]
WiredTiger release 2.2.0, 2014-05-21
------------------------------------
The WiredTiger 2.2.0 release contains new features, performance enhancements
and bug fixes. Significant changes include:
Changes relevant for upgrading applications:
Update the table create API to disable prefix compression by default.
Applications generally see better performance without prefix compression,
choosing space saving over performance is up to the application. [#981]
Change the default leaf_page_max setting from 1MB to 32KB. Choosing a large
default leaf_page_max led to poor performance in out of cache workloads.
Remove the `--enable-debug` option to configure. It is more standard to set
`CFLAGS="-g"` variable instead.
Save the wiredtiger_open configuration when a database is created, so that
settings like cache size, extensions and logging are set consistently by all
subsequent users of the database.
Add an `--enable-verbose` option to configure. In order to access the verbose
message functionality available as part of the wiredtiger_open and
WT_CONNECTION::reconfigure APIs, it is necessary to pass the `--enable-verbose`
option to configure.
Enhance the metadata cursor implementation (i.e: cursors created with a
"metadata:" prefix) so that they can be used to inspect metadata for internal
tables and now support altering the metadata. Add a new "read_only" flag to
cursor configuration that defaults to false for metadata cursors.
Fix several bugs in raw compression, including one that could cause data
corruption and some that triggered poor performance.
[#984][#991][#1007][#1008][#1013]
Improve the performance of recovery - we no longer need to scan all log files
looking for the last checkpoint.
Improve performance of read-only transactions, by deferring the allocation
of transaction IDs. [#978]
Several bug fixes in hot backup related to log
files, including:
* Always choose the right metadata version in the backup [#972]
* Don't require that hot backup copies log files in order [#976]
* Always copy log files before data files [#976]
* Fix a bug where recovery returned an error if the last log record was
incomplete [#994]
Speed up checkpoints by doing a better job of skipping pages that can't
contain changes that need to be included. [#954][#963][#1001]
Add ability to store zero length data items into LSM trees. [#540]
Add an asynchronous data access/manipulation API to WiredTiger. [#933]
Add the ability to configure multiple eviction server threads, to help with
keeping space available in the cache. [#918]
Add the ability to reconfigure the checkpoint and statistics log servers.
[#997][#1004]
Improve the performance of retrieving data for in cache workloads. [#970]
Improve the structure of the in-memory tree we are generating, by allowing
internal pages to be split. This significantly improves query performance
in some workloads. [#876]
Work around a bug in posix_fallocate on Linux, where it could corrupt already
written data.
Add the ability to leak memory on connection close via new leak_memory option
to WT_CONNECTION::close API. This allows for faster shutdown if a process is
going to exit when the WiredTiger connection is closing.
Allow salvage to run on any table type.
WiredTiger release 2.1.2, 2014-03-27
------------------------------------
The WiredTiger 2.1.2 release contains performance enhancements and bug fixes.
Significant changes include:
Update the configuration settings for shared_cache to make the distinction
between cache_size and shared_cache less confusing. See upgrading
documentation for more information.
Various performance enhancements to improve the performance of checkpoints.
Fix a bug that could cause a hang with small caches under heavy load. [#894]
WiredTiger release 2.1.1, 2014-03-04
------------------------------------
The WiredTiger 2.1.1 release contains new features, performance enhancements
and bug fixes. Significant changes include:
Fix a bug where a page could be marked clean when it contained uncommitted
changes. This bug could cause undefined behavior in transaction rollback
under load.
Fix a bug with shared caches when rebalancing between connections.
Add a new public API to WiredTiger that provides the ability to parse
WiredTiger compatible configuration strings. See the upgrading documentation
for further information. [#873]
A number of performance enhancements to the LSM implementation, particularly
for long running workloads.
A number of performance enhancements and bug fixes to cache eviction code.
Add an option to use direct I/O when reading from checkpoints. To enabled
the functionality add "direct_io=[checkpoint]" to your wiredtiger_open
configuration string. [#847]
WiredTiger release 2.1.0, 2014-02-04
------------------------------------
The WiredTiger 2.1.0 release contains new features, performance enhancements
and bug fixes. Significant changes include:
The WT_ITEM structure was changed so that the size field is a size_t rather
than a uint32_t. See upgrading documentation for details.
A change to the compress_raw interface around repeating the call with more
records. See upgrading documentation for details.
In LSM trees, the memory_page_max setting is ignored. The effective setting
is double the chunk size. [#861][#859]
Add support for zlib compression. [#855] [#865]
Various enhancements to how WiredTiger generates tree structures in memory to
help maintain consistent performance as table size grows. [#851]
Add support for Levyx Inc Helium as an external data source in WiredTiger
[#849][#850]
Improve insert performance when a table contains many identical overflow
items.
Various performance enhancements to btree searches. [#838][#839][#840]
Add support for newer versions of autoconf up to 1.14. [#599][#841]
Improve multi-threaded throughput of durable log writes, including changing
the default wiredtiger_open transaction_sync configuration from dsync to
fsync, see the upgrading documentation for further information. [#831][#832]
In the Python and Java APIs, automatically close handles to prevent invalid
accesses by applications. [#649][#800][#830]
Various enhancements to the LSM merge algorithm, including improvements to how
files are selected for merging, and throttling based on whether merges are
keeping up (to limit write amplification). Made the minimum number of chunks
chosen to merge configurable. [#817][#819][#822]
WiredTiger release 2.0.1, 2013-12-12
------------------------------------
The WiredTiger 2.0.1 release contains major new features, numerous performance
enhancements and bug fixes.
Significant changes include:
* WiredTiger now supports fine-grained durability via Write Ahead Logging (WAL).
Logging is enabled with the "log=(enabled)" configuration string to
wiredtiger_open. If the connection is not shut down cleanly and logging is
enabled, WiredTiger will automatically run recovery the next time it is
opened, rolling forward changes in the log until the last commit.
[#605]
* Many enhancements to the LSM implementation to improve the throughput and
reduce maximum operation latency including:
- Algorithmic improvements when multiple merge threads are configured.
- Improvements to bloom filter lookup speed.
- Enhancements to internal cursor management, to reduce search overhead.
- Prioritize switching to a new level 0 chunk in utility threads, to avoid
application thread pauses.
- More advanced logic in choosing when to create bloom filters.
* LSM specific WT_SESSION::create configuration option enhancements. Including:
- Move existing options into their own group, and strip leading lsm_ prefix.
- Add a new merge_max configuration option.
- Update the default chunk_size to be 10MB.
- Increase the default bloom filter bit and hash counts.
- Clean up files left after interrupted merges.
See the upgrading documentation for details.
[#784, #785, #786, #802]
* WT_SESSION::compact can now be used to merge LSM trees into a small number
of chunks on disk.
[#792]
* Enhanced the Java API, so that when WiredTiger automatically closes a
handle, the handle is automatically invalidated for the Java application.
[#485]
* Add a script that can create an interactive web page to view statistics
from a WiredTiger statistics log. Based on D3: http://d3js.org/
* Enhancements to the wtperf performance testing tool to add new features
WiredTiger release 1.6.6, 2013-11-19
------------------------------------
The WiredTiger 1.6.6 release is a bugfix and performance tuning release.
This release of WiredTiger contains a database format change. Database files
from previous releases will need to be upgraded.
A special note: the WiredTiger code base is now being regularly reviewed
using the Coverity Static Analysis Verification Engine. We'd like to
thank Coverity for their on-going support of Open Source projects like
WiredTiger!
Significant changes include:
* Performance changes include: limiting operations done inside update
serialization primitives, removing unnecessary memory barriers, replacing
spinlocks with atomic instructions, padding structures to avoid false
cache sharing, switching from per-file mutexes to per-page mutexes,
pre-allocating structures to avoid memory allocation while holding
mutexes, and using adaptive mutexes where available.
[#707, #718, #719]
* A number of LSM stability and performance improvements: changes include
better merge algorithms, reduced locking, and higher concurrency.
* A number of table compaction performance improvements, including changes
allowing compaction to no longer read unnecessary file blocks into the
cache, requires fewer passes over the file and support concurrent
checkpoints and eviction. This change required an underlying file
format change, see the upgrading documentation for details.
[#756, #761]
* WiredTiger statistics have been significantly improved:
Statistics logging has been changed to aggregate information from all
open handles. [#709, #717]
For performance reasons, statistics are now disabled by default, see
the upgrading documentation for details. [#715]
Statistics configuration has been changed so the connection and cursor
configuration are consistent, with matching changes to the "wt stat"
command-line utility; see the upgrading documentation for details.
* Update WT_EVENT_HANDLER interface to contain a new "handle close"
interface and to pass a WT_SESSION handle into all callbacks, see the
upgrading documentation for details. [#649]
Add timestamp, process ID and thread ID to messages generated via
WT_EVENT_HANDLER interface. [#753]
* WiredTiger eviction improvements, supporting larger data-to-cache size
ratios. [#754]
* Various fixes for handling overflow records. [#726, #743]
* Overflow records are no longer tracked during bulk-loads, significantly
increasing bulk-load performance for some data sets.
WiredTiger release 1.6.5, 2013-10-09
------------------------------------
This is primarily a bugfix and performance tuning release. The main changes are:
* Change the default statistics_fast configuration from false to true.
* Change WT_CURSOR::insert to not hold a position. [#673]
* Disallow WT_SESSION::compact operations on LSM trees.
* The 'sync' setting to wiredtiger_open has been renamed 'checkpoint_sync'.
* Add a "metadata:" cursor type. [#660]
* Fix race in the cache's dirty byte tracking. [#635, #699]
* Fix a bug scanning through a memory-mapped file with overflow items. [#701]
* Use hardware checksum instructions when available. [#582, #702]
* Several bug fixes related to tracking active transaction IDs and detection of
obsolete updates with high concurrency workloads. [#639, #643, #657, #683]
* Fix several bugs in LSM including races on shutdown and Bloom filter
creation. [#686, #687, #688].
* Fix a bug in LSM where we were not including Bloom filter files in backups.
[#684]
* Optimize the LSM throttle and merge algorithms. [#676]
* Make hot backups work concurrently with files being bulk-loaded. [#570, #653]
* Add full support for snapshot isolation to LSM: only switch LSM chunks if all
changes are globally visible and detect conflicts between transactions across
file switches. [#629]
WiredTiger release 1.6.4, 2013-08-20
------------------------------------
This is primarily a bugfix and performance tuning release. The main changes are:
* Make prefix compression of keys conditional on the amount of space saved.
A database format change was required for this enhancement. See upgrading
documentation for details. [#624]
* The default behavior of the wt utility's load command has been changed to
overwrite existing data.
* Add a WT_SESSION.create prefix_compression_min configuration option with a
default value of 4. [#624] and [#624]
* Fix "make install" of Python API. [#598]
* Require platform support for atomic read/write of 64 bit values. [#553]
* Support transaction semantics for custom data source implementations. Enhance
Memrata data source to support transactions.
* Changes to the wtperf testing tool related to how configuration options are
specified.
* Enhance cursor key/value memory management to be more efficient, consistent,
and have stricter checking of inputs and outputs.
* Increase the likelihood of being able to evict hot pages. [#604]
* Reference on-page keys instead of copying them to allocated memory. This
saves space in the cache and overhead when reading pages into cache.
[#592] and [#600]
* Add a btree search optimization that skips matching prefixes. [#595]
* Turn off Huffman encoding for keys on row-store internal pages. [#592]
* Add concurrent logging infrastructure that will be used to support write
ahead logging in a future release.
WiredTiger release 1.6.3, 2013-07-12
------------------------------------
This is a bugfix and performance tuning release. The main changes are:
* Change the default cursor overwrite configuration so that it is consistent
across all data sources. This change may alter the behavior of existing
applications without triggering any compilation or runtime warnings. See
the upgrade documentation for details. [#512]
* Require platform support for 64 bit atomic operations. [#553]
WiredTiger release 1.6.2, 2013-06-18
------------------------------------
This is a bugfix and performance tuning release. The main changes are:
* Fix a race in the WiredTiger pseudo random number generator that was leading
to poor distribution of numbers.
* Change the default compression configuration to "uncompressed".
* Fix a race between checkpoints and LSM that could result in a crash. [#543]
* Add an option to output version information at runtime. Configure by
including "verbose=[version]" in the wiredtiger_open connection
configuration string. [#564]
* Add a configurable prefix to error messages. [#527]
* Add two new extension APIs, one to return a transaction ID, one to return
if a transaction ID is visible to the current transaction.
* Add standard metadata functions to the extension API and make extension data
sources responsible for their own metadata entries.