forked from openzim/libzim
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
338 lines (268 loc) · 10.8 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
libzim 6.3.0
============
* Rewrite internal reader structure to use stream decompression.
This allow libzim to not decompresse the whole cluster to get an article
content. This is big performance improvement, it speedups random access by
2, with a very small cost when doing "full" incremental reading
(zim-check/zim-dump). (@veloman-yunkan)
* Better dirent lookup.
Dirent lookup is the process of locating article data starting from the url
or title. This improves reading of zim file up to 10% (@veloman-yunkan)
* Add basic, first version of `validate` function to check internal structure
of a zim file. (@veloman-yunkan, @MiguelRocha)
* Fix compilation of libzim without xapian (@mgautierfr)
* Remove zlib dependency (and support of very old files created using zlib
compression) (@mgautierfr)
* New unit tests and various small fixes.
libzim 6.2.2
============
* Check blob index before access it in the cluster.
* Refactoring of the cluster reading.
libzim 6.2.1 (release process broken)
=====================================
* Update readme and add link to repology.org packages list.
* Fix compilation on windows.
libzim 6.2.0
============
* Fix compilation of libzim on freebsd.
* Rewrite unit tests to remove python based test and use gtest all the time.
* Make libzstd mandatory.
* Support for meson 0.45.
* Fix multipart support on macos.
* Add a documentation system.
* Better cache system implementation (huge speed up).
* Various (and numerous) small refactoring.
libzim 6.1.8
============
* Increase default timeout for test to 120 seconds/test
* Compression algorithm to use can be passed to `zim::writer::Creator`
* Add automatic debian packaging of libzim.
* Fix using of tmpdir (and now use env var TMPDIR) during tests.
libzim 6.1.7
============
* Do not assume urlPtrPos is just after the mimetype list.
* Fix compilation of compression test.
* Do not exit but throw an exception if an ASSERT is not fulfill.
libzim 6.1.6
============
* Better (faster) implementation of the ordering of article by cluster.
* Fix compression algorithm.
libzim 6.1.5
============
* [Writer] Remove unused declaration of classes.
Those classes were not implemented nor used at all.
libzim 6.1.4
============
* [Writer] Fix excessive memory usage. Data of the cluster were clean at the
end of the process, not once we don't need it.
libzim 6.1.3
============
* [Writer] Use a `.tmp` suffix and rename to `.zim` at the end of the write
proces.
* Add unit tests
* Do not include uncessary `windows.h` headers in public zim's headers.
libzim 6.1.2
============
* [CI] Fix codecov configuration
* [Writer] Fix threads synchronization at end of writing process.
libzim 6.1.1
============
* Fix bug around the find function
libzim 6.1.0
============
* Compile now on OpenBSD
* [Test] Use the main function provided by gtest.
* [CI] Move the CI compilation to github actions.
* Add stopwords for 54 new languages.
* [Writer] Improve the way we are writing cluster at zim creation time.
- Clusters are directly written in the zim file instead of using temporary
files.
- mimetypes are limited to 944 bytes.
* Add a new type of iterator to iterate over articles in a performant way
reducing decompression of clusters. This is now the new default iterator.
* Add support for zim files compressed with zstd compression algorithm.
This is not possible to use zstd to create zim file for now.
libzim 6.0.2
============
* Fix search suggestion parsing.
libzim 6.0.1
============
* Fix crash when trying to open an empty file.
* Ensure that pytest tests are run on the CI.
libzim 6.0.0
============
* [Writer] Index the articles in differents threads. This is a huge speed
improvement as the main thread in not blocked by indexing.
* Index the title only if `shouldIndex` return true.
libzim 5.1.0
============
* Improve indexation of the title.
* Better pertinence of suggestions (only for new zim files)
* Improvement of the speed of Leveinstein distance for suggestions (for old
zims)
libzim 5.0.2
============
* Improve README.
* Remove gtest as embeded subproject.
* Better lzma compression.
* Better performance of the leveinstein algorithm (better suggestions
performance)
libzim 5.0.1
============
* Update README.
* [Writer] Add debug information (print progress of the clusters writing).
* [Writer] Correctly print the url to the user.
* [CI] Add code coverage.
libzim 5.0.0
============
* Fix thread slipping for win32 crosscompilation.
* Fix a potential invalid access when reading dirent.
* Fix memory leak in the decompression algorithm.
* [Writer] Fix a memory leak (cluster cleanning)
* [Writer] Write article data in a temporary cluster file instead of a
temporary file per article.
* [Writer] Better algorithm to store the dirent while creating the zim
file. Better memory usage.
* [Writer] [API Change] Url/Ns are now handle using the same struct Url.
* [Writer] [API Change] No more aid and redirectAid. A redirectArticle
have to implement redirectUrl.
* [Writer] Use a memory pool to avoid multiple small memory allocations.
* [Writer] [API Change] Rename `ZimCreator` to `Creator`.
* [API Change] File's `search` and `suggestions` now return a unique_ptr
instead of a raw pointer.
libzim 4.0.7
============
* Build libzim without rpath.
libzim 4.0.6
============
* Support zim file created with cluster not written sequentially.
* Remove a meson warning.
libzim 4.0.5
============
* Store the xapian database in the right url.
* Do not fail when reading very small zim file (<256b).
* Do not print message on normal behavior.
* [BUILDSYSTEM] Be able to build a dynamic lib (libzim.so) but using static
dependencies.
* [CI] Use last version of meson.
* [CI] Use the new deps archive xz
libzim 4.0.4
============
* Fix opening of multi-part zim.
* Fix convertion of path to wpath on Windows.
libzim 4.0.3
============
* Implement low level file manipilation using different backends
libzim 4.0.2
============
* [Windows] Fix opening of zim file bigger than 4GiB
libzim 4.0.1
============
* [Writer] Fix wrong redirectyon log message
* Make libzim compile natively on windows using MSVC
* Better message when failing to read a zim file.
* Make libzim on windows correctly open unicode path.
* Add compilation option to use less memory (but more I/O).
Usefull on low memory devices (android)
* Small fixes
libzim 4.0.0
============
* [Writer] Remove a lot of memory copy.
* [Writer] Add xapian indexing directly in libzim.
* [Writer] Better API.
* [Writer] Use multi-threading to write clusters.
* [Writer] Ensure mimetype of articles article is not null.
* Extend test timeout for cluster's test.
* Less memory copy for cluster's test.
* Allow skipping test using a lot memory using env variable
`SKIP_BIG_MEMORY_TEST=1`
* Explicitly use the icu namespace to allow using of packaged icu lib.
* Use a temporary file name as long as the ZIM writting process is
not finished (#163)
* [Travis] Do no compile using gcc-5 (but the default trusty's one 4.8)
libzim 3.3.0
============
* Fix handling of big cluster (>4GiB) on 32 bits architecture. This is mainly
done by :
* Do not mmap the whole cluster by default.
* MMap only the memory asociated to an article.
* If an article is > 4GiB, the blob associated to it is invalid
(data==size==0).
* Other information are still valid (directAccessInformation, ...)
* Fix writing of extended cluster in writer.
* Compile libzim on macos.
* Build libzim setting RPATH.
* Search result urls are now what is stored in the zim file. They should not
start with a `/`. This is a revert of the change made in last release.
(See kiwix/kiwix-lib#123)
* Spelling corrections in README.
libzim 3.2.0
============
* Support geo query if the xapian database has indexed localisation.
* Handle articles bigger than 4Go in the zim file (#110).
* Use AND operator between search term.
* Fix compilation with recent clang (#95).
* Add method to get article's data localisation in the zim file.
* Be able to get only a part of article (#77).
* Do not crash if we cannot open the xapian Database for some reasons.
(kiwix/kiwix-tools#153)
* Do not assumen there is always a checksum in the zim file.
(kiwix/kiwix-tools#150)
* Try to do some sanity checks when opening a zim file.
* Use pytest to do some tests (when cython is available).
* Use levenshtein distance to sort and have better suggestion results.
* Search result urls are now always absolute (starts with a '/').
(kiwix/kiwix-lib#110)
* Open the file readonly when checking the zim file (and so be able to check
read only file).
* Accept absolute url starting with '/' when searching for article.
* Fix various bugs
libzim 3.1.0
============
* Lzma is not a optional dependency anymore.
* Better handle (report and not crash) invalid zim file.
* Embed source of gtest (used only if gtest is not available on the system)
* Move zimDump tools out of libzim repository to zim-tools
* ZimCreator tools doesn't not read command line to set options.
libzim 3.0.0
============
This is a major change of the libzim.
Expect a lot new improvement and API changes.
* Add a suggestion mode to the search
* Fix licensing issues
* Fix wrong stemming of the query when searching
* Deactivate searching (and so crash) in the embedded database if the zim is
splitted
* Rewrite the low level memory management of libzim when reading a zim file:
* We use a buffer base entity to handle memory and reading file instead of
reading file using stream.
* MMap the memory when posible to avoid memory copy.
* Use const when posible (API break)
* Move to googletest instead of cxxtools for unit-tests.
* Fix endiannes bug on arm.
* Do not install private headers. Those headers declare private structure and
should not be visible (API break)
* Compile libzim with `-Werror` and `-Wall` options.
* Make libzim thread safe for reading article.
The search part is not thread safe, and all search operation must be
protected by a lock.
* Add method to get only a part of a article.
* Move some tools to zim-tools repository.
libzim 2.0.0
============
* Move to meson build system
`libzim` now use `meson` as build system instead of `autotools`
* Move to C++11 standard.
* Fulltext search in zim file.
We have integrated the xapian fulltext search in libzim.
So now, libzim provide an API to search in a zim containing embeded fulltext
index. This means that :
*libzim need xapian as (optional) dependencies (if you want compile with
xapian support).
* The old and unused search API has been removed.
* Remove bzip2 support.
* Remove Symbian support.
* Few API hanges
* Make some header files private (not installed);
* A `Blob` can now be cast to a `string` directly;
* Change a lot of `File` methods to const methods.