Add a METS with lots of files for testing

[kba]

No description provided.

[EEngl52]

@kba I guess this can be closed?

[kba]

I don't remember what I meant by this. I'll try to open more descriptive isssues in the future 😬

[bertsky]

I think this was to have a realistic test case for performance issues with large METS. Large could be many fileGrps or many files therein or many pages – or any combination of it. This came up earlier when some change to the PAGE model (esp. the pageId lookup) severely degraded performance on my workspaces to the point were it became unusable.

[kba]

OK, so a stress test of sorts, that should be doable.

[EEngl52]

probably sth like this? http://digital.slub-dresden.de/id336927223

[bertsky]

probably sth like this? http://digital.slub-dresden.de/id336927223

well, 300 pages is not that much of a stretch. How about: http://digital.slub-dresden.de/id507244877-18920000

That would cover the many pages scenario. But how about many fileGrps? The METS from Kitodo.Presentation is rather small (just FULLTEXT, ORIGINAL and various JPEG qualities). All I can think of is an OCR-D workspace after running lots of different workflows with many steps.

[bertsky]

That would cover the many pages scenario

Or rather: I could give you the METS built from https://github.com/bertsky/ocrd_publaynet – it contains 671407 pages in the training set and 56227 in the validation set.

[EEngl52]

my example above is 1400 pages, nothing compared to your publaynet though

[bertsky]

my example above is 1400 pages, nothing compared to your publaynet though

oh, right! Sorry, got confused. Yes, I do think the bible should be a test case. PubLayNet is an extreme (probably never used that way) – I actually recommend against having it included in the auto regression tests, as it's such a drag. (But it might help to have it somewhere ...)