Integration of Camelot in py-pdf

[MartinThoma]

Camelot is dead, but the camelot community is not. people are using it and bosd wants to maintain it.

I see two ways how py-pdf (the organization) could help:

1. Simply have a fork within py-pdf: I'd opt for a a clear re-name, hence not camelot2.
2. Same as (1), but also adding a hook within pypdf so that (1) we can re-use code and (2) the end-user has something like reader.pages[0].extract_table()

What do you think about it?

[MartinThoma]

Especially @pubpub-zz @stefan6419846 for the pypdf (the software) part and @Lucas-C for the extension of the organization.

[MartinThoma]

pytest and flake8 both allow 3rd party packages to extend their functionality by making use of entry-points (I'm currently preparing an article; I'll share it once it's ready :-) )

The idea was to do something like this:

from importlib.metadata import entry_points


class PdfReader:

    @classmethod
    def load_entrypoints(cls):
        for entry_point in entry_points().select(group="pypdf.reader"):
            loaded_python_object = entry_point.load()
            setattr(cls, entry_point, loaded_python_object)

I'm uncertain if that works though. And I'm also uncertain if it's a good idea. I might just have been excited about another way to let people extend pypdf (without having to maintain the code 😅 )

[MartinThoma]

The idea was to open a clear path to extending pypdf, especially to new stuff that might come up with LLMs.

[MartinThoma]

Giving it another thought: I think forking Camelot into the py-pdf organization and renaming it to pypdf-table-extraction would make sense. The interface-integration into pypdf.PdfReader can be done optionally.

The current interface is:

>>> import camelot
>>> tables = camelot.read_pdf('foo.pdf')
>>> tables
<TableList n=1>
>>> tables.export('foo.csv', f='csv', compress=True) # json, excel, html, markdown, sqlite
>>> tables[0]
<Table shape=(7, 7)>
>>> tables[0].parsing_report
{
    'accuracy': 99.02,
    'whitespace': 12.24,
    'order': 1,
    'page': 1
}
>>> tables[0].to_csv('foo.csv') # to_json, to_excel, to_html, to_markdown, to_sqlite
>>> tables[0].df # get a pandas DataFrame!

In the first release, users could just switch to import pypdf_table_extraction as camelot in their code.

After merging all of the improvements that users have done in other forks, the next major release could be an interface-alignment to pypdf:

from pypdf import PdfReader


reader = PdfReader("foo.pdf")
tables = reader.extract_tables()   # <--- this could be enabled by a plugin-mechanism in pypdf or by an optional dependency on pypdf_table_extraction

# rest can stay the same

[stefan6419846]

To be honest, I disagree with the fact that pypdf should start providing entry points - I do not see any real value for it at the moment and people might consider implementing nice additions as external plugins which are better suited for the core library. Alternatively, post-processing within own code is always possible - if this requires slightly changing the pypdf interface, this should not be much of a burden.

flake8, pytest, flitz etc. are user-facing tools which usually are not used on a library level and where plugin-based extensions make more sense. Contrary to libraries, they might not come with an official public interface to allow for reliable library usage. (pytest hides most of its module in a private submodule for example. From maintaining both a flake8 and pytest plugin relying on internal stuff as well due to missing public APIs and writing selftests accordingly, I know the hassle of more or less severe breakages due to internal changes.)

[MartinThoma]

Thanks for the feedback. Yes, I think I was just excited because I haven't understood the entrypoint-system before :-)

[stefan6419846]

I have never used camelot before, thus there is not much I can say about it, but I generally do not see any issue with continuing maintenance under the py-pdf organization as long as there are enough contributors and mainainers.

For the pypdf part: I do not know about how much value this would add to pypdf as usage seems to be rather simple anyway, but as long as it stays an optional feature without enforcing pypdf users to always install the somehow heavier dependency chain of camelot, I do not see any real issues.

[stefan6419846]

By the way: PyMuPDF (with its restrictive AGPL license) added table extraction support about 6 months ago: https://github.com/pymupdf/PyMuPDF/blob/main/src/table.py They apparently decided to base their initial implementation on pdfplumber.

[MartinThoma]

Huh, interesting. I didn't expect that. PyMuPDF is really on fire in the last 1-2 years! (In the positive sense - they add features pretty quickly)

[stefan6419846]

The only thing which makes it hard to use PyMuPDF is the strong copyleft effect (unless using the paid variant), where pypdf etc. are much nicer ;)

[bosd]

but as long as it stays an optional feature without enforcing pypdf users to always install the somehow heavier dependency chain of camelot, I do not see any real issues.

On the other hand, it would be great to have it ootb.
According to the docs the dependency chain is not too big..
Only Ghostscript and tkinter.
https://camelot-py.readthedocs.io/en/master/user/install-deps.html

But that does'nt seem accurate.. Why Tkinter, as camelot does'nt have a GUI.

[stefan6419846]

As already linked before, the correct dependencies are listed at https://pypistats.org/packages/camelot-py - especially numpy and pandas can make this a bit heavier. Ghostscript apparently is optional for Camelot (used for OCR-based "Lattice approach": https://camelot-py.readthedocs.io/en/master/user/how-it-works.html), Tkinter (part of the stdlib) might be required for the Camelot CLI to display graphical output using matplotlib.

[MartinThoma]

If we fork camelot into the py-pdf organization, which name should the fork have?

[MartinThoma]

I suggest pypdf_table_extraction

[bosd]

Makes sense 👍

[bosd]

Well, I'm onboard. Hopefully some more experienced dev's are interested in becoming maintainer as well.
A rename makes a lot of sense. Specifically when it makes more clear what the tool does. 😄