Skip to content

NoraInspection

Jump to bottom
OlaBauge edited this page Sep 21, 2009 · 21 revisions

Overview

In order to select the most effective method of text extract and parameterize text correction, it is relevant to distinguish the various ways in which the PDF documents in the NORA collection were produced, e.g. use of LaTeX, vs. M$ Word, vs. other word processing tools. In the LaTeX world, for example, it may matter which specific approach was used to output PDF, e.g. latex plus dvips plus ps2pdf, vs. pdflatex, vs. integrated tools like MiKTeX. Likewise, when using Word, results may vary according to which specific software version was used, or depending on whether Adobe Distiller or another tool for PDF creation was applied. When it comes to font choices and character encodings, it might also turn out that more basic properties of the original environment used for PDF creation are relevant, e.g. the choice of operating system (Linux vs. Windows, say) and default locale settings.

Presumably many dozens or hundreds of distinct software environments were at play in the production of the NORA PDF files, and hopefully most of this variation will be irrelevant for the WeScience0 effort. Furthermore, only quite limited information about the original environment is recorded in the PDF files, hence it may at times be impossible to give exact answers to the dimensions of variation listed above. However, we need to find out what information about the production process actually is available in PDF files, and we will need a simple tool to inspect PDF meta information and extract relevant parameters. It is possible that some of the text extraction tools for PDF (see the NoraExtraction page) can be put to use in this task too.

Preliminary Notes

A report on observations in the extraction of PDF documents can be downloaded here: [http://folk.uio.no/gisley/wescience0/ola-duokonvnot1.pdf]

Metadata Survey

"Producer/Creator" fields from circa 3056 documents: [http://heim.ifi.uio.no/olasba/nora/metadata-sort1.txt]

grep -i word metadata-sort1.txt | wc -l  yields 1784 entries where Word was involved in some way. The occurrences of "tex" work out to 259, "ghostscript" to 557.

A rough survey of this metadata was not obviously helpful. There's a great variety of different versions, and significant differences just among, e.g., the four documents produced with MiKTeX -- the great versatility of TeX may actually work against us, and the large percentage of documents from Word may actually make things easier.

Pathological Documents

Documents that are impaired or unreadable due to unconventional font encoding:

Ghostscript documents which only feature unconverted glyphs on their front pages:

In all these cases, other documents with an identical Creator field did not suffer similar problems.

Spacing issues:

Blank/unconverted documents:

For a more comprehensive list, observe the last screenfuls of ls -lS /logon/scratch/johanbev/raw-output/  

Breakdown of naive scrape 17/9

Not looking too bad. Of our ~9000 pdfs, only 205 failed to be extracted. Here fail means our current pipeline had to exit with an error for some reason. There is still many files with a lot of extraction artifacts.

Table of errors encountered
Num Error
72 BouncyCastle
12 No versioninfo
111 WrappedIO
10 Misc

The BouncyCastle error is caused by a missing jar-file which is related to encryption. This jar could probably be integrated before next scrape.

WrappedIO looks like problems with non-conforming pdfs. Much more on this on next scrape.

There are still some number of misc. errors most with either zipping or encodings.

Results are available at /logon/scratch/johanbev/raw-output with log at /logon/scratch/johanbev/errors.log

Non-PDF files in the corpus

The 111 errors above are not surprising considering  grep -iv '\.pdf$' links.txt  reveals 103 files which did not originally have a .pdf extension. (Full listing in [http://heim.ifi.uio.no/olasba/nora/non-pdfs.txt]) These break down as follows:
.zip 8 .html 5
.doc 65 .htm 1
.docm 1 .tex 3
.rtf 8 .ps 1
.odt 2 .wmv 1
(none) 8

A quick comparison of grep -v bouncycastle errors.log with non-pdfs indicates that these are largely the very files that cause WrappedIOException.

(HTML files sometimes cause versioninfo errors, sometimes WrappedIOException.)

The only non-.pdf files that do not come out with a filesize of 0 are 18821, 16425, 15446, 16511, 1250, 8995, 10685 — these are seven out of the eight files that did not originally have a filename extension; the eighth, 22431, is reported to be an Office document by file(1).

Home | Forum | Discussions | Events

Clone this wiki locally