[Dataprep] Reduce Upload File Time Consumption

[letonghan]

Description

File upload time too long according to feedback by customers.
Refine dataprep util with opencv and multithreading.
Time reduction:

• Load File: reduce by 70%~83%, depends on network and hardware
• Bottleneck of Save to DB: blocked by the performance of TEI service

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

Add pytesseract in requirements.txt.

Tests

Local tested.

• Time comsumption on Xeon:

file size	origin	func1	func2	refined
load document	8.119s	8.917s	3.039s	1.368s
save batch into db	12.492s	14.456s	13.055s	12.802s
total	20.725s	23.404s	16.166s	14.196s

• Time comsumption on Gaudi:

file size	origin	fun1	func2	refined
load document	8.47s	11.268s	4.164s	2.523s
save batch into db	1.123s	1.287s	1.285s	1.272s
total	9.613s	12.621s	5.462s	3.806s