[PERF] Lazily import heavy modules to speed up import times

[desmondcheongzx]

Introduce lazy imports for heavy modules that are not needed as top-level imports. For example, ray does not need to be a top level import (it should only be imported when using the ray runner or when specific ray data extension types needed. Another example would be UnityCatalogTable, which is a relatively heavy import despite only being needed when using delta lake.

Modules to import lazily were determined by the proportion of import time as shown by importtime-output-wrapper -c 'import daft' --format waterfall --depth 25.

The list of newly lazily imported modules are:

• daft.unity_catalog
• fsspec
• numpy
• pandas
• PIL.Image
• pyarrow
• pyarrow.csv
• pyarrow.dataset
• pyarrow.fs
• pyarrow.json
• pyarrow.parquet
• ray
• ray.data.extensions
• xml.etree.ElementTree

Uses #2836 in order to defer the import of pyarrow.

Additionally, we move all type-checking-only module imports into type checking blocks.

With these changes, import times go from roughly 0.6-0.7s to ~0.045s (~13-15x faster).

[codspeed-hq]

CodSpeed Performance Report

Merging #2826 will degrade performances by 42.95%

_{Comparing desmondcheongzx:add-lazy-import (f888e49) with main (dba931f)}

Summary

⚡ 2 improvements
❌ 1 regressions
✅ 13 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	desmondcheongzx:add-lazy-import	Change
❌	test_count[1 Small File]	13.6 ms	23.9 ms	-42.95%
⚡	test_explain[100 Small Files]	82 ms	33.2 ms	×2.5
⚡	test_show[100 Small Files]	318 ms	56.3 ms	×5.6

[samster25]

Looks great to me!! Nice work.

I was also wondering if we can put in some lint rules as a follow on that will prohibit folks in the future from raw import our list of "bad" deps

It seems that ruff does support this
astral-sh/ruff#2656

[samster25]

As a follow on, we should look into if we can just depreciate fsspec. I'm not sure if we need it as a default dep anymore

Any thoughts @jaychia?

[samster25]

Looks good!

[samster25]

Looks good!