Replace numpy usage and remove from pyproject.toml

[kevinjqliu]

Follow up to #1270
Replace all other references of numpy and remove numpy from pyproject.toml.

The current reference of numpy in poetry.lock is coming from pandas. From poetry show --tree,

pandas 2.0.3 Powerful data structures for data analysis, time series, and statistics
├── numpy >=1.20.3
├── numpy >=1.23.2
├── numpy >=1.21.0
├── python-dateutil >=2.8.2
│   └── six >=1.5 
├── pytz >=2020.1
└── tzdata >=2022.1

[Fokko]

Even better, thanks @kevinjqliu