You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The provided APIDataSet accepts only a single URL.
I'd like to request support for multiple, potentially a lot of, URLs similar to PartitionedDataSet interface, without being interrupted even if errors occur for some of the URLs.
Context
It is useful to get multiple contents (e.g. json, image, video, model, zip files) through API.
Kedro provides PartitionedDataSet to support multiple datasets in a dict, but cannot be used for this purpose because:
It is possible that some of the URLs return errors while the others work fine. An option to skip errors and get contents as many as possible will make sense.
It is inefficient to configure a TCP connection for every requests.request call. Instead, requests.Session allows to reuse the same TCP connection to handle multiple requests in less time.
Possible Implementation
Accept a dict of URLs similar to PartitionedDataSet.
Description
The provided
APIDataSet
accepts only a single URL.I'd like to request support for multiple, potentially a lot of, URLs similar to
PartitionedDataSet
interface, without being interrupted even if errors occur for some of the URLs.Context
It is useful to get multiple contents (e.g. json, image, video, model, zip files) through API.
Kedro provides
PartitionedDataSet
to support multiple datasets in a dict, but cannot be used for this purpose because:It is possible that some of the URLs return errors while the others work fine. An option to skip errors and get contents as many as possible will make sense.
It is inefficient to configure a TCP connection for every
requests.request
call. Instead,requests.Session
allows to reuse the same TCP connection to handle multiple requests in less time.Possible Implementation
PartitionedDataSet
.requests.Session
in__init__
method.requests.Session
object to send requests.The text was updated successfully, but these errors were encountered: