CSV plugin fails to list the files from a HTTP URL

[Subhajit97]

In the CSV plugin, If we configure the path with an HTTP URL, for example:

connection "csv" {
  plugin = "csv"
  paths  = ["https://www.cisa.gov/sites/default/files/csv/known_exploited_vulnerabilities.csv"]
  # paths = ["http://surl.li/hkxeb"]
}

We are getting the below error:

[ERROR] steampipe-plugin-csv.plugin: [ERROR] csv.csvList: failed to fetch absolute path="failed to get directory specified by the source https://www.cisa.gov/sites/default/files/csv/known_exploited_vulnerabilities.csv: error downloading 'https://www.cisa.gov/sites/default/files/csv/known_exploited_vulnerabilities.csv': XML syntax error on line 147: expected /> in element"

We have tried this URL with a sample go-getter code snippet, and it throws the same error.

I have tried to update the GetFiles function (where we implemented the go-getter functionality). Currently, it uses the Get method defined in the go-getter library.

I tried to update it to use GetAny method instead of Get, and the above URL works fine. But the S3 URLs we currently use fail in that case.

Issue link: hashicorp/go-getter#438

Considering a temporary workaround, maybe we can use something like the below?

// use GetAny for any https or http URL
// and, for other case use the Get method
if strings.HasPrefix(remoteSourcePath, "https://") || strings.HasPrefix(remoteSourcePath, "http://") {
	err = getter.GetAny(dest, remoteSourcePath)
	if err != nil {
		return "", "", fmt.Errorf("failed to get directory specified by the source %s: %s", remoteSourcePath, err.Error())
	}
} else {
	err = getter.Get(dest, remoteSourcePath)
	if err != nil {
		return "", "", fmt.Errorf("failed to get directory specified by the source %s: %s", remoteSourcePath, err.Error())
	}
}