[http][source] invalid IP address syntax for key address for key sources.my_source

[Diliz]

Hello Timberio PPLs!

I'm actually using the http source in my vector config, with an alias formated this way: api.region.something.something:port, actually this seems to cause a format error, I thing it's expecting less dots there or something else :/

Here my config:

[sources.test]
  type = "http"
  address = "api.region.something.something:80"
  encoding = "json"

[sinks.stdout]
  inputs = ["test"]
  type = "console"
  encoding.codec = "text"

The error:

vector --config druid/*.toml --dry-run -v
May 25 16:20:10.202  INFO vector: Log level "debug" is enabled.
May 25 16:20:10.202  INFO vector: Loading configs. path=["druid/pure_backend.toml"]
May 25 16:20:10.206 ERROR vector: Configuration error: "druid/pure_backend.toml": invalid IP 
address syntax for key `address` for key `sources.test`

My vector version:

vector --version
vector 0.9.1 (v0.9.1 x86_64-unknown-linux-musl 2020-04-30)

Seems you're importing a standard library there, I'm actually not a rust developper, saw this if it can help:

• Tracking issue for DNS in std::net rust-lang/rust#27705
• Deprecate std::net::lookup_host rust-lang/rust#47510

Hope this helps, and hope it's a good hint on solving this problem.

Thanks for reading, and thanks for your work 👍

PS: Is there a way to call specific endpoints like 0.0.0.0:port/endpoint? I'm actually not able to target a specific endpoint like this, and don't find it in the http source documentation as well, do you got any hint on this?

I dunno it this can help, but I think more advanced examples of http calls usages, with a body, etc... Can be helpful, with the same examples as curl commands, this is just ideas, but everybody knows what are doing curl commands (for example), so it's more a kind of traduction of what it will do with the provided parameters if you want to do a manual request (maybe I can open another ticket there to talk about this if you want 👍 )

[bill-bateman]

Hey - not a timberio person but I worked on http source.

For address it expects an IP address, not a domain name. There is a notion of doing DNS resolution in Vector, but as far as I know that's mainly for sinks (and might be removed anyway according to this issue #2635). I don't think there is any push for allowing domain name for sources but an actual timberio person would know better.

For the endpoint, right now that's something we can't add (at least not in a nice way) due to the fact that we're on an old version of the Warp library. The function for specifying the path is called warp::path

Old version: https://docs.rs/warp/0.1.18/warp/filters/path/fn.path.html
New version: https://docs.rs/warp/0.2.3/warp/filters/path/fn.path.html

The old version requires the input to be of type &'static str. (https://doc.rust-lang.org/rust-by-example/scope/lifetime/static_lifetime.html) There's no safe way to turn something into a 'static (you have to leak the memory). But with the new version of warp, we wouldn't need to. There's an issue to upgrade the warp version (#2643) - after the upgrade could add an extra config field for setting the path or something similar.

There was a way to get full path and maybe match on everything (https://docs.rs/warp/0.1.18/warp/filters/path/fn.full.html) - didn't try to add it though.

If you want a configurable endpoint might be good to open a separate issue.

Hope that helps!

[Diliz]

Thanks @bill-bateman for the explaination there, I understand the issue there

It's weird to not be able to make http requests on domain names since it's an http source, I think a lot of ppls need to do this if they use the http source/sink. Not being able to provide an endpoint here is a weird behavior too :/

I wanted to use the http source actually to make some calls to some druid apis, like this one:
https://druid.apache.org/docs/latest/querying/querying.html

I can't do it at all, even if it's a simple http query I'm trying to make with vector :/ Hope this will be changed in the future, I'll open an other issue asking to provide an endpoint attribute or being able to provide it in the http address part by some some way 👍

Thanks again, it helps a lot to understand what was wrong there :)