This repo contains the slides used for the Mercator workshop during the CENTR R&D workshop in Tallinn on 31 March 2022.
We at DNS Belgium are pleased to announce the release of the source code of our crawler called Mercator.
For now we use a private repo and will grant access to it to the CENTR community upon request. We hope to be able to move Mercator to a public repository in mid-2022.
Our crawler is capable of collecting information about DNS, SMTP, SSL, Web (HTTP headers, screenshots, web technologies, VAT number, ...). It is intended to run in AWS / Kubernetes in order to benefit from the scalability offered by these technologies.
We are running it internally and continuously crawl domain names based on certain criteria. We are also crawling all domain names of our zone every month. It is still in active development and we have lot of ideas for the future. We hope to bring the crawler to the next level by sharing the source code and possibly receiving contributions from other parties.
In order to obtain access to the repository, please send us an email with your GitHub username to [email protected]
During the workshop we will show
- how to build it
- how to locally start it, so you can try it out outside of AWS
- and how to deploy it in the cloud using Terraform and Kubernetes.
This workshop will take place during the CENTR R&D meeting at the end of march. You can either join remotely or onsite. You need to be registered in order to participate.
Take this as an opportunity to get onboard, ask questions, and start using Mercator!