Velen Crawler

Velen is a crawler written in Go for Hunter. It analyzes millions of public internet pages every month. We only crawl pages that are publicly accessible. Any page behind a login or any other authentication won\'t be crawled. Our goal with this crawler is to build business datasets and machine learning models to better understand the web.