A web scraping platform for the AI era.
git clone https://github.com/Datasilk/Collector
git submodule init
git submodule update --init
cd Collector/Collector.App
npm install
I started this project in 2015 by building Charlotte along with a web UI in ASP.NET Core using C#. I've built this project from the ground up several times, eventually transforming it into a plugin for Saber (a website builder), and so now I am turning it into a set of tools.
The new Collector app will be a .NET command-line-based web server with a web UI for managing all your collections of data.
A common .NET library that contains all the common functionality of the Collector App so that you can build your own app to collect data from the web.
A plugin for Command Center that allows the system to scrape the web for YouTube videos based on the user's needs.
A command-line tool that allows users to speak with an AI in real-time and utilize all of Collector's tools to gather, catalog, and parse intelligence from the web and beyond.