a decentralized search engine server standard
DecentSearch will consists of a network of independent servers. Each contributing its own index data to the network and hosting portions of others. None of these server have to run the same code, just implement the same API.
Disclaimer: This project is still work in progress anything written here, may and propably will change in the course of the next months.
It works by a ruleset that governs indexing, index distributen, discovery, search and security between all participating nodes.
The server will be given URLs to index. Their content is cleaned from any format syntax eg. HTML-Tags. The cleaned content is split into words. Each word is saved in a reverse index pointing to the source file. Additionally metadata for each file is saved. Each server manages its local search index and search parameters.
The user will access the search through a website that will send the request to a DecentSearch server. That server will then search through it's internal and external index and in parallel will send a search request for each word to known, trusted servers that are expected to have a result for that word.
The internal index is searched for matching words, the resulting URLs are ranked by the amount of contained words.
In the external index only words with a hash that is within its hash range, are searched. The words then are requested from servers that are known to have a hash range that matches that particular word.
Interested to contribute? contact me