Own index, use of AI & more in AstianGO

Own index, IA use & more in AstianGO

The use of AI has changed everything and has accelerated many processes so we regret to inform you that we will use AI for everything in Astian, No lies just kidding, yes we will use it but not in everything and later we will explain why? and what for?

Own Index

As you know AstianGO uses for the moment the Bing API, something that has allowed us to position ourselves, has allowed us to improve, listen to users and focus, but in our goal to generate the least possible dependence on external organizations that may affect our products, and that may compromise our values and processes either by privacy and / or lack of transparency we have decided to create our own index, that means that AstianGO over time will no longer depend on Bing but will have its own search base under the control of AstianGO which is available to users through an API.

We have been working and for those interested in observing and seeing how the process and architecture that we will be implementing in the development of our index here we leave the diagrams, which are stored in our cloud storage service and collaborative work Astian Cloud, , click here.

Use of AI in AstianGO

As communicated in the introduction of this article if we are going to use artificial intelligence in AstianGO, years ago when Google was born and was just consolidating its power and dominance there was no alternative but to do Web Spider or Web Scraping and multiple techniques to index all websites and then rank them according to pagerank, the same for Bing changes a little but has broadly speaking is not much difference, they still keep their bots, their spiders, their analyzers, indexing the entire web, taking into account this historical context, today we have many more tools to have our own index minimizing development time, implementation, deployment and feedback, that is where the use of AI enters in the analysis, ranking and evaluation of all websites.

For this task we will use Guild.ai is not only open source, but it allows us to do all the cleaning process, automate the process and streamline the construction process becoming a key piece and fundamentally of the construction of our independent search engine AstianGO shortening years in development time.

What we won't do!!!!!

The use of this tool is only limited to the use of evaluation and ranking of the billions of websites that exist on the web, we will not do things like:

  • We will not show results and/or assists or aids using AI.
  • We will not train any model with this, no, I repeat it will only be used to speed up the process of evaluating, ranking and rating the sites for our index.
  • We will not use personal data, we will only evaluate, rank and rate the content of websites for subsequent inclusion in the index.
  • No artificial intelligence models will be trained, it will only be used to rank and evaluate the thousands of websites.

As a measure to not depend on this functionality we will have our own crawler which will scan the web and the user can contribute to the expansion and construction of our index, how? manually sending the website to scan, example: A person Alfram search astian.org or youtube.com or facebook.com if AstianGO does not find anything, the same search engine will ask, Do you want to launch our crawler to the site you are looking for to index it? The user chooses whether to contribute or not voluntarily, our crawler will still pass by and capture the website.

Future for AstianGO

As time goes by AstianGO will step by step decrease its dependence on Bing and step by step will be mixing indexes, as we grow, as our index is nurtured and fed, until we reach the point of being a completely independent search engine. The road is long, there is a lot to go, to learn and to experiment and we want to do this with you, to build products where the PRODUCT IS NOT YOU.

Share this post