What is robot.txt file?

Question

I have heard that many social networking site use roboy.txt files for some purpose of security from search engines. I want to know exactly what that file does.

Answer 1

I recently read an article about Google and others thinking about ignoring the robots.txt file, because some websites don't deliver any information for the search results due to stupid configurations in the robots.txt.
search engines became intelligent enough to determine wether the content is relevant or not. everything publicly reachable on a website should be able to be found via search engines.

sorry if I made any grammatical mistakes here. got confused myself 😂

Answer 2

Hi,this file is placed on the server (in your installation of your web).This file contains information for searching agents (Like google bots and others)The robots file contains links which should be ignore by searching agents.The robots love this file helps them to distinguish relevan and irrelevant links of your web.

Answer 3

No! forget all what are you hear! This is the true about only one robot.txt file. One day google will be "alive" and there will be names of people who must to survive machine attack whom google will lead them ... and all others will be exterminated... :-D