Social Networking Influence: Common tendencies on social websites can spill above into search engines like google, affecting the look for volume of connected search phrases. txt file is then parsed and can instruct the robotic concerning which internet pages are not to get crawled. As being a online search engine https://www.youtube.com/watch?v=MxDLv74CqK0