Technical SEO: Technological Search engine optimization refers to Internet site and server optimization that assists the crawler in crawling, indexing and ranking operations to rank your internet site much better. txt file is then parsed and can instruct the robotic regarding which pages will not be to become crawled. Like https://www.youtube.com/watch?v=EGBkutWZZTk