Имя пользователя: 
Пароль: 
Ограничить сессию IP 
Вопросы  |  score: 1  |  1.21 3.53 3.41 |  Решена 11086 человеком/людьми  |  237587 views  |  с Dec 26, 2010 - 23:06:38

Training: WWW-Robots (HTTP, Training)

WWW-Robots
In this little training challenge, you are going to learn about the Robots_exclusion_standard.
The robots.txt file is used by web crawlers to check if they are allowed to crawl and index your website or only parts of it.
Sometimes these files reveal the directory structure instead protecting the content from being crawled.

Enjoy!
© 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023 and 2024 by Gizmore