What is the purpose of the 'robots.txt' file on a website?
A. To configure the web server
B. To store user login credentials
C. To instruct search engine crawlers which parts of the site to index
D. To block malicious users
Answer: Option C
Solution (By JKExamLibrary)
The robots.txt file is a standard used by websites to communicate with web crawlers, indicating which pages or sections should not be crawled or indexed.
Explanation:
The Deep Web encompasses parts of the internet not indexed by traditional search engines, including password-protected portals and private databases.
No comments yet. Be the first to start the discussion!