What is the purpose of the 'robots.txt' file on a website? MCQ with Answer and Explanation

What is the purpose of the 'robots.txt' file on a website?
A. To configure the web server
B. To store user login credentials
C. To instruct search engine crawlers which parts of the site to index
D. To block malicious users
Answer: Option C
Solution (By JKExamLibrary)
The robots.txt file is a standard used by websites to communicate with web crawlers, indicating which pages or sections should not be crawled or indexed.

This question belongs to: Computer Internet and Web Technologies

Discuss this Question (0)

No comments yet. Be the first to start the discussion!

Practice More Internet and Web Technologies Questions

Question #1 Report Error
Which of the following is a popular search engine that emphasizes privacy and does not track users?
A. Yahoo
B. Bing
C. DuckDuckGo
D. Google

Correct Answer: Option C


Explanation:
DuckDuckGo is a search engine that focuses on user privacy, not storing personal information or tracking search history.

This question belongs to: Computer Internet and Web Technologies
Question #2 Report Error
Which of the following parameters uniquely distinguishes a 'Deep Web' page from a standard surface web page?
A. It can only be accessed using specialized Tor networks.
B. It is encrypted using military-grade security.
C. It is not indexed by conventional search engine crawlers.
D. It requires high-speed fiber optic connectivity to load.

Correct Answer: Option C


Explanation:
The Deep Web encompasses parts of the internet not indexed by traditional search engines, including password-protected portals and private databases.

This question belongs to: Computer Internet and Web Technologies
Question #3 Report Error
What is the main function of a web proxy?
A. To block malicious websites
B. To provide web hosting
C. To encrypt web traffic
D. To cache web pages and forward requests

Correct Answer: Option D


Explanation:
A web proxy acts as an intermediary between clients and servers, often used for caching, filtering, or anonymity.

This question belongs to: Computer Internet and Web Technologies