Which file is placed on a website's server to give instructions to search engine web crawlers about which paths and directories should not be scanned or indexed? MCQ with Answer and Explanation

Which file is placed on a website's server to give instructions to search engine web crawlers about which paths and directories should not be scanned or indexed?
A. index.html
B. htaccess
C. sitemap.xml
D. robots.txt
Answer: Option D
Solution (By JKExamLibrary)
The robots.txt file is a standard text file used by websites to guide web robots and crawlers regarding which parts of the site they are allowed to index.

This question belongs to: Computer Internet and Web Technologies

Discuss this Question (0)

No comments yet. Be the first to start the discussion!

Practice More Internet and Web Technologies Questions

Question #1 Report Error
Which of the following is a version of IP that provides a larger address space to accommodate more devices?
A. IPv4
B. IPv6
C. IPv5
D. IPsec

Correct Answer: Option B


Explanation:
IPv6 (Internet Protocol version 6) was developed to address the exhaustion of IPv4 addresses, offering a 128-bit address space.

This question belongs to: Computer Internet and Web Technologies
Question #2 Report Error
Which of the following is not a web server software?
A. Nginx
B. IIS
C. Firefox
D. Apache

Correct Answer: Option C


Explanation:
Firefox is a web browser, not a web server. Apache, Nginx, and IIS (Internet Information Services) are web server software.

This question belongs to: Computer Internet and Web Technologies
Question #3 Report Error
What does a '405 Method Not Allowed' HTTP status code return imply about a client browser's interaction with a web application?
A. The input security certificate has an expired authorization date.
B. The web server facility has completely crashed down.
C. The requested URL address path has changed its location permanently.
D. The server recognizes the request method, but the specific target resource does not support that method (e.g., trying to POST to a read-only static text file).

Correct Answer: Option D


Explanation:
An HTTP 405 error occurs when the requested resource is configured to reject the specific HTTP method used (such as a browser attempting a POST action on a static page designed only to accept GET requests).

This question belongs to: Computer Internet and Web Technologies