Why would websites have robots.txt ?

By -
Few days ago, a new concept came landed on my desk - ROBOTS.TXT and I haven't heard about it before. I have asked many of my friends/colleagues.

Thanks to my guide - Thomas, who gave me a clear thoughts about robots.txt. Here is my learning notes from it.
Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Web site owners use the /robots.txt - text file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
This file is placed on web server under root folder and advises spiders and other robots which directories or files they should/ should not access.
Example: www.yourwebsite.com/robots.txt
What does it do exactly?When a keyword is searched, first thing web spiders visits is robots.txt file. It looks in the file to know what it should do - based on the instructions mentioned in robots.txt
Why robots.txt file?

1. Most people want robots to visit all the web pages and content in their website. 
           How to make this work?1. Don't have robots.txt file in the web server2. Add empty robots.txt file without any instructions3. Add robots.txt file with the following instructions:User-agent: *Disallow:Here, * - means any web robots.
Disallow -  tells the robots what folders they should not look at in website.* List of various web robots: www.robotstxt.org/db.html

2. Few people want robots to restrict some files in the web server and few files allows to visit.User-agent: * Disallow: /checkoutAllow: /imagesThe "Allow:" instructions tells web robot, that it is okay to see a files in images folder.

How to test robots.txt file?
1. We can test robots.txt with the Google Webmaster tool - robots.txt Tester2.  To find out if an individual page is blocked by robots.txt you can use this online tool which will tell you if a page is blocked or not.
Location: Mysore, Karnataka, India

Popular Posts

JMeter Producing Error: Windows RegCreateKeyEx(...) returned error code 5

By -
After running JMeter.bat on Command Line, Problem java.util.prefs.WindowsPreferences <init> WARNING: Could not open/create prefs root node Software\JavaSoft\Prefs at root 0x80000002. Windows RegCreateKeyEx(...) returned error code 5. Cause Environment Resolving the problem The error occurs because java.util.prefs.WindowsPreferences is trying to save information in HKEY_LOCAL_MACHINE\Software\JavaSoft\Prefs instead of under HKEY_CURRENT_USER\Software\JavaSoft\Prefs. Windows 7 64-bit The work around is to Open JMeter.bat file as the administrator and so it creates the key HKEY_LOCAL_MACHINE\Software\JavaSoft\Prefs.
Read more

Understanding about Contract Testing

By -
Image
Yesterday Morning (9th May 2020), I had good discussion with  Sundaresan  Krishnaswami on " Contract Testing ". I thank  Sundaresan   for his time and sharing his knowledge. It started with an example: The Owner provides the house or Place of Residence to the Tenants. "Rent Agreement" is provided for the house and It contains agreement to use the house. The house comes with like kitchen, living room, bed room, fans, geysers etc.. It also contains who resides and what is the contractual period, obligation (money) etc.. Here, Owner is the Server and Tenant is Client.                                         Client-Server Architecture. Illustrating the same in the E-Commerce Context: In Some Development Companies, There can be two teams developing the Payments and Orders APIs. In B2B or B2C Ecommerce applications, we place an order using web browser through payment methods and Order is created. Order is successful only when Payment is successful. In general, Payments te
Read more