What is robots.txt and how does it work and how is it created? What else do you know about robots.txt? how is it created and why it is useful and how to add it to your website or blog or blogger, then read this post completely, if you liked this post then leave a lovely comment What is robots.txt? how to make

What is robots.txt?

As the name suggests, it is a .txt file i.e. only text can be written in it and by the name Robots we can understand that the robot will work on whatever data is saved in this file, it will work on the site ranking and SEO is. Useful in this file, which tells the search engine which data or post is to be shown in the search engine and which is not, using which we can prevent our personal information or page from appearing in the search engine.

Why robots.txt is  important?

You know why robots.txt is importaint, using this we can determine which search engine can read your website or not and whether it can read and show it in search

• Search engines ignore the duplicate pages on your website
• If it does not index the internal search results pages then
• If search engines do not index some of the pages directed to you
• Do not index some of your files like some images, PDFs etc.
• Telling search engines where your sitemap is located
• Your sensitive information can be kept private using robots.txt.
• With the help of Robots.txt, “canonicalization” problems can be kept away or multiple “canonical” URLs can also be kept. This problem is also called “duplicate content” problem. With this, you can also help Google Bots, this website is important in rank for indexing the pages.

How robots.txt works?

This is a simple process, whenever a search engine sees or indexes your website or blogger or post, it first reads robots.txt that the post or page it is going to index in robots.txt. Is. Whether it is allowed to be indexed and if robots.txt allows it, it starts showing that post in search and does not index it if it is not allowed. When the search engine reads robots.txt, the robots. The code we write in txt reads it

How to create robots.txt?

Although you can create your own robots.txt as a txt file, but you can also create it for your own ease from any website for free like:-
1.seoptimer.com
2.internetmarketingninjas.com
3.seobook.com
You can use these websites to generate robots.txt and download or copy and use it.

How to add robots.txt?

If you want to add robots.txt, then first of all check that if your website does not already have robots.txt then put /robots.txt after the domain in your website like :- www.hindihelper. com/robots.txt If the page is open and on it
User-agent: *

Disallow: /search

Allow: /
Sitemap: https://www.hindihelper.com/sitemap.xml

As any code comes then robots.txt is already on your website and you can edit it according to you and can also replace it with new code and if 404 page or note page found then you can do it in this way can add

Add to blogger :-
Click on Search Preferences in Blogger’s Settings and then click on robots.txt and click on Edit

A small box will open, paste the robots.txt code in it and save it.

Add to WordPress:-
If you use WordPress then you must have installed Yoast SEO then you have to go to yoast seo and click on Tools then click on File Editor
How to add robots.txt to WordPress?

Now a small box will open, paste the robots.txt code in it and save it.

Add to website :-
Upload it to the root directory of the website.
If you use subdomains then you need to create separate robots.txt file for all subdomains

Understand robots.txt code

Allow: If you allow something inside a text file, it means that you want this part to appear in search engines.

Disclaim: If you’re disapproving a post, it means you don’t want this part to show up in search engines.

User-agent: Using this option, you can show your visitors what you want to appear on your site, such as the source of the image, where this file is saved, etc.

Today we told you about robots.txt? How to make has been told. Tell us how you liked this information, if you liked this information, then do not forget to share it at all. We keep adding new information every day, so connecting to our blog’s email service is a must.