What is Robots.txt File and How to Set It

As a blogger, maybe you are ever to heard the word of 'robots.txt'. I believe, you surely confused when heard it. Well, I also confused when i heard it first time. But Now, i had know it and i will share and explain it to you as my best.

What is Robots.txt

Robot.txt file is a file with text(.txt) format that owned by each sites that online in internet. Any sites should have this file or the security of that site will broken. Robot.txt serves to control and regulate which the pages or directories will published in search engines. how it works is by put some codes inside it to control search engines to crawl or not the pages in that website. It surely to protect your system pages to known by people. Administrator pages is a page which abstinence to known by peoples or indexed by search engines. So, all of sites surely disallowing it administrator pages to indexed by search engines. Well, It is did by robot.txt. Robot.txt allow you to manage which the pages may crawled and indexed by search engines.
same as website, Blog also have robots.txt file. Only, robots.txt in a blog usually have set as default by the blogger platform. For blogspot blog, the default of the robots.txt is set like this :

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://blogURL/feeds/posts/default?orderby=UPDATED


The meaning of the code above is the robots.txt file allow the google adsense crawler to crawl all pages of your site invariably, and allow all crawler to crawl all of your blog pages except the page with url prefix blogURL/search. The pages with this url prefix is labels and archives post pages.

How to Set Robots.txt

If you want to modify and set your robots.txt, firstly you must be careful because an error of your setting will make your website SERP on search engines disappear. You must have a clear purpose because it is not a little thing for your site/blog.
Okay, now I will tell you how to set it by right. You must know the meaning of each codes in robots.txt file above.

" user-agent : Mediapartners-Google " : it is means that the codes which placed under it apply only with Mediapartners-Google crawler. Mediapartners-Google is google adsense crawler.

" Disallow: " : It is means that no pages disallowed to crawl by crawler. By other word, allow the crawler to crawl all pages on the site.

" user-agent : * " : it is means that the codes which placed under it is apply with all of search engines/crawler

" Disallow: /search " : Disallow the crawler selected to crawl the url that have prefix blogurl/seach...

" Allow: / " : Allow the crawler to crawl all of pages.

" Sitemap: http://blogURL/feeds/posts/default?orderby=UPDATED " : this is sitemap of your site which submitted to crawled by crawler.

Not only the codes above can use in robots.txt. Just for the example, i want to blocked this post by robots.txt in order do not indexed by all of search engines. so i can do it by put this code.

user-agent : *
Disallow: /2012/11/what-is-robots-and-how-set-it.html

well, to block the root folders or directories(included it contents) in your site, you can do this :

user-agent : *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/

we can also set robots.txt in order to do not crawl and index a specific keyword. the example we want to block the word "people" to no-indexed by all search engines. we can set like this :

user-agent : *
Disallow: /search/*people

we can also set robots.txt to block the file in your site with specific format. The example we will block file with format php(.php), css(.css) and javascript(.js).

user-agent : *
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.css$

User Agent

In setting of your robots.txt, you can set which the crawler you allowed or disallowed to crawl. This is did by setting the user agent. Therefore, you must know the names of some famous crawlers/spiders. This is it lists :

Google adsense - Mediapartners-google
Google - Googlebot
Altavista - Scooter
Lycos - Lycos_Spider_(T-Rex)
Alltheweb - FAST-WebCrawler/
INKTOMI - Slurp
Yahoo - Yahoo Slurp
MSN- Msnbot

*note :
-Any example i explain above is for blogspot, except the example for block folders/directories (it is for wordpress. Each sites have different set of robots.txt. It is because each sites have different directories too. The setting robots.txt of wordpress blog surely will different with blogspot blog, but the rules of codes are same.
-If you have a sites which no use blogspot platform. you must add robots.txt file and locate it on main root of your site.

well, that's all. Still any questions ? don't hesitate to ask me. Now, you have know and can set your robots.txt file of your site. i waiting for your response. Hopefully useful for you. Keep happy blogging guys :)

Read More...

How to Submit/Register Your Blog/Site to Google Webmaster Tools

After you read the previous article : "google webmaster tools and the features", i sure, you want to use google webmaster tools because the advantages that i said in previous article, Right?. Probably, when you will use google webmaster tools for your site, you confuse to do it ?
Well, the first step is submit your site to google webmaster tools. So, in this article i will explain you how to submit or register your blog/site to google webmaster tools.
Okay, just follow the steps below :

1. You must have a google account. if you not registered yet, just sign up to google.

2. After you have a google account, just visit google webmaster tools and sign in by your google account.

3. After you log in, See and click the button "Add A Site" on right top corner.




4. Just fill the textbox with your site/blog url.

5. Then, Follow the way to verify your site. make sure you followed by right.




6. After you done the step 5, just Click "verify" on the same page.

7. Finish, your site verified and you can use google webmaster tools.

*for you who use blogger/blogspot platform, generally your blog has registered automatically. you just verify your site, then you can use google webmaster tools.

That's it. hope you helped by this article. i waiting for your response in the comment box below :)

Happy Blogging :)

Read More...

Google Webmaster Tools and The Features

Yo.. we will talking about webmaster tools again. Now the topic is about google webmaster tools.
google webmaster tools is webmaster tools owned by google and the best webmaster tools now. Google webmater tools is so important to used because it is related with your SEO in google search engine. you surely have been known that google search engine is the visitors source of most sites in the world. About 80% of the blogs/sites visitors in the world is source from google search engine.
Google webmaster tools also have complete features than other webmaster tools. the features is various and very useful. it can improve your SEO, tell you about the error of your sites, repair your sites perfomance on google search engines and others.

Well, Therefore, Here i will explain you about the features of google webmaster tools :

1. Configuration
as name, it is for configuration of your sites on google search engine. Configuration divided become 6 menu. they are :
-Setting. To set the visitors target based geographic location, to set preferred domain, and crawl rate on google search engine.
-Sitelinks. To add sitelinks on search engine result for your site.
-URL parameter.
-Change of address.
-Users. to add and manage the users/owners of the sites.
-Associates. To add and manage associates users

2. Health
It is to check and know your site health. it is so important for your site index on google search engine. Health divided become 6 menu too. they are :
-Crawl Errors. to check crawl errors of URL on your sites. these error found by google crawler
-Crawl Stats. to see the statistic of google crawl on your sites
-Blocked URLs. To see how many blocked Url by robot.txt on your site. you also can test your robot.txt file.
-Fetch as google. instruct google to fetch and index the url you put.
-index status. To see index status of your site.
-malware. to check whether your site contains malware.

3. Traffic
it is to see the report of site traffic. Traffic divided become 3 menu. they are :
-Search queries. To see what the queries or words that used by google search engine user to find and visit your site. you also can see the impression and click of your site result on search engines
-Links to your site. external links (from other site) to your site
-internal links.

4.  Optimization
I think This feature is most important. Because it is to optimize your site on google search engine. Optimization divided become 6 menu. they are :
-Sitemaps. To submit your sitemaps of your site to google search engine
-Remove URLs. To remove the url(page,post, and other url) on site that indexed or detected by google crawler.
-HTML improvements. To check HTML improvements of site.
-Content keywords. To know what the keywords used by google search engine user to find the content of your site. the keyword is arranged from the keyword that most used to least used.
-Structured data. To see your Structured data of your site.
-Other resources.

5. Labs
it is for testing and experimenting your site on google search engine. Labs divided become 4 menu. they are :
-Author stats. to see the author statistics.
-Custom Search. to test your site result on google search engine. the result is only your site and the contents.
-Instant Previews.
-Site performance.

That is the features of google webmaster tools. besides the 5 features above, you also get other services. that is you can get help if you have question about google webmaster tools. Just choose the Help link that locate at below of the features list above.

That's it.
What is your opinion ? I waiting you to leave comment below ;)
Happy Blogging :)

Read More...

How important Webmaster tools for your blogs/sites

Are you know webmaster tools ?.. Okay, let me to tell you. Webmaster is a tool that usually used by website/blog owners to check their web or blog perfomance on a search engine. webmaster tools usually provided by search engines. almost all of famous search engines in the world, provide this tools. surely, this service is free for user.

You can access the webmaster tool of a search engine if you have been registered on related search engine. To submit and register your sites to search engines, just read also this article : "Submit Blogs/sites to 20 search engines/directories all at once". However, if you use blogger as your platform, your blog will automatically registered to google search engine and it webmaster tool.

Webmaster tools allows you to check and control your blogs/sites perfomance on related search engine. Webmaster tools provide many services that help you to repair and optimize your blog SEO and perfomance on search engines. Example services is to submit sitemap, Check error URL and many others (google webmaster tool services). can be said, Webmaster tools is one of vital point of your SEO. these is that make webmaster tools is so important.

Until now, At Least there are 2 famous webmaster tools that most important to you used. those is google Webmaster tools and Bing webmaster tools. It is Because both of them owned by famous search engines. your sites will easy to found by peoples through these search engines (google and bing).

However, if i talk about which the best of webmaster tools, i will answer : it is google webmaster tools. Why ? it is because the reason below:
1. google is most famous and most used search engine in the world
2. google webmaster tools have many services that is not found in other webmaster tools
3. google webmaster tools allows you to improve your blog SEO in google search engine.
4. The error of your blogs/sites can detected by google webmaster tools and report it to you so you can repair it immediatelly.

So, I will strongly recommend you to use google webmaster tools more of others. And i suggest you to frequently check your blogs perfomance on google webmaster tools.

Okay, That's it. I will post about webmaster tools tips and how to use it after some times from now. Just wait my other post..

What's your opinion about this article ? i waiting for your response here.. Hope this useful and Happy Blogging :)

Read More...