Comment by trante on Preventing high bandwidth usage from Yandex
Because these types of requests come to my site with very systematically. 10-15 times every day, with similar intervals. I don't think a normal visitors would have such patience to make it every day.
View ArticleComment by trante on Preventing botvisits to website
Yes. My server is heavily loaded, I'm thinking to lower the load. I don't think any visitors come to my site by seeing my content from these sites..
View ArticleComment by trante on Preventing Twitterbot to access website
Thank you. I only have a Twitter card for my android app. My website isn't suitable for creating different cards for each page. Because only authorized user can see his/her data.
View ArticleDisabling frequent reloads of visitor
In my website i have some pages, that makes connections to 3-4 sites and checks the results with my database and gathers a full table. This table includes all the data that the visitor would need.You...
View ArticleAnswer by trante for How can I force Google to re-index my site?
Just create a sitemap with the new style of URLs. Be sure that old URLs give 301 redirect when crawler reaches old style pages. Then all you can do is to wait. It will be well in 1-2 weeks. And...
View ArticlePreventing Twitterbot to access website
I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And...
View ArticleAnswer by trante for In Robots.txt, how do I disallow all pages except for...
I use like this:User-agent: YandexAllow: /index.htmlDisallow: /User-agent: GooglebotAllow: /index.htmlDisallow: /
View ArticleGooglebot fetches my pages very frequent, rel-nofollow, meta-noindex or...
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent.I decreased crawl rate from Google webmaster tools.But I'm supposing to use...
View ArticleInforming Googlebot for deprecated pages
I publish timetables in my website.For example last year I published Number 2 bus Summer 2013 timetable.I has pretty good ranking on Google SERPs for number 2 bus timetableBut this year I added a new...
View ArticleI18n website and URL prefix iso639
I'm adding i18n to my website. For translated pages I add iso639 code of the language like this:http://example.com/en/mypage.phpBut I'm curious about language code. Should I use iso639-1 (en) or...
View ArticleRestrict all robots except Googlebot and Yandexbot
I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages.All other bots will not be accessed my pages.Is this robots.txt suitable...
View Article
