1

Topic: Suspiciously high numbers of page views have stopped - here is why

For about a week this forum experienced a sudden spike in both page views and visitors shown here.
The interesting thing was that only a few topics were viewed while many others were not.
In particular all Ron Paul topics, video and documentaries topics - not much else.
In one Ron Paul topic we got about 500 page views per day.

I used Awstats to process my logfile and it only showed a dozen visitors or so which one would expect in a new forum with few users yet.
I also eventually got Google analytics to work and it agreed more or less with Awstats.

I just figures out what happened. A look at my raw logfiles revealed that I was visited by the Chinese/Japanese search engine Baidu specifically its robot: Baiduspider every few seconds or so! it kept changing IP addresses too.
Upon creating the robot.txt file with

User-agent: Baiduspider
Disallow: /

the number of visitor fell from about 7 to 3. Upon blocking the IP range 119.63.192.0 - 119.63.199.255 it went down to near zero... what a relieve :)
I find it interesting that only topics that affected US foreign politics were spidered so intensely. I have seen at list one poster in a different forum who complained the baidu only spidered his index page - well, his forum was about software.
I have copied below the info on the baiduspider, in particular the ip range may of interest for someone with similar problems.

I spent some time to see if there is a way to reduce the unnecessarily high frequency of visits by that spider and found the following:

User-agent: Googlebot
Crawl-delay: 5

This will limit Googlebot requests rate to 1 per 5 seconds.

at http://www.wawuk.net/quick-tips/how-to-write-robots-txt

As I would not mind being spidered but perhaps once a day would be enough.

I have now completely blocked the spider with an IP block but might try the delay option one day or manually allow the baiduspider every now and then. Pitty I had to block it.
-------------------------------------------------------
april 2011:
IP Location: Japan Tokyo Baidu Japan Inc


inetnum: 119.63.192.0 - 119.63.199.255
netname: BAIDUJP
descr: Baidu Japan Inc.
descr: Roppongi-Hills Mori-Tower 20th Floor,
descr: 6-10-1 Roppongi Minato-ku, Tokyo 106-0032 Japan
country: JP
admin-c: JNIC1-AP
tech-c: JNIC1-AP
status: ALLOCATED PORTABLE
remarks: Email address for spam or abuse complaints :
changed: 20070122
changed: 20100414
mnt-by: MAINT-JPNIC
mnt-lower: MAINT-JPNIC
source: APNIC

role: Japan Network Information Center
address: Kokusai-Kougyou-Kanda Bldg 6F, 2-3-4 Uchi-Kanda
address: Chiyoda-ku, Tokyo 101-0047, Japan
country: JP
phone: +81-3-5297-2311
fax-no: +81-3-5297-2312
e-mail:
admin-c: JI13-AP
tech-c: JE53-AP
nic-hdl: JNIC1-AP
mnt-by: MAINT-JPNIC
changed: 20041222
changed: 20050324
changed: 20051027
source: APNIC

inetnum: 119.63.192.0 - 119.63.199.255
netname: BAIDUJP-CIDR-BLK-JP
descr: Baidu Japan Inc.
remarks: Email address for spam or abuse complaints :
country: JP
admin-c: RS2845JP
tech-c: RS2845JP
remarks: This information has been partially mirrored by APNIC from
remarks: JPNIC. To obtain more specific information, please use the
remarks: JPNIC WHOIS Gateway at
remarks: http://www.nic.ad.jp/en/db/whois/en-gateway.html or
remarks: whois.nic.ad.jp for WHOIS client. (The WHOIS client
remarks: defaults to Japanese output, use the /e switch for English
remarks: output)
changed: 20080122
changed: 20100414
source: JPNIC