twitter in the setting of Robots, with or without WWW WWW to set up two Robots standard. As shown below: (A is not www Robots B WWW settings, Robots settings)
first, Twitter PR and many pages are not grasping is caused by their technology, Never mind with Google search mechanism.
twitter led the unrest has five major technical problems:
as everyone knows, twitter some time ago, the PR value decreased from 9 to 0, and the number of pages is not Google Robot crawl, everyone in an uproar! Although it has been restored. But how is this going? Beijing website optimization research center of editing in the wind Valley Austria Dugu reference content for everyone to decrypt the twitter site
Every that might possibly # BOT read and respect this User-agent: *
#Yahoo! Search Engine Robot
turmoil the truth!
Search Engine Robot
#Microsoft Search Engine User-Agent: msnbot
A:The file at twitter贵族宝贝/robots.txt looks as follows: