|
|
网站一般欢迎蜘蛛访问，因为蜘蛛意味着搜索排名和流量,但是某些情况下，大量垃圾蜘蛛甚至爬虫很影响性能，特别是服务器配置不高的情况下，那么我们该怎样屏蔽掉垃圾蜘蛛呢？蜘蛛抓取是好事，但是一些无用的蜘蛛如MJ12bot等天天来“扫”你的网站、甚至是一些根本不存在的网页或者目录、文件，平白给网站增加负担、带来安全隐患，所以干脆写个免扰“通知”，告诉它们是不受欢迎的访客，不欢迎访问。以下整理的绝大部分垃圾蜘蛛屏蔽方法。
User-agent: Abonti
Disallow: /
User-agent: ADmantX
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: BacklinkCrawler
Disallow: /
User-agent: BPImageWalker
Disallow: /
User-agent: check_http
Disallow: /
User-agent: checks.panopta.com
Disallow: /
User-agent: curl
Disallow: /
User-agent: DLE_Spider.exe
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: EMail Exractor
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: GbPlugin
Disallow: /
User-agent: gooblog
Disallow: /
User-agent: gsa-crawler
Disallow: /
User-agent: HolmesBot
Disallow: /
User-agent: ichiro
Disallow: /
User-agent: Infoseek SideWinder
Disallow: /
User-agent: Infoseek SideWinder/2.0B (Linux 2.4 i686)
Disallow: /
User-agent: ip-web-crawler.com
Disallow: /
User-agent: Jakarta Commons-HttpClient
Disallow: /
User-agent: Java
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: libwww-perl
Disallow: /
User-agent: linkdex.com
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: moget
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: PEAR HTTP_Request class
Disallow: /
User-agent: PHP
Disallow: /
User-agent: Pixray-Seeker
Disallow: /
User-agent: PycURL
Disallow: /
User-agent: python-requests
Disallow: /
User-agent: Python-urllib
Disallow: /
User-agent: rmnbot
Disallow: /
User-agent: Ruby
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SEOENGWorldBot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: sitebot
Disallow: /
User-agent: Squider
Disallow: /
User-agent: ssearch_bot
Disallow: /
User-agent: suchmaschinenoptimierung.de
Disallow: /
User-agent: Synapse
Disallow: /
User-agent: Wget
Disallow: /
User-agent: woopingbot
Disallow: /
User-agent: Wotbox
Disallow: /
User-agent: Xenu Link Sleuth
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: seoprofiler
Disallow: /
User-agent: *
Crawl-delay: 15
|
|
发表于 2021-3-6 14:21:38