屏蔽AI蜘蛛和防止网站文章采集方法

cocojock

屏蔽AI蜘蛛和防止网站文章采集方法,宝塔防火墙设置屏蔽AI爬虫（我用的是破解版宝塔，免费版不知道能不能设置）

1. Amazonbot
   
2. ClaudeBot
   
3. PetalBot
   
4. gptbot
   
5. Ahrefs
   
6. Semrush
   
7. Imagesift
   
8. Teoma
   
9. ia_archiver
   
10. twiceler
    
11. MSNBot
    
12. Scrubby
    
13. Robozilla
    
14. Gigabot
    
15. yahoo-mmcrawler
    
16. yahoo-blogs/v3.9
    
17. psbot
    
18. Scrapy
    
19. SemrushBot
    
20. AhrefsBot
    
21. Applebot
    
22. AspiegelBot
    
23. DotBot
    
24. DataForSeoBot
    
25. java
    
26. MJ12bot
    
27. python
    
28. seo
    
29. Censys

复制代码

1. #禁止Scrapy等工具的抓取
   
2. if ($http_user_agent ~* (Scrapy|Curl|HttpClient|crawl|curb|git|Wtrace)) {
   
3. 
   
4.      return 403;
   
5. 
   
6. }
   
7. 
   
8. #禁止指定UA及UA为空的访问
   
9. if ($http_user_agent ~* "CheckMarkNetwork|Synapse|Nimbostratus-Bot|Dark|scraper|LMAO|Hakai|Gemini|Wappalyzer|masscan|crawler4j|Mappy|Center|eright|aiohttp|MauiBot|Crawler|researchscan|Dispatch|AlphaBot|Census|ips-agent|NetcraftSurveyAgent|ToutiaoSpider|EasyHttp|Iframely|sysscan|fasthttp|muhstik|DeuSu|mstshash|HTTP_Request|ExtLinksBot|package|SafeDNSBot|CPython|SiteExplorer|SSH|MegaIndex|BUbiNG|CCBot|NetTrack|Digincore|aiHitBot|SurdotlyBot|null|SemrushBot|Test|Copied|ltx71|Nmap|DotBot|AdsBot|InetURL|Pcore-HTTP|PocketParser|Wotbox|newspaper|DnyzBot|redback|PiplBot|SMTBot|WinHTTP|Auto Spider 1.0|GrabNet|TurnitinBot|Go-Ahead-Got-It|Download Demon|Go!Zilla|GetWeb!|GetRight|libwww-perl|Cliqzbot|MailChimp|SMTBot|Dataprovider|XoviBot|linkdexbot|SeznamBot|Qwantify|spbot|evc-batch|zgrab|Go-http-client|FeedDemon|Jullo|Feedly|YandexBot|oBot|FlightDeckReports|Linguee Bot|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|EasouSpider|LinkpadBot|Ezooms ) {
   
10. 
    
11.      return 403;
    
12. 
    
13. }
    
14. 
    
15. #禁止非GET|HEAD|POST方式的抓取
    
16. if ($request_method !~ ^(GET|HEAD|POST)$) {
    
17. 
    
18.     return 403;
    
19. 
    
20. }

复制代码

　　设置完了可以用模拟爬去来看看有没有误伤了好蜘蛛，以上屏蔽的蜘蛛名不包括以下常见的6大蜘蛛名：百度蜘蛛：Baiduspider谷歌蜘蛛：Googlebot必应蜘蛛：bingbot搜狗蜘蛛：Sogou web spider360蜘蛛：360Spider神马蜘蛛：YisouSpider爬虫常见的User-Agent如下：

1. FeedDemon       内容采集
   
2. BOT/0.1 (BOT for JCE) sql注入
   
3. CrawlDaddy      sql注入
   
4. Java         内容采集
   
5. Jullo         内容采集
   
6. Feedly        内容采集
   
7. UniversalFeedParser  内容采集
   
8. ApacheBench      cc攻击器
   
9. Swiftbot       无用爬虫
   
10. YandexBot       无用爬虫
    
11. AhrefsBot       无用爬虫
    
12. jikeSpider      无用爬虫
    
13. MJ12bot        无用爬虫
    
14. ZmEu phpmyadmin    漏洞扫描
    
15. WinHttp        采集cc攻击
    
16. EasouSpider      无用爬虫
    
17. HttpClient      tcp攻击
    
18. Microsoft URL Control 扫描
    
19. YYSpider       无用爬虫
    
20. jaunty        wordpress爆破扫描器
    
21. oBot         无用爬虫
    
22. Python-urllib     内容采集
    
23. Indy Library     扫描
    
24. FlightDeckReports Bot 无用爬虫
    
25. Linguee Bot      无用爬虫

复制代码