cho phép googlebot truy cập vào trang web bằng csf

Question

Chào mọi người,

Hiện tại mình dùng csf để chặn IP của nướ mỹ (US) đồng nghĩa chặn luôn googlebot. vậy có cách nào cho phép googlebot truy cập vào trang web không? mình add IP nhưng ko ăn thua. mình dùng csf làm firewall.

thanks

Answer 1

Chào bạn đây là range list ip bạn có thể tham khảo:

Alexa Bot IP Addresses
204.236.235.245, 75.101.186.145

Baidu Bot IP Addresses
119.63.196.0/24, 115.239.212./24, 119.63.199.0/24, 122.81.208.0/22, 123.125.71.0/24, 180.76.4.0/24, 180.76.5.0/24, 180.76.6.0/24, 185.10.104.0/24, 220.181.108.0/24, 220.181.51.0/24, 123.125.67.144/29, 123.125.67.152/31, 123.125.68.68/30, 123.125.68.72/29, 123.125.68.80/28, 123.125.68.96/30, 202.46.48.0/20, 123.125.68.80/30, 123.125.68.84/31

Bing Bot IP Addresses
65.52.104.0/24, 65.52.108.0/22, 65.55.24.0/24, 65.55.52.0/24, 65.55.55.0/24, 65.55.213.0/24, 65.55.217.0/24, 131.253.24.0/22, 131.253.46.0/23, 157.55.16.0/23, 157.55.18.0/24, 157.55.32.0/22, 157.55.36.0/24, 157.55.48.0/24, 157.55.109.0/24, 157.55.110.40/29, 157.55.110.48/28, 157.56.92.0/24, 157.56.93.0/24, 157.56.94.0/23, 157.56.229.0/24, 199.30.16.0/24, 207.46.12.0/23, 207.46.192.0/24, 207.46.195.0/24, 207.46.199.0/24, 207.46.204.0/24

Duckduck Bot IP Addresses
46.51.197.88, 46.51.197.89, 50.18.192.250, 50.18.192.251, 107.21.1.61, 176.34.131.233, 176.34.135.167, 184.72.106.52, 184.72.115.86

Facebook Bot IP Addresses
31.13.97.0/24, 31.13.99.0/24, 31.13.100.0/24, 66.220.144.0/20, 69.63.189.0/24, 69.63.190.0/24, 69.171.224.0/20, 69.171.240.0/21, 69.171.248.0/24, 173.252.73.0/24, 173.252.74.0/24, 173.252.77.0/24, 173.252.100.0/22, 173.252.104.0/21, 173.252.112.0/24, 2a03:2880:10::/48, 2a03:2880:11::/48, 2a03:2880:20::/48, 2a03:2880:2020::/48, 2a03:2880:2050::/48, 2a03:2880:2110::/48, 2a03:2880:2130::/48, 2a03:2880:3010::/48

Google Bot IP Addresses
203.208.60.0/24, 66.249.64.0/19, 72.14.199.0/24, 209.85.238.0/24, 2001:4860:4801:2::/64, 2001:4860:4801:5::/64, 2001:4860:4801:6::/64, 2001:4860:4801:2001::/64, 2001:4860:4801:2002::/64

Sogou Bot IP Addresses
220.181.125.0/24, 123.126.51.64/27, 123.126.51.96/28, 123.126.68.25, 61.135.189.74, 61.135.189.75

Yahoo Bot IP Addresses
67.195.37.0/24, 67.195.50.0/24, 67.195.110.0/24, 67.195.111.0/24, 67.195.112.0/23, 67.195.114.0/24, 67.195.115.0/24, 68.180.224.0/24, 72.30.132.0/24, 72.30.142.0/24, 72.30.161.0/24, 72.30.196.0/24, 72.30.198.0/24, 74.6.8.0/24, 74.6.13.0/24, 74.6.17.0/24, 74.6.18.0/24, 74.6.22.0/24, 74.6.27.0/24, 98.137.72.0/24, 98.137.206.0/24, 98.137.207.0/24, 98.139.168.0/24, 114.111.95.0/24, 124.83.159.0/24, 124.83.179.0/24, 124.83.223.0/24, 183.79.63.0/24, 183.79.92.0/24, 203.216.255.0/24, 211.14.11.0/24

Yandex Bot IP Addresses
37.9.115.0/24, 37.140.165.0/24, 77.88.22.0/25, 77.88.29.0/24, 77.88.31.0/24, 77.88.59.0/24, 84.201.146.0/24, 84.201.148.0/24, 84.201.149.0/24, 87.250.243.0/24, 87.250.253.0/24, 93.158.147.0/24, 93.158.148.0/24, 93.158.151.0/24, 93.158.153.0/32, 95.108.128.0/24, 95.108.138.0/24, 95.108.150.0/23, 95.108.158.0/24, 95.108.156.0/24, 95.108.188.128/25, 95.108.234.0/24, 95.108.248.0/24, 100.43.80.0/24, 130.193.62.0/24, 141.8.153.0/24, 178.154.165.0/24, 178.154.166.128/25, 178.154.173.29, 178.154.200.158, 178.154.202.0/24, 178.154.205.0/24, 178.154.239.0/24, 178.154.243.0/24, 37.9.84.253, 199.21.99.99, 178.154.162.29, 178.154.203.251, 178.154.211.250, 95.108.246.252

Youdao Bot IP Addresses
61.135.249.200/29, 61.135.249.208/28

Answer 2

chào bạn,
bạn mở vi /etc/csf/csf.rignore

bỏ rào (#)  .googlebot.com restart lại csf là ok

Answer 3

bạn có thể allow list danh sách sau đây

• 216.239.32.0/19 # Googlebot
• 64.233.160.0/19 # Googlebot
• 72.14.192.0/18 # Googlebot
• 209.85.128.0/17 # Googlebot
• 66.102.0.0/20 # Googlebot
• 74.125.0.0/16 # Googlebot
• 66.249.64.0/19 #Googlebot