Magento 2 Optimization Speed Up Magento by Blocking Spam Bots
Each time a user visits your Magento site, a small percentage of your server resources are used to handle the request. If enough users visit your site at the same time, your server will run out of resources and either the customer will be unable to access your site or some key service on your server will crash (this is often MySQL but can be any of the key services your Magento website relies on) and your user will see a nasty - and confidence destroying - error message.
The above scenario is true when the user is a valid customer who is looking to purchase your products but it's also true when the user is another computer, a spam bot simply scanning your entire site for forms that it can post spam to. Each time the spam bot requests a page on your site, server resources are used up. These server resources could be used to serve a valid customer but instead are being wasted on malicious spam bots. What makes this problem even worse is that spam bots don't load the JS on your page, which means tracking agents like Google Analytics don't show these bots. The only way to see these bots would be to analyse your web server access logs.
How to stop invisible spam bots?
There are 2 ways that you can easily stop this problem. The first way is to decrease the amount of resources the server uses for each request. This means that each spam bot takes up less of your server resources. The best way to do this is to use a Magento full page cache extension as this will load a cached version of the page, which will use a lot less resources. An even better way is to block all spam bots from accessing your site, but how?
Blocking Spam Bots with NoBots
When I first noticed this problem on a clients' site, I was shocked at the amount of bots crawling the site. Based on my research, I found over 1,000 bots crawling the entire site each month. This client was using a cloud based server where they paid based on their bandwidth consumption. This means that each bot is slowing down their server and costing them money. To help this client, I created NoBots, a spam blocking Magento extension. NoBots works in multiple inventive ways to protect your forms from spam and identify and block spam bots. As soon as NoBots is installed, spam bots will start to be blocked. You can view the progress of NoBots in the Magento Admin, where you can see a simply list of all bots that have been blocked.
With NoBots installed, any spam you're recieving from forms (default Magento forms and custom forms) will be blocked immediately. I have seen people getting 100's of spam emails a day through their Magento site and then this instantly stops the moment NoBots is installed.