For the past couple of weeks, I'd noticed that my site was randomly crashing due to MySQL failing on the server. In case any of you saw it, this is what was causing the "Error establishing connection to database" page (which I will customize for the future when I have a spare moment.) This seemed incredibly odd to me because my site is very low traffic, so it shouldn't be having issues like this. I had recently installed the Jetpack plugin for WordPress, and shortly after that, I started noticing all of my problems. As a result, I mistakenly blamed the plugin for my woes and immediately removed it. The problems persisted, however, which led me to do some research. At first glance, many blog authors and commenters suggested simply increasing the memory on the server. Considering that 30 hits in a single day is quite an accomplishment for me at this time, I didn't think that was the case, and I wasn't about to start paying more on my server bill if it wasn't completely necessary.
Luckily for me, DigitalOcean has a fantastic support community where people can ask and answer questions related to everything a SysAdmin would need to know, and this post in particular hit the nail on the head for me. To summarize, I was being brute-force attacked, which means that someone was repeatedly trying to guess my password for my account on here, and that was overloading my server with unnecessary requests. Like the article says, I was able to determine that this was the cause of the problem by checking out the log file, and boy was I surprised:
/var/log/apache2/access.log
191.96.249.13 - - [14/May/2016:22:46:13 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:13 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:13 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:13 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:14 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:14 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:15 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:17 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:17 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:19 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:19 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:19 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:19 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:20 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:20 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:21 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:22 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:23 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:24 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:24 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:25 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:26 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:26 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:26 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:26 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:28 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:28 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:29 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:30 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:31 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:31 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:31 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:32 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:32 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:33 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:34 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:34 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:34 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:35 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:36 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:37 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:37 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:37 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:38 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.9 - - [14/May/2016:22:46:38 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:39 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.13 - - [14/May/2016:22:46:39 -0400] "POST /xmlrpc.php HTTP/1.0" 500 585 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
As you can see from the time stamps, I was getting hit multiple times per second on a particular part of the site that's used to gain access to the administrative parts. I initially followed the instructions from the DigitalOcean support thread, banning the attackers and ensuring that they wouldn't be coming anywhere near my site again, by using the following commands:
iptables -A INPUT -s 191.96.249.9 -j DROP
iptables -A INPUT -s 191.96.249.13 -j DROP
Then I went ahead and saved my iptables rules so that they would persist and added them to the rc.localfile so they'd be called in on reboot every time
iptables-save > /etc/iptables_rules
# /etc/rc.local:
/sbin/iptables-restore < /etc/iptables_rules
I didn't stop there, however. That may have been enough to save me this time, but this certainly won't be the last time I'll be attacked. The next step for me was to go ahead and get ufw working. I then set up a few rules to keep my site a little safer:
ufw enable
ufw allow 22
ufw allow 443
ufw default deny incoming
In case you aren't familiar with these, I'll go over them quickly. The first command simply starts up the firewall, and makes sure that it'll run at boot time. The second command allows port 22, so that I can still use SSH to access my machine. The third command allows port 443, which is the port used to access my website over HTTPS. I left out port 80 (the standard HTTP port) because I don't want people accessing my site insecurely anyways. Lastly, I went ahead and blocked any other types of connections, because they won't be necessary for my purposes here.
My site is still not secure enough for me to be confident in it, but this will do for now. The attacks have ceased and my site should be fully operational once again. I have absolutely no idea why anyone would even want to hack my site, or what they think they could possible gain from it (maybe they just can't wait for me to publish my next article and absolutely MUST read the drafts :P) The next steps for me will be to set up a custom error log for failed WordPress authentications, and link that up to fail2ban, which would then create rules to block out specific problematic IP addresses. Once I get that up and running, I'll be sure to share the details so that we can all enjoy a safer web.
