We’ve been experiencing some higher levels of bot traffic on Xanga lately, which have been keeping us very busy.  Bot traffic can cause the servers to get slower (or worse, to be inaccessible), so we wanted to share an update on what's going on and what we've been doing to address it.

1. LOTS OF BOTS

There are a growing number of "web crawler" bots that are trying to crawl our site, and sites in general.  As Wikipedia puts it, “a Web crawler is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing.”

It used to be that only large search engines like Google or Bing could afford to buy lots of servers and storage, and crawl websites.  But the rates for servers/storage have come down a lot lately, and we’ve noticed a big increase in bots and web crawlers.  Social media is also a big contributor: every time someone pins a post, Pinterest sends a bot to crawl the page being pinned.  With all the pinning going on these days, Pinterest is sending a lot of bots out there into the world!

As a result of all this, bot traffic is way up on the web and it’s been putting a strain on our servers.  We should be fine for now - and when bot traffic gets especially bad, we tweak our network settings to prevent the bots from overloading the servers.

Server maintenance is proving to be a lot of work though, so we’re looking find other ways to address the surge in bot traffic.

2. DDOS

We experience Denial-of-Service attacks from time to time as well.  (“In computing, a denial-of-service (DoS) or distributed denial-of-service (DDoS) attack is an attempt to make a machine or network resource unavailable to its intended users.” - Wikipedia)

Recently we were the inadvertent target of a DDoS attack on our servers.  Our webhost had a bunch of random servers compromised by a security vulnerability known as Shellshock. Our servers were not compromised (we had patched them immediately after the security vulnerability came to light), but some hackers used the other compromised servers to attack another set of servers on our web host.

Even though we weren’t directly attacked, the compromised servers caused significant congestion on our webhost’s network - and that congestion caused our servers to be unable to reach the Internet. So Xanga went down for about 2 hours.

By the time we had figured out what was going on, the webhost had resolved the issue on their end.  And with some server tweaks on our end, we were able to once again get out servers connected to the Internet.  Our webhost has made changes to make sure that this won't happen again, and we will continue to monitor for server vulnerabilities so we can keep our servers safe.

———

Just wanted to share a little bit about the sort of server maintenance we’ve been doing a lot of lately.  Next up, we’re pulling together our thoughts on some of the top feature requests, starting with how we might upgrade our customizable theme system.