One of my servers has been compromised. Sound the alarms. Rally the troops. This shit's about to get ugly!

Earlier this week I woke up to numerous customers emailing me to let me know that they could no longer send or receive mail. Since mail is the crutch of most business' I quickly sprang into action.

What I found out was not pretty. Usually this kind of thing is the result of a PHP or CGI form. We've all seen that before, people just posting to them directly with 1000 emails in the TO field. Unfortunately, it didn't look like this was the case. This time around a legit user account was being used.

The first thing I did was find the IP address in question and started a tcpdump for all traffic on port 25. I captured a connection and thought I was off to the races. Not being an expert with this, I ran into a problem though. Since all of the traffic was Chinese, none of the text I got back was English. It wasn't even convert-able as far as I could tell. Here's a sample of the text I got back from tcpdump.

c@?I??^C^@H^@^@^@H^@^@^@^@^[^L?X?^@^S /?|^H^@E^@^@:.^?@^@@
^F{?K}????j:^@^Y^M0p?zŤ??|P^X^V?^T^T^@^@334 UGFzc3dvcmQ6^M

Say... what? I Google'd the crap out of this and couldn't find anything so I moved on. It occurred to me why not just block these bastards at a firewall level? After investigating this a little further some other people talked about how you can use iptables rate limiting and simply automatically block SMTP connections if a single IP tries to connect more than "x" times per minute. Clean, easy and totally effective.

    1 iptables -I INPUT -p tcp --dport 25 -i eth0 -m state --state NEW -m recent --set
    2 iptables -I INPUT -p tcp --dport 25 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 2 -j LOG --log-prefix "RATELIMIT SMTP: "
    3 iptables -I INPUT -p tcp --dport 25 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 2 -j DROP

The first rule sets netfilter to track new connections on port 25, the second rule is optional, this logs attempts that exceed the threshold, and the third line drops any new connections over 2 within 60 seconds.

What happens if a legit user tries to connect more than twice in a minute? Well, they'll get blocked and I'll likely soften the rule in a couple of days but for now, I'd rather deal with 1 customer than have everyone get affected.

As I mentioned in an earlier post, I've had to spent a great deal of time getting a system in place that was capable of scaling out to millions of requests per day on (preferably) cheap vanilla hardware. After days and days of doing nothing but code optimizations it became clear that I needed to do more than just that. The real trick came when I decided to switch everything over to nginx (I'll save a post for that later) but another thing I decided to do was switch the API box over to use VMware Server. There's a bunch of reasons why I thought this was a good idea so I thought I'd share.

Aside from the fact that I work with an enthusiastic VMware guy who was helpful in some of the implementation, it occurred to me that I'd like a system that was easily scaled across new machines as I needed it to be. I didn't want to have to worry about things like physical hardware since it really doesn't matter to me. All I should have to worry about is my sites staying online and capable of serving the number of requests I was getting.

Some advantages to the VMware Server setup are:

1. A re-usable machine image that can deployed on any host OS/hardware I have available.
2. Easily backed up
3. Easily upgradeable
4. Easily moved from one location to another

As it so happened, The Planet was having a pretty incredible server special that I would have been stupid not to move forward with so the latest incarnation of the API was born powered by VMware Server no less.

The turn around from getting VMware installed and running was under and hour (that included installing the VM). I was configuring and running an instance of the API within 2 hours. Since then, the machine has been stopped for backups and some regular maintenance. I haven't had to touch a thing. If I decide I need a more powerful box tomorrow, I will stop the VM, copy the image to the new machine and start it up. That's it.

If anyone reading this is in the position where you know it's likely you'll be upgrading, I would seriously give this method a shot. It's a great feeling to know that you can handle scaling/upgrades with (basically) no down time.

Here at work I use multiple tools during the day to get my job done. HelpSpot, Jira, our internal admin etc... Something I've found to be more and more of a problem is, as I will call it here, lazy searching.

If you sit down and think about it you shouldn't be very surprised. We're all so used to having Google or Spotlight just "work" that we've been getting to be more and more lazy with our searching techniques.

I don't mean to pick on certain tools but it's pretty much unanimous here at the office that HelpSpot's search is the shits. You simply cannot find what you're looking for easily by entering text in a single text field. You can even stretch this further (I'm sorry Ben, but I have to mention this) that we're relying on Google to fix things like spelling mistakes.

With Jira, it's just a simple design problem. The search works but there's too many options. I am not saying there isn't the need for them in certain cases, but having a simple single search field and filling it in should still work.

I do ultimately blame Google. They are making it so easy that we as users, forget that making things easy to find is hard. I don't think it should change of course, it's just annoying when you're using products that don't work the way I expect.

Just a quick thank you to all of you who donated to TMDb last week when I put out the call for help. The amount collected will pay for almost 3 full months of the server fees and you guys have truly blown me away.

What I think I will do is send out a donation call 4 times a year. If you've already donated, please don't feel pressured into doing so. My goal is try and get all of the new members we sign up involved like you guys did and hopefully TMDb can become self sustaining. That is really the only goal I have ever had for it, it's never been my intention to try and make money, just cover my costs and I'm a happy man.

Again, thanks so much and here's to another awesome 3 months!

I am happy to report incredible growth in these few short months with TMDb being online. We went from being just a hobbyist side project catering to a few media center's, to being a full on, hundreds-of-users-contributing movie database. Words cannot express how exciting this has been.

The downside to this rapid growth (and lack of funding) is that one, I still have to pay for all of this myself and two, almost all of my dev time has been sucked into making our architecture scale rather even have time to think about fixing/adding new features.

To those of you who are programmers you know how time consuming doing this properly can can be. The bigger, better infrastructure that I am building around TMDb will allow us to have API mirrors and serve data from multiple locations. For anyone wondering, this is why API keys have been slow to move out. The new infrastructure will also let us serve somewhere around ~450 requests/sec on commodity "vanilla" hardware so growing the API without having to have a farm of servers will be possible.

How many requests have we been doing? Well, January saw 23 million requests through the website and API. I'd estimate the API to represent somewhere in the neighborhood of 22 million so you can understand why this is the priority.

Where do I see TMDb going this year?

I'd love to see TMDb become a site that not only us media center, movie collectors have heard of and to be honest we're on our way. Having said that, the UI needs a fair bit of love and these infrastructure improvements have to be completed. With those 2 BIG TICKET items finished, I think we could be in a place where we're truly rocking it.

To everyone who has supported this site, and everyone who has dedicated their free time to making TMDb relevant, I sincerely thank you.

Here's to another year of rocking it!