The Joyent Community

Does that splain the brief hiccup?

Eugene wrote:

Does that splain the brief hiccup?

You mean the persistent disruptions of my weekend luncheons?

Yes.

All MT. All mt-comments.cgi.

We typically run at a load less than 1 just about everywhere. Despite having millions of emails and millions of web hits every day now.

One.textdrive.com is a relatively expensive server with dual xeons, 6GB hard wired RAM, 6x73GB RAID-arrayed SCSI 10K Seagate Cheetahs and a FreeBSD kernel optimized for running web and database servers. It's also on a 100Mbps switch with two Gigabit ethernet cards. I have a Linux-threaded static build of MySQL and I load that build into 1GB of RAM when the database server loads up. I have a fast static build of Apache in front of that.

The server is designed to be the kind of thing that could do 20,000,000 requets a day without a problem. In many ways no one uses servers like these for shared hosting.

If you were to lease one.textdrive.com's hardware and it's network connection, if you would cost you about $12,000+ a year. If you were to contract someone like myself to build it and get it up and running, it would cost you about $10,000. If you were to hire someone like me to watch over it, it would cost you well into the six figures a year.

So that said.

MT-comment.cgi, it's inherent nature and the fact that it's targeted so can push this server to loads of nearly 300 (of course, once mt is killed, all returns to normal which is a fine testiment to the setup).

And why is there still spam on http://www.sixapart.com/corner/archives … erie.shtml ? They cleaned up the assault from the 2nd and 3rd, but the 11th is a fresh new day.

How does Six Apart handle the deluge of comment spam and referer spam on typepad and its servers? I mean, aren't they likely to be hit worse than TD with this cr*p?

TypePad = 100% MT
TextDrive = 5-10% MT? (Just a guess.)

Oh and sorry about your lunch.

Last edited by Eugene (2004-12-12 00:26:52)

Eugene wrote:

Oh and sorry about your lunch.

Thanks but this has been a good solid 8 weeks of this, and it keep tracing back to the same culprit: an old non-threaded perl cgi script that's writing to mysql and doesnt' seem to know how to let go under load.

That's fugly.

compooter wrote:

The worst part is that everyone just keeps saying a collective "WTF", because MT's so popular you can't just pull the plug.

Sure I can. I can't let a few sites be responsible for all of my web servers downtime can I?

Pulling the plug on comments sounds pretty good too right now. I think maybe we need a "switch" campaign similar to the Apple and Firefox campaigns. There are free, opensource alternatives out there like Txp and WP which don't fail as miserably as MT, and can do an equally good job, all for free.

... and if Daring Fireball can live without comments, what site REALLY needs them, apart from those trying-to-be-a-forum blogs like whitespace.

Does the mt-close plugin reduce this traffic? It certainly has stopped my comment spam (although I am sure they are still trying to leave their crap on my blog).

Perhaps a server-wide cron job that ran mt-close on all MT installs might suit? Part of the TxD prerequisites: if you want to run MT, you need to close off old comments...

Do spammers even bother about the form? I'm guessing they go straight for mt-comments.cgi.

First of all, I don't recall hearing about hosts cutting off MT until the past month or so. It seems there has been a real escalation that has brought us (and other hosts) to this point.

Secondly, I'd bet the reason you only have 4 out of ~120 MT users getting hammered is because those 4 have the highest Page Rank (he said, pointing the finger back at Google) ... likely a 6/10 or better.

Thirdly, if Six Apart can't at least emit a hint they are working on this, users have to take the hint from their host. If your app is creating out-of-control demand levels and causing you to regularly get shut down (as well as regularly degrade the performance of the server you are on for all who are on it), and there appears to be no new solution on the horizon, you need to find a new alternative.

Finally, those of you who are poo-poo-ing MT over this also need to realize that when the spammers are done with them (for whatever reason), they will move on to targeting whatever has become the new predominant app, via whatever flaws it might have. And it may be yours.

So removing MT from the equation does not solve the problem, it would just mute the current symptoms. Jason, is this the kind of attack that could be minimized by some server level mod_security, once we get the proper keyword/URL filters in place?

Last edited by reid (2004-12-12 13:24:14)

my biggest concern, which reid also touched on, is the lack of information from sixapart on this. i checked the news on their sites yesterday and there was no mention of the issue. that almost bothers me more than the issue.

daniel wrote:

my biggest concern, which reid also touched on, is the lack of information from sixapart on this. i checked the news on their sites yesterday and there was no mention of the issue. that almost bothers me more than the issue.

Yeah, you should try grabbing them by AIM.

jason wrote:

It does leave out the sick-o's that make spamming profitable but getting rid of people who buy viagra from goat-love sites is well ... also not really doable.

Someone needs to write mod_sucker for Apache.

Reid - a fantastic article.

reid wrote:

Finally, those of you who are poo-poo-ing MT over this also need to realize that when the spammers are done with them (for whatever reason), they will move on to targeting whatever has become the new predominant app, via whatever flaws it might have. And it may be yours.

Very true, but like you said, the open-source nature of the alternatives provide an advantage that Six Apart lacks.

I'm wondering if it'd be out of line to require MT users to rename mt-comments.cgi (and yes I do realize that this would not come close to solving the problem entirely). If we're going to use drastic measures such as shutting down the script, why not just require its renaming?

gotcha

Anil from Six Apart has just left a comment on my article with a bit of new info. There's apparently a new version of MT-Blacklist in the works, and an announcement may come soon.

reid wrote:

There's apparently a new version of MT-Blacklist in the works, and an announcement may come soon.

How noncommittal and non-informative of them :)

justin french wrote:

Can something be done in a htaccess or httpd.conf to stem the flow? I'm thinking something like "If my comments.cgi file is being requested more than X times per second, refuse service for a minute" or something like that...

This. That. Is what I want.

I want comment and trackback spam stopped at a level above me. I'm a user, I have the technical chops of Deputy Dawg on a good day... I'm not a developer and I'm not a coder. I'm a user. I want tools that work and I feel entitled to them because I pay for hosting, donate to projects I care about and will test damn near anything. I do what I can.

Google isn't going to change. SixApart isn't going to change and the WordPress guys can't even publicly announce a bug fix. TXP may have been overlooked, so far, but it's only due to a lack of adoptation. I have no doubt that as soon as WP, TXP or anything else reaches the market penetration that MT enjoys it will be the exact same thing.

Jason and Ryan seem to be getting some numbers gathered and researching this more than any other host I've come across. That's a great start -- you guys know that if you could come up with some server-side voodoo to make this all go away it would be huge. Huge like Google. Huge like MT. Nobody else is addressing comment spam at a server level, they all want us to install the latest and greatest plugins or upgrade to the new (sometimes more expensive) release. That's not fixing it. That's not helping me, the clueless guy who wants comments from his mother to work. That's not helping the client I just sold on the power of weblogs (who suddenly is spending, or hiring someone to spend, hours a day managing a inherently unsound system).

Look for the magic bullet... it's there, and if you find it we WILL come. Boy will we come, in droves. Meanwhile, I'm firing up a text editor and deleting databases. It's reached the point where I can write markup by hand quicker than I can delete spam and manage my spam managing plugins. It's just not worth the effort anymore.

Sorry about the rant, but I have too much time invested in these tools to go down quietly.