Of Spam, Robots, and Captchas

[DzM]

Today's update: In less than a week (Nov 17 through Nov 22) the changes to the spam filtering system have automagically blocked 3027 spam messages with two semi-false positives (meaning two messages were blocked that were not spam but they would have been removed anyway). This is 673 spam messages a day with a 0.0007% failure rate. As far as I'm concerned this is a pretty fantastic success. So - welcome to the new normal, gang!

It's worth repeating that the spam filters are stringently applied to Guest posts and the posts of freshly registered users. Users that have more than a small number (well less than 10) posts are considered to have proven themselves members of the community and their messages are not run through filters.

[Mike from Boston]

I am a bit worried that you posted this on Thanksgiving when most people are in a turkey induced coma. Not saying you need a life DzM but statistical analysis on a holiday?? In any case, I am thankful for the work you do keeping this site running.

[DzM]

I am a bit worried that you posted this on Thanksgiving when most people are in a turkey induced coma. Not saying you need a life DzM but statistical analysis on a holiday??

I worry for anyone that is in a turkey induced coma by 11:05am (the local time when I posted that update). Thanks for your concern though.

[NewJerseyRich]

I am a bit worried that you posted this on Thanksgiving when most people are in a turkey induced coma. Not saying you need a life DzM but statistical analysis on a holiday??

I worry for anyone that is in a turkey induced coma by 11:05am (the local time when I posted that update). Thanks for your concern though.

true we see it in east coast time.

[DzM]

Spam Trivia: Since the new filters have been put in place 22,732 spam messages have been blocked. There are some number of false positives in there, but so few that I can't find them (the signal is very much drowned out by the huge volume of noise). Zuzana and I haven't had to do anything at all (emptying moderation queues, etc), so this is a Big Fricken Win.

FYI there.

[DzM]

Since Nov 17, 2012 there have been 93,123 messages clocked as spam. It's possible some of them are legitimate messages, so sorry about that if one is yours. But hey - That's ~93,000 opportunities for Generic Viagra and Air Jordans you haven't had to see. Success!

FYI there.

[lesh]

But...but...I actually *do* need a new pair of sunglasses...

[Mike from Boston]

I have also heard Generic Viagra isn't bad. If things go wrong, you only get blurry vision in one eye and a two hour erection...

[DzM]

Since Nov 17, 2012 there have been 93,123 messages clocked as spam. It's possible some of them are legitimate messages, so sorry about that if one is yours. But hey - That's ~93,000 opportunities for Generic Viagra and Air Jordans you haven't had to see. Success!

FYI there.

In the interest of More Exciting Spam Trivia:

Since Nov 17 of 2012 (a year and a week ago at the time of this writing) 533,567 minutes have passed. In those 533,567 minutes the filter system has blocked 218,374 messages as spam (I couldn't begin to say how many are false positives - I just don't have the patience to skim page after page of that crap to see if any I can spot a valid message). I'm sure you've all noticed a number that have been missed as false negatives too (sometimes as many as three-to-five per day). Anyway - this spam rate boils down to:

~ 0.41 spam messages blocked per minute
~ 30 spam messages blocked per hour
~ 720 spam messages blocked per day

etc

What's all this mean? Not much, really. I find this stuff interesting and suspect some of you might too.

Keep on rawking, spammers. You're making the Interwebs a better place.

[DzM]

So once upon a time one of Medusa's spam defenses was building an epic black list of spammers and proxy servers that spammers use and arbitrarily ignoring them (meaning when those addresses try to access Medusa nothing happens. They just get ... nothing. At all. Not even an error message.) About two years ago I changed something that broke this black hole, but because it's not a great line of defense I didn't spend much effort trying to fix it. Today, however, I had another reason to look into the problem and this means that today I fixed it.

So what's this mean to you? Hopefully nothing. It COULD mean that if your IP address is one of 4423 currently ignored addresses, though, then you will get nothing at all when you try to access pogues.com. Of course this also means that you won't see this message and will probably just assume that pogues.com has gone to the graveyard of dead domains. And of course that's not true! So for any of you that are accidentally caught up in this black hole and who think the web site has died - sorry about that. If you let me know you're having a problem then I'll fix it for you.

For the rest of you - Medusa rejects about 250 spam messages a day. Over the last week or so one or two of them have been leaking through the filters (you may have noticed some posts with really helpful info about Search Engine Optimization or some Manager's Big Band, or some such crap). Hopefully this will slow the number of both public and hidden spam messages Medusa gets. The net impact to all of you good folk is that the one or two messages per day you've been seeing should hopefully fall to one or two a week. Or maybe fewer. Fingers crossed.

As always, please let me know if something that used to work seems no longer to do so.

[Low D]

As always, please let me know if something that used to work seems no longer to do so.

Given how many of us are over 40 years old, that's very generous of you, DzM! I've got these two weird bumps on my legs...

[DzM]

As always, please let me know if something that used to work seems no longer to do so.

Given how many of us are over 40 years old, that's very generous of you, DzM! I've got these two weird bumps on my legs...

Those are called kneecaps. A hammer will help with them. Just keep hitting until the lumps go away.

[firehazard]

As always, please let me know if something that used to work seems no longer to do so.

Given how many of us are over 40 years old, that's very generous of you, DzM! I've got these two weird bumps on my legs...

Those are called kneecaps. A hammer will help with them. Just keep hitting until the lumps go away.

Thanks, DzM. Though given that remedy, I'm having second thoughts about asking your advice about certain other things that, with age, no longer seem to function as vigorously as they used to...

[DzM]

Just a small status update on this. Since I've resurrected this method of hitting spammers roughly 8000 systems that have posted spam to Medusa have been added to a blocking filter. That seems to be working well so far. I've also found a way to identify some spammers by the behavior of their broken spambots. I doubt a real person actually sees the results of their posting efforts bt if anyone DOES watch the results they'll now be treated to some of the Internet's finer porn. I hope they enjoy those penises.

As far as judging the efficacy is concerned I have a limited sample set so have to take this with a bit of skepticism. Prior to this change Medusa was receiving 250+ spam posts per day (of which 0.5-1% were slipping through to public view). Since these changes the volume is down tremendously. For example, in the last 24 hours 16 spam messages have been posted and captured by the spam detection system. So far I think 1 spam message in the last week has made it through to public view.

Barring evidence that this filter system is breaking things really badly for real users this system will be staying in place. For anyone caught up accidentally, I'm sorry.

[NewJerseyRich]

just above this topic some Adidas ads snuck under the fence!