This post is long overdue and contains three topics:

(1) Expiration times for ivmSIP listings have changed from a fixed amount of days to a dynamically determined amount of time.

(2) We are now using rWhois data to further prune ivmSIP/24 listings. Back in March, we announced the fact that ivmSIP/24 now very often lists subranges instead of the whole /24 block. That has recently been further improved/expanded through the use of rWhois data.

(3) We recently signed up what is now our second largest subscriber. This will stir up the pot somewhat.
 

(1) Expiration times for ivmSIP listings: Previously, ivmSIP listings expired at about the 12 day mark (since the last spam seen from that IP--this was previously a secret--but not anymore since it no longer applies). The problem with that somewhat long and fixed expire time is that (a) snowshoe spammers could still leapfrog it and get a ‘clean slate’ on their next spam run, and, (b) this was unfair to those listed due to a one-time security problem which was quickly fixed. They would sometimes stay listed many days after the spam-sending security problem was already fixed.

This has now changed. The ivmSIP expiration times are dynamic and many IPs automatically expire in < 48 hours! At the same time, in cases where there is a history of frequent spam issues from a single IP spread out over time and/or an IP in a ‘bad neighborhood’, the expiration time is increased dramatically, and in proportion to the evidence/history available. Often, that expiration time is somewhat greater than the original 12 days.

Part of the reason for this change is because we were simply overrun with removal requests from those who had a security problem which caused a short-term and one-time (though typically massive!) spam outburst. The time spent trying to process those requests was driving our business into the ground and far surpassed the labor hours justified by current subscription revenues. But with these now shorter expire times, the volume of removal requests has greatly dropped. This is helping us to keep up with such requests from this point forward... September/09 onward now begins a new era! At the same time, the spammers won't get a free pass. That was key--finding a way to shorten expiration times without giving spammers a free pass (particularly snowshoe spammers who try to leapfrog expiration times)--and this new system is accomplishing that goal very effectively.

(2) RWHOIS data for ivmSIP/24 listings: This past March, we announced that ivmSIP/24 now often lists subranges instead of the entire /24 block. This mostly involved analysis of individual IP reputation to create ‘barriers’ that would generate listings of smaller sub-sections. We have since determined that some legitimate IPs are not going to have enough ‘good reputation’ for that new system to prevent such IPs from getting blacklisted on ivmSIP/24 --specifically-- where that innocent sender is sharing a /24 block with a spammer. The good news is that these types of senders are typically the smallest in volume and importance. Therefore, they often went unnoticed by all invaluement subscribers because those listings still generated extremely few FPs. But they still should not be blacklisted, if at all possible. Hey--being a relatively small business ourselves, we empathize with the ‘little guy’, even if he is (probably unknowingly) paying for hosting with a grayhat or blackhat hoster--or has a totally malformed rDNS, etc!

For that reason, we have now further improved this system whereby rWhois data is factored into ivmSIP/24 listings so that, a majority of the time, only the spammer’s allocated subrange of IPs get listed on ivmSIP/24, leaving the innocent sender’s IPs untouched--even small and seemingly insignificant senders.

However, there are some limits to this system. (a) Many deliberate spammers artificially divide up their /24 block into subsections which are really owned by the same spammer, or spam gang. (b) Many hosters are so blackhat that their /24 blocks are just divided amongst different spammers, with extreme rare innocent senders involved.

Therefore, there are reasonable limitations to how far we will go with this system. Once X number of sub-blocks on a /24 are seen emitting spam, then all bets are off and, at that point, innocent senders have only themselves to blame for choosing such a blackhat hoster. (And, by that time, the entire /24 block is typically outright blocked by Hotmail, Comcast, Yahoo, and several other ISPs--all of whom mentioned do NOT use invaluement data--at least not yet!)

But even when rWhois data does not specify subranges, our March 2009 improvements will often kick in and still keep innocent senders off ivmSIP/24, even if the block is crawling with many ‘de-facto’ subsections used by different spammers.

Admittedly, we ‘put the cart before the horse’. The improvements in March/09 involved the harder methods and many more ‘trade secrets’--and those improvements were critical since many hosters do not maintain subdivided rWhois data for their different clients’ subranges. This more recent improvement was the easier way to keep innocent subranges from ivmSIP/24. But both were mission critical improvements. IOW, many innocent ranges go untouched by ivmSIP/24 (due to the March/09 improvements) even when not distingished from the spammer's subranges by rWhois data.

(3) Our new 2nd Largest Subscriber: We recently signed up what is now our second largest subscriber, 2nd both in terms of their annual payment, as well as their own gross revenues! And they have some noteworthy distinctions:

(a) They are a large regional ‘household name’ U.S. ISP with several million mailboxes to protect. Therefore, they are easily our largest mostly-residential-mailboxes subscriber.

(b) They (tentatively) plan to (eventually) outright block on ivmSIP.

Many in anti-spam circles have lamented that, while the invaluement lists are blocking much spam missed by other lists, the invaluement lists have no ‘teeth’ because no extremely large ISP is using them in a significant way. But, obviously, with this new large residential-focused ISP on-board, that is about to change in a big way.

Prior to these improvements, our false positives were already extremely low. But every incremental step closer to perfection is very difficult to achieve, but much needed when an ISP with millions of mailboxes uses a DNSBL as a significant factor in their spam filtering. Therefore, it was the preparation for this large ISP which inspired the hundreds of hours of work (over the summer of '09) which went into the improvements described earlier in this post, amongst other unmentioned improvements.
 

Given that the stakes are now higher, these improvements were mission critical. And now that all of this intensive programming is behind us, we will be turning our attention to faster handling of removal requests and deeper auditing of our data to try to move even another notch closer to perfection.