This is a question I frequently get asked by new clients. They know there is a Twitter API available to collect tweets, but they have no idea how the results differ from just asking for tweets with Search.Twitter.com. I’ve recently explained the fact that a tweet database lets you create a long-term store that cannot be reproduced or purchased any other way. That is just the starting point. The real advantage of Twitter API programming is the way it allows you to add value to a collection of tweets:

• You can apply quality control rules that let you filter out false positives for the keywords you are using in your collection query.
• I also like to apply simple “filth controls” to all tweet streams that get displayed on sites. This starts with a list of George Carlin’s 7 words you can’t say on television, and grows into a list of the more creative racist and misogynist words so popular on Twitter. Excluding tweets with these words makes Twitter seem much more civilized.
• A simple language detection algorithm will let you tweets for a specific language and exclude all other languages.
• By checking the tweets you receive for spammy words, like free, coupon, buy now, or sale, you can clean out a high percentage of spam tweets, and if you check new tweets for duplicates, you can identify spammers and blacklist them.
• If you screen the user account data for each tweet’s author, you can exclude accounts that have a spammy profile, such as a default avatar, no followers, or an account that has only been in existence a few days.
• Or you can come up with an influence algorithm, such as follower count or frequency of mentions, to select tweets from the most influential users.

These are just the generic ways to add value to a tweet aggregation site. Once you start working with a client with specific application needs, there are many ways to add value to Twitter. This is an iterative process that keeps improving the quality of your tweet collection.

So the simple answer to the question is that Twitter programming produces much higher quality results than Twitter search.

• YO! this guy has been spreading some really badnasty rumors about u… bit.ly/KTsPo
• Hello some person is posting very bad things about you… bit.ly/KYxvJ
• lol…OMG I’m laughing so hard at this pic of u someone found bit.ly/LIMvO

I now get about 20 to 30 of these Direct Messages from people I follow on Twitter every day, and I know I am not alone. These DMs are not just an annoyance, like normal Twitter spam. The difference is that they come from Twitter accounts I have followed, and as a result they are making it impossible for me to trust any DM I get, even if I know the source. I believe that these DMs will eventually destroy Twitter if they aren’t brought under control. Is that too extreme? Think about what Twitter is made up of. It’s really a simple model. There are relationships with other users, messages sharing ideas, and links in those messages that you click on. These DM spams are destroying all 3 of these aspects of Twitter.

If I can’t trust any URL in any DM, then I won’t click them. That eliminates the value of DMs for me and thus for Twitter. If I can’t trust that the DM is actually being sent by the person I followed and not the result of some kind of hack, I no longer believe that I actually have a relationship with that person. So this DM scam is breaking trust, relationships, and the ability to click on links. What other value does Twitter have?

I’ve been tracking the accounts that send these DMs, and they never seem to repeat. When I look at their behaviors, tweets, friends, followers, etc., they look like legitimate accounts. They do not seem to be pure spambot accounts. So why are they sending this nonsense? The only possible answer is that the DMs are being sent by one or more rogue apps. These are Twitter applications that have been authorized by users, that are then taking advantage of this authorization to send DMs without the user’s permission or knowledge. That is the only possibility. The cost of creating and maintaining all these accounts just to send the DMs would be prohibitive. This has to be the work of bad apps. So the other critical part of Twitter, the one I earn my living from, is being destroyed. Once users recognize that rogue apps are out there, they will stop signing into any apps.

Trust, social relationships, the value of messages received through Twitter, the ability to click on a link sent through Twitter, and the ability to sign up and use a third party Twitter app, all destroyed. Am I being too extreme in saying that this could be the end of Twitter?

The amazing thing is that the problem could be solved in a few minutes by Twitter. All they have to do is include the name of any app that sends a DM inside the DM email. They already publish the app source in tweets. You can easily see this in clients like Tweetdeck. If they did the same thing with DMs, we could all see which apps were sending these DMs, and everyone would revoke their authorization for these apps. Twitter could be notified, and kill the bad apps. Problem solved. Why won’t Twitter make this simple change?