The idea is floating around that Twitter will allow users to edit their tweets, possibly for a limited period of time. I agree with the desire for this feature, but it won’t work. Once a tweet is published, copies of it are delivered in real-time to thousands of data collection scripts capturing tweets with the streaming API. These copies are then put into thousands of databases, in some cases with the goal of permanent storage. It is impossible for Twitter to force changes to these stored copies of tweets once are delivered by the API.

This problem is apparent right now when tweets are deleted. Old copies remain all over the Web. In theory, the Twitter Terms of Service requires developers to remove deleted tweets from their own tweet storage. The streaming API even sends out a signal to alert developers of deleted tweets, so they can be removed. In practice, this issue is ignored, and deleted tweets remain permanently available at sites like Topsy.com.

Keeping track of changes due to editing tweets will be even more difficult. The edit may be allowed on Twitter.com, but then a mechanism must be added to the streaming API to notify all consumers of this stream that an edit has been made. Even if API users wanted to update their own databases to reflect the edits, pushing through these changes to an entire tweet collection system will require major rewrites. The contents of each tweet such as tags, @mentions and URLs are spread throughout complex data structures with the expectation that they are written once and never change. I see it as highly unlikely that these rewrites will be made. So unlikely that it just won’t happen as a general rule.

Twitter can try to allow edits to tweets, but the results will just be a mess, with multiple copies of each tweet popping up from different sources.

I have a bad habit acquired from my years as a Dot Com CTO. When the time comes to pick a server for a new project, I always overbuy. I’d rather pay a hundred dollars more per month then have a server that can’t take the load. One of the driving forces behind this decision is the time it takes to migrate to a more powerful server, if I discover that it is needed. So I have a collection of dedicated servers that I lease from various webhosts that collectively waste about $500 a month. That’s a small percentage when considered as a cost of doing business across all my clients, but it is still wasteful.

I’ve thought about moving to Amazon’s AWS system, but every time I look at the docs I get turned off. I’m an application builder, not a professional sysadmin. I have no problem managing a Linux based server, but when it comes to configuring and optimizing Apache and MySQL, I turn to professionals. The AWS docs make it clear that this system is built by people who LOVE tweaking servers. They also seem to love really detailed command driven operations spanning several lines, with very complex parameter names, with very odd capitalization. I couldn’t care less about that. If I could just say, “Create a new server instance, and make it this big.” I’d be thrilled.

That is what I have now found with Rackspace.com. Their cloud servers let me clone multiple server instances, and upsize or downsize them with a menu. I’m testing this for a client who wants to collect tweets that have a lot of flow. His search terms for the Twitter streaming API retrieve about 60,000 tweets an hour. If I had to lease a dedicated server for this, I would have spent at least $150 to $200 a month to be sure I could handle the load. Instead I had my sysadmin create the cheapest server instance at Rackspace, at $11 per month. The entire pricing structure is here.

Once the basic server configuration with all of my code was set up, I made a server image with Rackspace’s control panel that could be used to create a new server instance in minutes without having to pay my sysadmin again. I ran the tweet collection for a few hours, and found that this server size was too small. The server load went up above 3.0, and queries were completely stalled. All I had to do was ask for the next size server ($22 a month) using the menu, and 10 minutes later I was up again with the new configuration. This ran much better for inserting tweets, but queries were still too slow. So after a few hours of watching the server, I decided to bump it up again to the next size at $44. This configuration looks like it will work. Server load is about 0.5, and the queries we need to run complete in a few seconds.

Overall this has been a great experience. I love the idea of being able to size up gradually until I see the server handling a real world load, and then downsizing if that load drops. I’m going to start moving some of my existing sites across to Rackspace next with the hope of saving at least $300 to $400 a month. Then I’l be experimenting with clusters of different sized servers to handle more complex site requirements.

This week we got crushed by the State of the Union speech. We normally get about 30,000 to 50,000 tweets per day in the 2012twit.com database, and our largest server can handle that without any showing any appreciable load. During the SOTU tweet volume exploded. We got 500,000 tweets in about 4 hours. I was able to keep the server going by shutting down some processes that weren’t needed, but it was a challenge. This issue of bursts of tweets seems to be getting worse. In the case of Twitter and politics people are getting used to talking back to the TV through Twitter. With 9 months left until the election I needed to find some solutions.

I spent a lot of time over the last 2 days trying to find the problem, and discovered that it was not parsing the tweets that was killing us, but inserting the raw tweet data into the json_cache table. I use a two phase processing system with the raw tweet delivered by the streaming API getting inserted as fast as possible in a cache table, and then a separate parsing phase breaking it out into a normalized schema. You can get the basic code for this as open source.

It looks like Twitter has been steadily increasing the size of the basic payload that it sends for each tweet in the streaming API. That makes sense, since people are demanding more data. Yesterday they announced some insane scheme where every tweet will include data about countries that don’t want tweets with specific words to be displayed. This will only get worse.

I realized that I have never actually needed to go back and reparse the contents of json_cache, and I had long ago added purging code to my 2012twit system to delete anything in that table older than 7 days. I tried clearing out the json_cache table on my server and modifying the code to delete each tweet as soon as it was parsed. This cut the size from several hundred thousand rows on average in this table to about 50. The load on that server dropped right away and during the GOP debate last night, the load stayed very low.

We’ve been working on a college football recruiting site called DirectSnap.com for a couple of months, and the most interesting aspect of the technology behind this site is the quality control algorithm I had to develop. Most of the tweet streams we work on, such as 2012twit.com, are based on collecting tweets for either a set of screen names or real names that are distinctive, such as politicians. When you find a match for Newt Gingrich or Mitt Romney, you can be fairly sure you have the right person.

In the case of DirectSnap, the tweet collection is based on the first and last name of 250 high school football players. Right away I knew I would have a problem when I found Michael Moore in the list of potential recruits. Randy Johnson was going to be even trickier, since the baseball player with this name was likely to be tweeted about by the same sports fans as the football recruit we were tracking. Identifying college teams is also tricky. For example, the word ‘Florida’ in a tweet with a player’s name could refer to the University of Florida or Florida State University.

The solution I came up with was creating a list of exclusion keywords for each player and team. If a tweet contains ‘Michael Moore’, but it also has words like fat, hypocrite, film, or liberal, it probably is not about the football player. A tweet with a player’s name is assigned to the University of Florida if it contains ‘Florida’, but not ‘Florida State’. This first level of screening did a good job of filtering out false positives, such as the wrong Michael Moore, but we wanted to curate the tweets automatically to select the highest quality. The goal was to end up with a tweet stream that was much more interesting than what you could get with Twitter’s search.

To do this we added a set of high quality words to the quality screen, like the team position or hometown name of each player. We found that tweets with this extra information was generally from users who were serious about reporting details, not just random fans chanting a player’s name repeatedly. We used these quality words in two ways. Each time a quality word was found in a tweet, 1 point was added to a quality score for the tweet and for the user who sent the tweet. This allows us to select tweets for display that have a minimum quality score, and that are from a user with a minimum quality score.

To see how well this system works, try comparing the DirectSnap page for Michael Moore, and Twitter search for the same words. My experience is that users find false positives very upsetting. They think computers actually understand what they are searching for, and when they see a false positive, the reaction is always that the website is “stupid”. My favorite example of this is when people complain about Google Alerts for their own name returning blog posts or tweets they have written. The reaction is usually “How stupid can Google be? Doesn’t it know that I don’t want to be alerted about my own writing?” On the other hand, they never seem to be upset about missing results. So ending up with a subset of all possible matches, but with no visible false positives is always the best goal.

Our work with Twitter and politics has now moved beyond the 2012 election. We just set up a tweet collection database to track the Occupy Wall Street movement. It uses the 140dev framework to collect all tweets containing #ows, #occupy, and #occupywalllstreet.

This will be used to document our tools and methodology to automate a Twitter engagement campaign. In this case we’ll be building up a new political account we started called @4more. You can follow this work on our 140elect.com blog.

Update, 1/21/12: The tweet flow for #OWS has dwindled down to a point where it isn’t worth tracking, so we’ve turned this system off for now. Hopefully the Occupy movement will become active again in the spring.

The thing I find so great about doing Twitter consulting is the wide range of vertical applications that are completely open for new development. One of them is higher level education. We’ve just launched a new site called Social Studies that applies the techniques of tweet aggregation to help students study collectively.

This work was done under contract with a Colgate University freshman student, Peter McGrath, who has a vision for the next generation response to Facebook. Peter grew up with social networks, and now wants to use his experience to add a social layer to college study. I think this may be an example of Web 3.0 in its earliest stages.

This first version of Social Studies is a tweet database that lets you view tweets based on colleges and individual classes, but the next phase of development will add a point system and game dynamics to make the whole experience a lot more fun.

Do you have a vision of the next great application for Twitter? We can help you bring it to life.

Part of the sales process for Twitter consulting is convincing a new client that Twitter is more than just another way to broadcast their message. You have to show them that what appears to be a random stream of tweets is really a collection of highly qualified sales prospects. By aggregating Twitter users as well as their Tweets, you can extract a great set of sales leads along with their contact info. One way to quickly demonstrate the value of tweet aggregation is to deliver an Excel spreadsheet of sales prospects that meet the client’s needs.

When you aggregate tweets from the Twitter streaming API, it also returns the complete account profile for each user. You can data mine this collection of users to extract highly targeted lists of users, along with their geographical location and home page URL.

The free 140dev Twitter framework is an example of the code you will need to do the tweet aggregation. The schema for the MySQL database it creates shows you it has a table for all the aggregated tweets, which links to the list of tweeting users. Since all of this data is collected for a specific set of keywords, you can then extract personal details on the users who tweet these keywords the most with a simple SQL statement:

SELECT count(*) AS cnt, users.screen_name, users.name, users.location, users.url
FROM tweets, users
WHERE tweets.user_id = users.user_id
AND users.location != ''
AND users.url != ''
GROUP BY tweets.user_id
ORDER BY cnt DESC
LIMIT 1000

The 140dev framework’s example database collects tweets for the keyword “recipe”, so this query gives us the most active tweeters in the food world. Here are the results in phpMyAdmin:

You can then export the results from phpMyAdmin to an Excel spreadsheet, and email it to your client. This gives them solid data in a familiar form. Twitter doesn’t deliver email addresses, and doesn’t even collect phone numbers, but you do get each user’s home page URL. This can be used to gather other contact info, a task that is easily farmed out to people on freelance sites like Mechanical Turk.

So the next time you want to convince a client that Twitter is not just a bunch of kids talking to each other, you can just create a tweet aggregation database for the client’s industry keywords, let it collect data for a few days, and pull out a list of targeted users.

The first day has gone well. I announced the code on the Twitter dev list, and got 16 visitors to the site. The good thing is that the average pages per visitor was 7, and they spent an average of 11 minutes on the site. So people who get to the code are giving it a good amount of attention.

I’m also using this code as an starting point to teach my son Web programming. He has taken a course in Java in school, and has experience setting up WordPress blogs, but hasn’t done any PHP or server based coding. Working through the install process for 140dev with him was very informative. I’m going to rewrite the install page for the Twitter Database Server based on his feedback.

The biggest problem is identifying the target audience. Is it people who have never used Telnet or worked at a Unix-style prompt? Is it someone who has coded in PHP for a while, but has never used the Twitter API? One solution is to produce tutorials and programming primers to help bridge this gap.

I also want to make the install process much simpler. That is the obvious blocking point. If I can make the install on the database server module easy, the rest of the framework will be a breeze.

Overall, I’m happy with the first day’s results.

I just opened up access to version 0.10 of my free Twitter source code library. At first it just has modules for a tweet aggregation database and a tweet display plugin. My next module will be a WordPress display plugin. I also have a lot of tutorials planned that can use this source code as examples.

I’ve been cleaning up the code the last few days. I finally got around to switching to tweet entities. They’ve been around for a while, but every time people complained about a problem Twitter HQ said that developers should code for their occasional disappearance. I don’t have time to code for a major data component being there part of the time. The complaints have died down, so I guess it is stable now. I don’t disagree with the Twitter model of getting improvements out early, but I stay away from the bleeding edge.

One of the nice things about entities is that they include the disambiguated versions of shortened URLs. That saves a ton of processing time for each user of the API, and even more bandwidth for the target of popular URLs. Unfortunately, they only contain the original value of URLs that are shortened by Twitter itself through its t.co domain.

I’m also making sure that every piece of text is pulled out into config files. I want this code to work for any language.

My son is coming back from school for the weekend, and I want to review it with him. He’s my target user. He knows a little PHP and Javascript. If I can make this system easily installable for him, then it will be ready to make public.