For the last couple of years we’ve been helping clients with a wide range of Twitter engagement tools. We’ve helped non-profits, political campaigns, and businesses identify and engage with the best Twitter accounts to help spread their message. Now we’ve committed our efforts to a free site that helps progressives unite for political action. UniteBlue.com combines a great set of Twitter accounts on the left with the same type of engagement tools we’ve perfected in our client work. Our first press release on the site has the details.

This week we got crushed by the State of the Union speech. We normally get about 30,000 to 50,000 tweets per day in the 2012twit.com database, and our largest server can handle that without any showing any appreciable load. During the SOTU tweet volume exploded. We got 500,000 tweets in about 4 hours. I was able to keep the server going by shutting down some processes that weren’t needed, but it was a challenge. This issue of bursts of tweets seems to be getting worse. In the case of Twitter and politics people are getting used to talking back to the TV through Twitter. With 9 months left until the election I needed to find some solutions.

I spent a lot of time over the last 2 days trying to find the problem, and discovered that it was not parsing the tweets that was killing us, but inserting the raw tweet data into the json_cache table. I use a two phase processing system with the raw tweet delivered by the streaming API getting inserted as fast as possible in a cache table, and then a separate parsing phase breaking it out into a normalized schema. You can get the basic code for this as open source.

It looks like Twitter has been steadily increasing the size of the basic payload that it sends for each tweet in the streaming API. That makes sense, since people are demanding more data. Yesterday they announced some insane scheme where every tweet will include data about countries that don’t want tweets with specific words to be displayed. This will only get worse.

I realized that I have never actually needed to go back and reparse the contents of json_cache, and I had long ago added purging code to my 2012twit system to delete anything in that table older than 7 days. I tried clearing out the json_cache table on my server and modifying the code to delete each tweet as soon as it was parsed. This cut the size from several hundred thousand rows on average in this table to about 50. The load on that server dropped right away and during the GOP debate last night, the load stayed very low.

Our work with Twitter and politics has now moved beyond the 2012 election. We just set up a tweet collection database to track the Occupy Wall Street movement. It uses the 140dev framework to collect all tweets containing #ows, #occupy, and #occupywalllstreet.

This will be used to document our tools and methodology to automate a Twitter engagement campaign. In this case we’ll be building up a new political account we started called @4more. You can follow this work on our 140elect.com blog.

Update, 1/21/12: The tweet flow for #OWS has dwindled down to a point where it isn’t worth tracking, so we’ve turned this system off for now. Hopefully the Occupy movement will become active again in the spring.

A lot of my Twitter consulting for clients has involved automating the process of Twitter engagement. Now I’m finally documenting this as part of the Twitter politics work I’m doing with my son. We are building up a political account called @4more, and writing up the whole process on our 140elect.com blog.

I’ve been quiet here for the last couple of months as I’ve worked on a new Twitter dashboard for the 2012 presidential election. We launched it last night at http://2012twit.com, and I’m very pleased with the result. This will be a lot of fun. I’ll be blogging here about development issues related to the site.

The next big step will be adding full Twitter client capability to the site: account login, tweeting, following, etc. I want to build an open source module for these features that I can add to the 140dev framework.

The 2012 candidate system is going to need a more fault tolerant collection system. Since the Egyptian protest started Twitter has been even more unreliable, and this is affecting the connection to the streaming API. I use a monitoring system to warn me when tweet collection has stopped, but it hasn’t been necessary in the past to automatically restart the connection. I saw some communication about this on the Phirehose mailing list. I need to run the collection script and capture the process id. Then kill the process with the same id and restart it when no new tweets are received for 10 minutes.

Update: This was pretty easy. I can add this code to the collection script to have it write it’s process id to a text file when it starts:
$fp = fopen('process_id.txt','w');
fwrite($fp,getmypid());
fclose($fp);

Then I can modify the monitoring script to kill the current collection script when no tweets are added for 10 minutes, and then restart it:
$process_id = file_get_contents('process_id.txt');
exec('kill -9 ' . $process_id);
exec('nohup php get_tweets.php > /dev/null &');

I’m collecting all the tweets for possible 2012 candidates with the Streaming API, and I wanted to make sure I was getting every one of their tweets. I built a backfilling script to go through every tweet in each of these accounts, and add any that weren’t already in the database. This uses the /statuses/user_timeline call to get the past tweets. I ran into a problem with tons of 502 errors from the Twitter API, as many as one every two or three API calls.

Taylor Singletary on the Dev mailing list suggested dropping the count parameter to avoid timeout errors, and this has helped a lot. I was using a count of 200 tweets per call to keep the number of calls low. This gave me all the data in about 100 calls, but with the errors I wasn’t able to complete the process before hitting the rate limit. I tried dropping the count to 100, and this allowed the script to finish with a total of 298 calls.

So now I have the catch 22 of needing to do more API calls to avoid the errors that cause too many API calls. The only solution I see is to cut the count parameter to a level that is low enough to avoid errors, and then spread the backfilling out over multiple hours to stay within the rate limit.

I think the ultimate solution is to do a steady level of backfilling spread over the entire day. I haven’t had to do backfilling in the past, because I was treating the Streaming API tweet collection as a high volume sampling mechanism. As long as I got lots of tweets on a particular subject, it was good. Now that I want to maintain a database of every tweet made by the candidates I have to backfill to make sure nothing was missed by streaming. This seems to be necessary, since every time I run the backfill I get two to three tweets that didn’t get sent by streaming.

I’ve built a tweet aggregation database for the 2012 election, and I’m using the streaming API to collect all tweets for the possible candidates. I want to go a lot further than just collecting their tweets, though. I think there will be all kinds of fascinating data to mine from the way these Twitter accounts develop over the next two years. How fast are they gathering followers, what tags do they use, how many people mention them? Right now I’m doing basic research, but I think this could justify an entire site.

Two groups on Twitter that have been ignoring each other are about to collide, and it won’t be pretty when they do. Politicians from world Presidents to American Congressmen have embraced Twitter over the last year, and now hacktivists are using Twitter to manage and promote the Wikileaks cyberwar. They are operating side by side, and it is only a matter of time that the hacktivists turn on the politicians. Twitter is known as being easy to hack, with false Tsunami reports in Indonesia as a recent example.

What happens when the hacktivists turn on the politicians directly and threaten to continually disrupt and take over the accounts of world leaders? What will the world do? Will the FCC be given control over Twitter, since it is a US company? Will UN resolutions try to control it? The WikiWar is fundamentally about communication and its use as a political tool/weapon. This is the logical progression from tweeting during the Iraq elections and using it to manage student protests in the UK. The bomb throwers and terrorists are next, and there are highly vulnerable targets on Twitter.

Politicians overreact when threatened. Their next threat will come from Twitter. This is going to get ugly.

I’m saving this tweet from Russian President Medvedev so I can point to it the next time I read an article pontificating on what Twitter should become. Twitter doesn’t have to become anything, it already is. Twitter is a tool that lets the President of Russia cut through any filters and communicate with anyone he wants in a form that can be passed on to anyone else. I think that’s enough.