Twitter API Consulting
140dev@gmail.com
781-879-2960
@140dev

Streaming API

Source code and tips for streaming API programming

The Twitter streaming API will deliver tweets for any keywords in real-time. Learn how to cache the results in a database and display them online.

One of my primary goals for this codebase has always been pedagogical. I thought of it as a set of training wheels for the streaming API. The number of moving parts is so great with the streaming API that getting data flowing from the API to a database using just the official docs is a […]

{ 0 comments }

I’ve been getting emails from people asking for advice on making the switch to the 0.30 version of this code. First of all, there is only one database change that you must make before running the new code. You have to add a new is_rt (tinyint) field to the tweets table  to flag retweets. This […]

{ 0 comments }

Update: This code has now been put into production. You can get the latest copy here. The last few upgrades of this code were purely for compatibility with internal changes in the streaming API, so I wanted to give people code that didn’t require any changes to their own programming. Those upgrades seem to have […]

{ 0 comments }

Twitter is upgrading the streaming API on January, 6th to require http 1.1 connections. This will break the Phirehose library that manages the API connection in my 140dev streaming API framework. If you’re running this framework code, it’s essential that you replace your copy of Phirehose in the next week or tweet collection will stop. […]

{ 1 comment }

I’m not the author of the Phirehose library for streaming API connections. I just use it as the network layer of my streaming API code. I don’t have the network coding skills to write this library’s code or to fix it, but I can encourage anyone using Phirehose or my 140dev Twitter framework code to […]

{ 0 comments }

When I built the 140dev framework 3 years ago, my goal was to provide a set of training wheels. Coders who were new to the API could get a complete working system in miniature. My emphasis was on simplicity, certainly not production level functionality. Since the Twitter IPO and the $200M acquisition of Topsy, collecting […]

{ 0 comments }

Twitter has announced that they will be turning off basic authentication for the Streaming API on May 7th. I’m preparing a new version of the 140dev Framework to use OAuth now, and will have it posted tomorrow. That will be good for new users of this code, but if you already have the code in […]

This week we got crushed by the State of the Union speech. We normally get about 30,000 to 50,000 tweets per day in the 2012twit.com database, and our largest server can handle that without any showing any appreciable load. During the SOTU tweet volume exploded. We got 500,000 tweets in about 4 hours. I was […]

The 2012 candidate system is going to need a more fault tolerant collection system. Since the Egyptian protest started Twitter has been even more unreliable, and this is affecting the connection to the streaming API. I use a monitoring system to warn me when tweet collection has stopped, but it hasn’t been necessary in the […]

I’m collecting all the tweets for possible 2012 candidates with the Streaming API, and I wanted to make sure I was getting every one of their tweets. I built a backfilling script to go through every tweet in each of these accounts, and add any that weren’t already in the database. This uses the /statuses/user_timeline […]

Since the announcement of the Twitter-Gnip partnership, there have been lots of news stories and blog posts stating that this is the end of the independent developer, because there is no more free Twitter data. This is completely wrong. You can get all the Twitter data you need, as long as you don’t want *all* […]