Twitter has announced that they will be turning off basic authentication for the Streaming API on May 7th. I’m preparing a new version of the 140dev Framework to use OAuth now, and will have it posted tomorrow. That will be good for new users of this code, but if you already have the code in [...]
Streaming API
Source code and tips for streaming API programming
The Twitter streaming API will deliver tweets for any keywords in real-time. Learn how to cache the results in a database and display them online.
This week we got crushed by the State of the Union speech. We normally get about 30,000 to 50,000 tweets per day in the 2012twit.com database, and our largest server can handle that without any showing any appreciable load. During the SOTU tweet volume exploded. We got 500,000 tweets in about 4 hours. I was [...]
The 2012 candidate system is going to need a more fault tolerant collection system. Since the Egyptian protest started Twitter has been even more unreliable, and this is affecting the connection to the streaming API. I use a monitoring system to warn me when tweet collection has stopped, but it hasn’t been necessary in the [...]
I’m collecting all the tweets for possible 2012 candidates with the Streaming API, and I wanted to make sure I was getting every one of their tweets. I built a backfilling script to go through every tweet in each of these accounts, and add any that weren’t already in the database. This uses the /statuses/user_timeline [...]
Since the announcement of the Twitter-Gnip partnership, there have been lots of news stories and blog posts stating that this is the end of the independent developer, because there is no more free Twitter data. This is completely wrong. You can get all the Twitter data you need, as long as you don’t want *all* [...]
The first day has gone well. I announced the code on the Twitter dev list, and got 16 visitors to the site. The good thing is that the average pages per visitor was 7, and they spent an average of 11 minutes on the site. So people who get to the code are giving it [...]
I’ve been cleaning up the code the last few days. I finally got around to switching to tweet entities. They’ve been around for a while, but every time people complained about a problem Twitter HQ said that developers should code for their occasional disappearance. I don’t have time to code for a major data component [...]
I just finished a tutorial on the two methods of searching for tweets. Whenever this subject comes up on the Twitter developers mailing list, the usual response is that the streaming API is best, but that depends on your goals and programming ability. If you want to search for tweets in the past, or if [...]
Along with providing the source code for the Twitter engineering tweet database, I also want to use it as a model for explaining the proper database and code architecture for aggregating tweets. I’ve got my first tutorial up now on the benefits of decoupling your website Twitter app from the Twitter API.
To build the Twitter dev tweeet aggregator I used the follow option with the streaming API, which accepts a list of user ids. I’m doing this with the Phirehose library, so I’m actually using the Phirehose setFollow() method. This is the first time I’ve used the streaming follow for users rather than track for keywords. [...]
Twitter has a really open development process with the devs using their Twitter accounts freely to discuss their work. I’m going to aggregate their accounts and display them on the site. The display app isn’t in place yet, but it will soon appear here. I’ll also publish the list of accounts being followed. Building this [...]
