Twitter API Consulting
140dev@gmail.com
781-879-2960
@140dev

Streaming API

Source code and tips for streaming API programming

The Twitter streaming API will deliver tweets for any keywords in real-time. Learn how to cache the results in a database and display them online.

Yesterday’s streaming API post described a multiple server model for handling high rate tweet collection. Today I’d like to cover a different architecture that addresses this problem with a single server running multiple databases. Let’s say you want to display tweets for the most active stocks each day. The streaming API lets you collect tweets […]

{ 0 comments }

Now that I’ve upgraded the streaming API framework to make it easier to manage keyword tweet collection, the next step is handling the increased data flow that results from more keywords. One simple solution is to upgrade your server. MySQL loves as much RAM as it can be given, and switching to a solid state […]

{ 0 comments }

My latest set of enhancements to the streaming API framework is moving along nicely towards my goal of making this code a true production level tweet collection system. While I’m waiting for feedback on the new code, I wanted to take a minute to think about where this system can go. I see several possible […]

{ 0 comments }

I wanted to fit one more enhancement into this new version of the streaming API framework. The major limit on performance of a tweet collection system is the number of rows in each table. This is most important during the parsing phase, when lots of insertions are done. MySQL slows down dramatically when it has […]

{ 1 comment }

This set of changes is a significant improvement to parse_tweets.php. To allow you to use this separately from the production version, I’m calling it parse_tweets_keyword.php. You can place it in the same /db directory as the rest of the streaming API framework code. The only changes you will have to make is adding the following […]

{ 0 comments }

Yesterday I introduced the database changes needed to control tweet collection with a table of keywords. Now we can modify get_tweets.php to collect tweets based on the contents of the collection_words table. To avoid confusion with the production version of get_tweets.php, I’m calling this new version get_tweets_keyword.php. This will let you place it in the […]

{ 1 comment }

The previous post had an overview of my planned enhancements to the streaming API framework. The first step is adding some new tables to the 140dev tweet collection database: collection_words The words in this table will be used to collect matching tweets. We’ll see how to add this to the get_tweets.php script so that the […]

{ 0 comments }

The next few posts will describe a set of enhancements to the streaming API framework that will greatly expand the capabilities of the code for collecting tweets based on keywords. I thought I’d start with an overview of what I want to accomplish: Add a collection_keywords table to the database to hold keywords to be […]

{ 0 comments }

Google Hangouts for streaming API code?

by Adam Green on January 31, 2014

in Streaming API

I’d like to start working on a book on the streaming API, but I’ve learned that actually walking people through code and installs is best before writing it all down. I’m learning a lot about where the sticking points are by helping people install the new version of the framework. It seems like running processes […]

{ 0 comments }

I’m a typical programmer when it comes to reading documentation before I try to run someone else’s code. First I try to run the code, then I read the docs. Hey, if I needed clear documentation to get things working, would I have become a Twitter API programmer? Unfortunately, Twitter’s streaming API requires a unique […]

{ 0 comments }

The streaming API is pretty amazing when you start seeing all that free data pouring into a database, but sadly the connection is not completely stable. I typically see a failure at least every couple of months. It’s hard to be sure what causes the failure. It could be the Phirehose library that talks to […]

{ 2 comments }

I’ve been spending the last few days helping people install the latest version of the streaming API framework. This has reminded me of how many moving parts there are, and how this can get in the way of building a mental model of what is actually going on. One of the biggest confusions seems to […]

{ 0 comments }