The streaming API changed its behavior on April 22, breaking all the copies of the Phirehose library I have in use with my 140dev streaming API framework. Thankfully, a patched version of Phirehose was posted within a few hours. If you are running any copies of the 140dev framework, you need to replace your current copy of Phirehose.php with the latest patched version. If you are running the framework code, you will find your current copy of Phirehose.php in the /libraries/phirehose folder.

I’ve been spending the last few days helping people install the latest version of the streaming API framework. This has reminded me of how many moving parts there are, and how this can get in the way of building a mental model of what is actually going on. One of the biggest confusions seems to be the idea that I wrote Twitter’s streaming API. Actually, all I’ve done is put a thin layer of code on top of a very deep stack. That code may tie things together, but there are many levels of code that need to be installed and configured. Let’s work our way up from the basic server level:

- Operating system. The streaming API code will run on *nix variants, Windows, and Mac OS X machines. Windows has it own unique quirks, but if you are willing to run a Windows machine as a Web server, you have already discovered that.
- Apache must be installed and configured to run PHP. You should also configure Apache to run PHP within HTML pages. This is not always set by default.
- PHP runs within Apache. You will need version 5.2 or greater. I’ve recently seen problems on Windows servers unless 5.2.17 or greater of PHP is installed.
- cURL is a library that runs within PHP and allows connections to remote servers, such as the Twitter API. You won’t need to call cURL directly in your code, but it is used by the Phirehose and tmhOAuth libraries. cURL should be enabled by default, but some webhosts turn it off.
- MySQL. I try to use version 5.0 or greater.
- The db_lib.php code in the framework uses the mysqli PHP library to communicate with MySQL, so that must be installed within PHP.
- Phirehose is the library that makes the actual connection to the streaming API in get_tweets.php. I didn’t write this, but the author, Fenn Bailey allows me to include it in the framework’s source code. It lives here.
- tmhOAuth is a library that lets you make OAuth calls to Twitter’s REST API, such as searching and reading timelines. It isn’t used by the streaming API framework, but is part of my engagement programming code, and many sample scripts on this site, so I’m including it here, It is written by Matt Harris and lives here.
- Finally we get to my streaming API framework code, which rests on all this work by thousands of other people. Open source is an amazing thing, but finding the right path to an app isn’t easy at first.

Twitter is upgrading the streaming API on January, 6th to require http 1.1 connections. This will break the Phirehose library that manages the API connection in my 140dev streaming API framework. If you’re running this framework code, it’s essential that you replace your copy of Phirehose in the next week or tweet collection will stop.

Luckily it’s a really easy upgrade:
1. Download the new Phirehose files. You can get them from the Github repository, or download them from my site.
2. Replace your copy of Phirehose.php and OauthPhirehose.php with the new versions. You’ll find the existing files in /140dev/libraries/phirehose.
3. Kill the running copy of get_tweets.php.
4. Restart get_tweets.php in the background with: nohup php get_tweets.php > /dev/null &

I’m not the author of the Phirehose library for streaming API connections. I just use it as the network layer of my streaming API code. I don’t have the network coding skills to write this library’s code or to fix it, but I can encourage anyone using Phirehose or my 140dev Twitter framework code to test the new version of this library. On January 6th, Twitter plans on switching from http 1.0 to 1.1 for the streaming API. I know what that means conceptually. The http header exchange and error code allow a client and server to talk to each other. That is the depth of my technical knowledge here. The result, if this isn’t fixed in time, is that my streaming API code and anything else using Phirehose will stop running.

The good thing is that once Phirehose is patched for this change, all you have to do is download a new version of the library into your app’s directory. Nothing in your code should change. Darren Cook has now written this Phirehose http 1.1 patch and needs people to help test it. Please add your findings to this thread on the forum.

Twitter posted a notice today about the need for new SSL certificates:

At the end of 2013, all Browsers and Certificate Authorities will no longer support 1024 bits RSA certificates to be compliant to National Institute of Standards and Technology (NIST) guidelines [1].

The SSL certificate currently used on api.twitter.com is signed with the older Verisign G2 root CA certificate.

Due to NIST guidelines, api.twitter.com will change to a new certificate on Dec 10th, 2013. The new certificate will be signed with VeriSign Class 3 Secure Server CA – G3, which has the 2048 bits key length needed to meet recommended security levels.

This means that all HTTP clients used by your application must trust the new root certificate, otherwise you won’t be able to connect in the API. To ensure proper SSL certificate verification across all of Twitter’s services, your software should include all Verisign Root Certificates in its CAFile or other respective keystore. The root certificates are available at the following link:
* Verisign (https://www.symantec.com/page.jsp?id=roots)

I have to admit here that I don’t really know what an SSL certificate is. I know that it is a file that allows SSL to work, but beyond that my perception of this announcement was like Bart Simpson’s dog:

Blah blah blah Dec 10th blah blah everything breaks blah blah.

What I could tell was that I might need to download something and warn everyone using my code to do the same. So I immediately sent queries to the authors of Phirehose and tmhOAuth, the two libraries upon which all my Twitter code depends. The responses were that we are all good, and I could just ignore this cryptic, yet frightening warning.

Fenn Bailey, author of Phirehose, replied:

Phirehose has no visibility of SSL itself but simply uses PHP (and your OS’s) SSL layer. Basically, if PHP will work with these new certificates, Phirehose will.

And Matt Harris, author of tmhOAuth, said:

If you use the included cacert.pem you’ll be using the same SSL CA roots that Mozilla uses as of Sat Dec 29 20:03:40 2012 (the latest version) — so things should be fine.

Hooray! Everything is good. Nothing to worry about. If you were as mystified as me by the SSL certificate announcement, you can relax.

On a personal level, this lack of understanding of the layers beneath the code that I produce is like the perception of other scientific disciplines when I was in college. I was an organic chemistry major, and we used to laugh at the molecular biology majors, because they didn’t have good enough memories to hack Organic Chem. The physical chemists used to laugh at us, because we couldn’t handle the math to understand Pchem. The physicists laughed at the physical chemists, because they couldn’t base all their work on Quantum Theory. Who knows what the mathematicians thought of the physicists.

Modern Web development has so many moving parts and layers in its software stack that you just have to trust that the guys building the parts your code rests on know what they are doing. I just wish that the Twitter folks understood that many of their 3rd party devs are not CS majors, and yet we are able to build lots of cool apps.

The 2012 candidate system is going to need a more fault tolerant collection system. Since the Egyptian protest started Twitter has been even more unreliable, and this is affecting the connection to the streaming API. I use a monitoring system to warn me when tweet collection has stopped, but it hasn’t been necessary in the past to automatically restart the connection. I saw some communication about this on the Phirehose mailing list. I need to run the collection script and capture the process id. Then kill the process with the same id and restart it when no new tweets are received for 10 minutes.

Update: This was pretty easy. I can add this code to the collection script to have it write it’s process id to a text file when it starts:
$fp = fopen('process_id.txt','w');
fwrite($fp,getmypid());
fclose($fp);

Then I can modify the monitoring script to kill the current collection script when no tweets are added for 10 minutes, and then restart it:
$process_id = file_get_contents('process_id.txt');
exec('kill -9 ' . $process_id);
exec('nohup php get_tweets.php > /dev/null &');