The streaming API changed its behavior on April 22, breaking all the copies of the Phirehose library I have in use with my 140dev streaming API framework. Thankfully, a patched version of Phirehose was posted within a few hours. If you are running any copies of the 140dev framework, you need to replace your current copy of Phirehose.php with the latest patched version. If you are running the framework code, you will find your current copy of Phirehose.php in the /libraries/phirehose folder.

I already have two sets of open source Twitter API code: the streaming API framework, and my Engagement Programming book. Both of require a fair amount of setup, and the code is very interrelated. This new project will be a set of individual tools that can be run by themselves or called from any application that seems some Twitter integration.

All you’ll need to run these tools will be a common set of library files, which I’ll document in the next post. The tools will need to be able to connect to database and make API calls with OAuth. I’ll package these together in a single zip to make installation easy.

I already some tools planned out, such as getting all followers and posting tweets, but if you have any suggestions, please send them along by tweet or email, or post them here as comments.

The next post has download instructions for the common libraries used by all the tools.

I’ve been getting emails from people asking for advice on making the switch to the 0.30 version of this code. First of all, there is only one database change that you must make before running the new code. You have to add a new is_rt (tinyint) field to the tweets table  to flag retweets. This won’t interfere with existing code, so you can add this field while your existing code is running. The other two changes are removing the entities field from the tweets table, and removing the parsed field from the json_cache table. Neither of these fields are used by the new code. You can safely start running the new code with the entities and parsed fields in place, and then remove them once you have the new code running.

There are a lot of code changes, but the overall structure and flow of the new code is still basically the same. It should be fairly easy to display the old and new versions of the following scripts side-by-side to see what’s been changed. You can then either switch to the new code, or copy the changes back into the old code. Here is a list of the scripts that have changed:

Database module

140dev_config.php. The DB_CONFIG_DIR and CODE_DIR constants have been removed. The AJAX_URL constant has been moved to twitter_display_config.php in the display module. The TIME_ZONE constant has been moved to db_config.php.

db_config.php. The TIME_ZONE constant has been moved here from 140dev_config.php.

get_tweets.php. The paths have been simplified for the require_once() functions, and a test has been added for a missing or malformed id_str property. This has long been a source of error notices. Now tweets without this value are ignored, eliminating these notices from the error log.

parse_tweets.php. Paths are also simplified here. Tweets are now deleted from json_cache when they are parsed. The tweet property of lang is checked for English tweets only. Any language code can be used instead. Tweets are checked to see if they are retweets using the retweeted_status element. If this is found, the entities object is used from the original tweet, and the tweet is flagged as a retweet. When tweets are saved in the db, the entities object is no longer included, and the new is_rt field is set to 0 or 1.

Display module

display_lib.php. This is a new script that has linkify() and twitter_time() functions. The linkify algorithm no longer uses the entities object. It now linkifies tweet text using string substitution.

index.php. This has been renamed as twitter_display.php.

index.html. The request for index.php has been changed to twitter_display.php.

linkify.php. This code has been moved to the display_lib.php script.

twitter_display_config.php. The AJAX_URL constant has been moved here from the 140dev_config.php script in the database module.

Performing the switch over

The exact procedure for making the switch is going to depend on how much you’ve customized the original framework, but this is the process I would use if you have made a lot of changes to the code and database:

1. Make a new copy of the database. In this copy  add the is_rt (tinyint) field to the tweets table, drop the entities field from the tweets table, and drop the parsed field from the json_cached table.
2. Install the beta test copy of the new framework code in a new directory.
3. Incorporate any customizations you have made to the proper scripts in the new copy of the code.
4. Copy your OAuth tokens from 140dev_config.php in the old code to the new copy of this script.
5. Fill in the db account values for db_config.php in the new code.
6. Fill in the AJAX_URL value in the twitter_display_config.php script in the new code.
7. Kill the running copies of get_tweets.php and parse_tweets.php.
8.  Start the new copies of get_tweets.php and parse_tweets.php.

If you have any problems making this conversion, get in touch with me (adam@140dev.com), and I’ll help get the new code running.

Update: This code has now been put into production. You can get the latest copy here.

The last few upgrades of this code were purely for compatibility with internal changes in the streaming API, so I wanted to give people code that didn’t require any changes to their own programming. Those upgrades seem to have gone well for everyone. Now its time to start modifying the code to turn this framework into a production tweet collection system. This will require changes to the db schema and the core code in scripts like get_tweets.php and parse_tweets.php. I also want to rewrite the tweet display module, which has been causing problems for some people.

I have a beta test ready for a new version of both the database server and tweet display modules. They have quite a few changes. I have tested it as much as I can, but I’d like some people to try it out before making a general release. The files contain change logs, but here is an overview of some of the enhancements:

• Tweets from json_cache are deleted after parsing to improve insertion speed.
• The entities object is no longer stored in the tweets table for linkifying tweet text.
• Retweets are flagged in the tweets table.
• The twitter display has been rewritten for several browser issues.

If you can give me a hand by trying out this code, I’d appreciate it. You can download the beta code for version 0.30 of the framework here. Please share any bugs or comments with me (adam@140dev.com)

Twitter is upgrading the streaming API on January, 6th to require http 1.1 connections. This will break the Phirehose library that manages the API connection in my 140dev streaming API framework. If you’re running this framework code, it’s essential that you replace your copy of Phirehose in the next week or tweet collection will stop.

Luckily it’s a really easy upgrade:
1. Download the new Phirehose files. You can get them from the Github repository, or download them from my site.
2. Replace your copy of Phirehose.php and OauthPhirehose.php with the new versions. You’ll find the existing files in /140dev/libraries/phirehose.
3. Kill the running copy of get_tweets.php.
4. Restart get_tweets.php in the background with: nohup php get_tweets.php > /dev/null &

I’m not the author of the Phirehose library for streaming API connections. I just use it as the network layer of my streaming API code. I don’t have the network coding skills to write this library’s code or to fix it, but I can encourage anyone using Phirehose or my 140dev Twitter framework code to test the new version of this library. On January 6th, Twitter plans on switching from http 1.0 to 1.1 for the streaming API. I know what that means conceptually. The http header exchange and error code allow a client and server to talk to each other. That is the depth of my technical knowledge here. The result, if this isn’t fixed in time, is that my streaming API code and anything else using Phirehose will stop running.

The good thing is that once Phirehose is patched for this change, all you have to do is download a new version of the library into your app’s directory. Nothing in your code should change. Darren Cook has now written this Phirehose http 1.1 patch and needs people to help test it. Please add your findings to this thread on the forum.

Twitter posted a notice today about the need for new SSL certificates:

At the end of 2013, all Browsers and Certificate Authorities will no longer support 1024 bits RSA certificates to be compliant to National Institute of Standards and Technology (NIST) guidelines [1].

The SSL certificate currently used on api.twitter.com is signed with the older Verisign G2 root CA certificate.

Due to NIST guidelines, api.twitter.com will change to a new certificate on Dec 10th, 2013. The new certificate will be signed with VeriSign Class 3 Secure Server CA – G3, which has the 2048 bits key length needed to meet recommended security levels.

This means that all HTTP clients used by your application must trust the new root certificate, otherwise you won’t be able to connect in the API. To ensure proper SSL certificate verification across all of Twitter’s services, your software should include all Verisign Root Certificates in its CAFile or other respective keystore. The root certificates are available at the following link:
* Verisign (https://www.symantec.com/page.jsp?id=roots)

I have to admit here that I don’t really know what an SSL certificate is. I know that it is a file that allows SSL to work, but beyond that my perception of this announcement was like Bart Simpson’s dog:

Blah blah blah Dec 10th blah blah everything breaks blah blah.

What I could tell was that I might need to download something and warn everyone using my code to do the same. So I immediately sent queries to the authors of Phirehose and tmhOAuth, the two libraries upon which all my Twitter code depends. The responses were that we are all good, and I could just ignore this cryptic, yet frightening warning.

Fenn Bailey, author of Phirehose, replied:

Phirehose has no visibility of SSL itself but simply uses PHP (and your OS’s) SSL layer. Basically, if PHP will work with these new certificates, Phirehose will.

And Matt Harris, author of tmhOAuth, said:

If you use the included cacert.pem you’ll be using the same SSL CA roots that Mozilla uses as of Sat Dec 29 20:03:40 2012 (the latest version) — so things should be fine.

Hooray! Everything is good. Nothing to worry about. If you were as mystified as me by the SSL certificate announcement, you can relax.

On a personal level, this lack of understanding of the layers beneath the code that I produce is like the perception of other scientific disciplines when I was in college. I was an organic chemistry major, and we used to laugh at the molecular biology majors, because they didn’t have good enough memories to hack Organic Chem. The physical chemists used to laugh at us, because we couldn’t handle the math to understand Pchem. The physicists laughed at the physical chemists, because they couldn’t base all their work on Quantum Theory. Who knows what the mathematicians thought of the physicists.

Modern Web development has so many moving parts and layers in its software stack that you just have to trust that the guys building the parts your code rests on know what they are doing. I just wish that the Twitter folks understood that many of their 3rd party devs are not CS majors, and yet we are able to build lots of cool apps.

We converted the Twitter API Engagement Programming book to Kindle version last week, and it is now available in Amazon stores all over the world. If you don’t own a Kindle, Amazon provides a free reader app for Windows, Mac, and all major mobile devices: iPad, iPhone, Android, etc.
http://www.amazon.com/gp/feature.html?docId=1000493771

This is the first Kindle book I’ve produced, and the process has given me lots of ideas for new projects. I’m going to work on the native iPad format also, but for now the Kindle reader works fine on an iPad.

You can download the kindle version of the book on the main Amazon.com site at:
http://www.amazon.com/dp/B00G3OCA0K

Or you can go to the kindle store in your local Amazon site and search for “Twitter API Engagement”.

The print version of the book will remain on the Amazon.com store only for now, but you can have it shipped from there to anywhere in the world.

I’ve been building custom Twitter engagement systems for clients since 2009, and now I’ve combined all of my favorite techniques and tools into a new book available on Amazon:
amazon.com/dp/0989875806

Just as with my 140dev Twitter Framework code and my ebooks on topics like OAuth, the code in this book uses PHP and MySQL. It shows how to capture tweets, mentions, retweets, tags, URLs, friends and followers. There is a complete report system for sharing this information with clients. A range of automated engagement tools, such as auto-tweeting, automatic DMs, and other forms of automation are included.

This entire engagement system is fully compliant with Twitter’s API version 1.1, and the latest version of the Terms of Service.

I’ve also made all of the source code from the book available for free download under the MIT open source license:
140dev.com/twitter-api-engagement-programming/source-code/