Adam Green
Twitter API Consultant

Conversion strategy for the new streaming API framework

by Adam Green on January 14, 2014

in 140dev Source Code,Announcements,Streaming API

I’ve been getting emails from people asking for advice on making the switch to the 0.30 version of this code. First of all, there is only one database change that you must make before running the new code. You have to add a new is_rt (tinyint) field to the tweets table  to flag retweets. This won’t interfere with existing code, so you can add this field while your existing code is running. The other two changes are removing the entities field from the tweets table, and removing the parsed field from the json_cache table. Neither of these fields are used by the new code. You can safely start running the new code with the entities and parsed fields in place, and then remove them once you have the new code running.

There are a lot of code changes, but the overall structure and flow of the new code is still basically the same. It should be fairly easy to display the old and new versions of the following scripts side-by-side to see what’s been changed. You can then either switch to the new code, or copy the changes back into the old code. Here is a list of the scripts that have changed:

Database module

140dev_config.php. The DB_CONFIG_DIR and CODE_DIR constants have been removed. The AJAX_URL constant has been moved to twitter_display_config.php in the display module. The TIME_ZONE constant has been moved to db_config.php.

db_config.php. The TIME_ZONE constant has been moved here from 140dev_config.php.

get_tweets.php. The paths have been simplified for the require_once() functions, and a test has been added for a missing or malformed id_str property. This has long been a source of error notices. Now tweets without this value are ignored, eliminating these notices from the error log.

parse_tweets.php. Paths are also simplified here. Tweets are now deleted from json_cache when they are parsed. The tweet property of lang is checked for English tweets only. Any language code can be used instead. Tweets are checked to see if they are retweets using the retweeted_status element. If this is found, the entities object is used from the original tweet, and the tweet is flagged as a retweet. When tweets are saved in the db, the entities object is no longer included, and the new is_rt field is set to 0 or 1.

Display module

display_lib.php. This is a new script that has linkify() and twitter_time() functions. The linkify algorithm no longer uses the entities object. It now linkifies tweet text using string substitution.

index.php. This has been renamed as twitter_display.php.

index.html. The request for index.php has been changed to twitter_display.php.

linkify.php. This code has been moved to the display_lib.php script.

twitter_display_config.php. The AJAX_URL constant has been moved here from the 140dev_config.php script in the database module.

Performing the switch over

The exact procedure for making the switch is going to depend on how much you’ve customized the original framework, but this is the process I would use if you have made a lot of changes to the code and database:

  1. Make a new copy of the database. In this copy  add the is_rt (tinyint) field to the tweets table, drop the entities field from the tweets table, and drop the parsed field from the json_cached table.
  2. Install the beta test copy of the new framework code in a new directory.
  3. Incorporate any customizations you have made to the proper scripts in the new copy of the code.
  4. Copy your OAuth tokens from 140dev_config.php in the old code to the new copy of this script.
  5. Fill in the db account values for db_config.php in the new code.
  6. Fill in the AJAX_URL value in the twitter_display_config.php script in the new code.
  7. Kill the running copies of get_tweets.php and parse_tweets.php.
  8.  Start the new copies of get_tweets.php and parse_tweets.php.

If you have any problems making this conversion, get in touch with me (, and I’ll help get the new code running.

Leave a Comment

Previous post:

Next post: