Installation note: All of the Twitter API tools are built around a set of common library files that must be downloaded and installed before running the code shown here. To run the scripts shown here, just copy them into the same directory as the common files. You can then run it as a web URL, from the command line of a Telnet or SSH client, or as a cron job.

The last API tool showed you how to collect all the members of any Twitter list. Today we will look at another useful technique for lists: collecting details on the lists owned by any account. This is done with the /lists/ownerships API call.

The list data collected by this tool will be stored in a database table called list_ownerships, so it can be used as input for other tools. The MySQL creation statement for this table is at the start of the script. You can copy this statement and paste it into the SQL box in phpMyAdmin.

Before looking at the code, here is some background. Every list on Twitter has a unique list id value, which should be stored as a 64-bit unsigned number. In MySQL terms, this is an unsigned Bigint data type. The other identifier for each list is the slug, which is the list name with spaces replaced by dashes. The combination of the list owner’s screen name and the slug is used to create the URL for the list. This script will also record the full list name with spaces included.

Lists can be either public or private, and this value is returned by the API in an element called mode. When requesting /lists/ownerships, you are given only the account’s public lists, unless the account whose OAuth tokens are being used is the list’s owner. Remember to preserve the privacy of all users. If you do collect data on a private list, you should never display it or share it with anyone.

list_ownerships.php

<?php
// Copy lists owned by a user to a database table
// Copyright (c) 2014 Adam Green. All rights reserved. 
// Contact info: http://140dev.com, @140dev, adam@140dev.com
// Released as open source under MIT license

/* Create this table to store lists
CREATE TABLE IF NOT EXISTS `list_ownerships` (
  `list_id` bigint(20) NOT NULL,
  `owner_screen_name` varchar(20) NOT NULL,
  `slug` varchar(100) NOT NULL,
  `name` varchar(100) NOT NULL,
  `created_at` datetime NOT NULL,
  `description` varchar(100) DEFAULT NULL,
  `mode` enum('public','private') NOT NULL,
  `members` int(11) NOT NULL,
  `subscribers` bigint(20) NOT NULL,
  PRIMARY KEY (`list_id`),
  KEY `owner_screen_name` (`owner_screen_name`),
  KEY `slug` (`slug`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
*/

// $owner_screen_name is a string with list owner's screen name
// $table_name is a string with name of DB table for users
// $clear_table is 1 to empty table first, 0 to leave intact
function list_ownerships($owner_screen_name, $table_name, $clear_table=0) {
  
  if (empty($owner_screen_name) || empty($table_name)) {
    print "ERROR: Invalid arguments";
    exit;
  }
    
  // Connect to the database
  require('db_lib.php');
  $oDB = new db;
  
  if ($clear_table) {
    // Clear the table of old entries for this owner
    $oDB->select("DELETE FROM $table_name
        WHERE owner_screen_name = '" . $oDB->escape($owner_screen_name) . "'");
  }

  // Connect to API with OAuth
  require('oauth_lib.php');
  $connection = get_connection(); 
  
  // Loop through pages of lists, each page has 20-1000 members
  // This is rate limited to 15 calls per 15 minute window
  // Start cursor at -1, end when cursor becomes 0
  $cursor = -1; 
  while ($cursor<>0) { 
    $connection->request('GET', $connection->url('1.1/lists/ownerships'), 
      array('screen_name' => $owner_screen_name,
        'count' => 100,  // Asking for too many lists can cause a timeout
        'cursor' => $cursor));
        
    // Exit on API error
    if ($connection->response['code'] <> 200) {
      print "ERROR: " . $connection->response['code'] . "\n";
      print $connection->response['response'];
    }
  
    $results = json_decode($connection->response['response']);
    $lists = $results->lists;
    foreach($lists as $list) {
      
      $list_id = $list->id_str;
      
      // Prevent duplicates
      if (!$oDB->in_table($table_name,"list_id=$list_id")) {
        
         // Escape string values that may contain quotes
         $field_values = "list_id = $list_id, " .
          "owner_screen_name = '" . $oDB->escape($owner_screen_name) . "', " .
          "slug = '" . $oDB->escape($list->slug) . "', " .
          "name = '" . $oDB->escape($list->name) . "', " .
          "description = '" . $oDB->escape($list->description) . "', " .
          "created_at = '" . $oDB->date($list->created_at) . "', " .
          "members = " . $list->member_count . ", " .
          "subscribers = " . $list->subscriber_count . ", " .
          "mode = '" . $list->mode . "'";
          
        $oDB->insert($table_name, $field_values);
      }
    }

    // Get the cursor for the next page of results
    $cursor = $results->next_cursor;
  }
}

?>

You can test this tool with list_ownerships_test.php, which is set to collect all the lists owned by the @twitter account.

list_ownerships_test.php

<?php

require('list_ownerships.php');
list_ownerships('twitter','list_ownerships',1);

?>

Installation note: All of the Twitter API tools are built around a set of common library files that must be downloaded and installed before running the code shown here.

Twitter lists are a really underutilized feature, especially now that the limits have been raised to 1,000 lists, each with 5,000 members. I doubt if anyone will need 5,000,000 list members, but there are more uses for lists than most people realize. I think lists make a great way for users to generate a set of accounts that can be used by other apps.

Instead of building an entire user interface by hand to allow users to input a set Twitter accounts for your apps, you can let them build and manage a list in Twitter, and then read that list as input for your processing. Another benefit is that users can share their lists with others. A team can collaborate through lists by creating a Twitter account and sharing the login, so they can all edit the lists.

Today’s tool reads any public Twitter list from any account, and adds complete account profiles for the members to a database. It allows you to collect multiple lists into a single table, while keeping their original list identity. It also clears the members of any list from the table as an option, so you can refresh the database copy when users change the list.

Since this tool needs a database table for storage, I have included the MySQL creation statement at the start of the script. I’m going to keep this model for all the tools. It also serves as useful documentation. You can copy this statement and paste it into the SQL box in phpMyAdmin.

list_members.php

<?php
// Copy list members to a database table
// Copyright (c) 2014 Adam Green. All rights reserved. 
// Contact info: http://140dev.com, @140dev, adam@140dev.com
// Released as open source under MIT license

/* Create this table to store list members
CREATE TABLE IF NOT EXISTS `list_members` (
  `owner_screen_name` varchar(20) NOT NULL,
  `slug` varchar(100) NOT NULL,
  `user_id` bigint(20) unsigned NOT NULL,
  `screen_name` varchar(20) NOT NULL,
  `name` varchar(40) DEFAULT NULL,
  `profile_image_url` varchar(200) DEFAULT NULL,
  `location` varchar(30) DEFAULT NULL,
  `url` varchar(200) DEFAULT NULL,
  `description` varchar(200) DEFAULT NULL,
  `created_at` datetime NOT NULL,
  `followers_count` int(10) unsigned DEFAULT NULL,
  `friends_count` int(10) unsigned DEFAULT NULL,
  `statuses_count` int(10) unsigned DEFAULT NULL,
  `protected` tinyint(1) NOT NULL,
  PRIMARY KEY (`owner_screen_name`,`slug`,`user_id`),
  KEY `screen_name` (`screen_name`),
  KEY `owner_screen_name` (`owner_screen_name`),
  KEY `slug` (`slug`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
*/

// Arguments for list_members()
// $owner_screen_name is a string with account screen name
// $slug is a string with list name, spaces are replaced with - 
// For example: https://twitter.com/twitter/lists/official-twitter-accts
// $owner_screen_name = twitter
// $slug = official-twitter-accts

// $table_name is a string with name of DB table for users
// $clear_table is an optional value, if set to 1 the table is truncated first
function list_members($owner_screen_name, $slug, $table_name, $clear_table=0) {
	
  if (empty($owner_screen_name) || empty($slug) || empty($table_name)) {
    print "ERROR: Invalid arguments";
    exit;
  }
		
  // Connect to the database
  require('db_lib.php');
  $oDB = new db;
	
  if ($clear_table) {
    // Clear the table of old entries for this list
    $oDB->select("DELETE FROM $table_name
      WHERE owner_screen_name = '" . $oDB->escape($owner_screen_name) . "' " .
      "AND slug = '" . $oDB->escape($slug) . "'");
  }

  // Connect to API with OAuth
  require('oauth_lib.php');
  $connection = get_connection();	
	
  // Loop through pages of members, each page has 20 members
  // This is rate limited to 480 calls per 15 minute window
  // The total is 9,600 members, more than the 5,000 member limit
	
  // Start cursor at -1, end when cursor becomes 0
  $cursor = -1;	
  while ($cursor<>0) { 
    $connection->request('GET', $connection->url('1.1/lists/members'), 
      array('owner_screen_name' => $owner_screen_name,
      'slug' => $slug,
      'include_entities' => false,  // We don't need user's entities
      'skip_status' => true, // We don't need user's last tweet
      'cursor' => $cursor));
				
    // Exit on API error
    if ($connection->response['code'] <> 200) {
      print "ERROR: " . $connection->response['code'] . "\n";
      print $connection->response['response'];
    }
	
    // Escape string values that may contain quotes
    $owner_screen_name = $oDB->escape($owner_screen_name);
    $slug = $oDB->escape($slug);
		
    $results = json_decode($connection->response['response']);
    $users = $results->users;
    foreach($users as $user) {
			
      $screen_name = $oDB->escape($user->screen_name);
			
      // Prevent duplicates
      $where = "owner_screen_name = '$owner_screen_name' 
        AND slug = '$slug' AND screen_name = '$screen_name'";
      if (!$oDB->in_table($table_name,$where)) {
				
        // If user is not protected
        if (empty($user->protected)) {
          $protected = 0;
        } else {
          $protected = 1;
        }				
				
        $field_values = "owner_screen_name = '$owner_screen_name', " .
          "slug = '$slug', " .
          "user_id = " . $user->id_str . ", " .
          "screen_name = '$screen_name', name = '" . $oDB->escape($user->name) . "', " .
          "profile_image_url = '" . $user->profile_image_url . "', " .
          "location = '" . $user->location . "', " .
          "url = '" . $user->url . "', " .
          "description = '" . $oDB->escape($user->description) . "', " .
          "created_at = '" . $oDB->date($user->created_at) . "', " .
          "followers_count = " . $user->followers_count . ", " .
          "friends_count = " . $user->friends_count . ", " .
          "statuses_count = " . $user->statuses_count . ', ' . 
          "protected = $protected";
					
         $oDB->insert($table_name, $field_values);
       }
    }

    // Get the cursor for the next page of results
    $cursor = $results->next_cursor;
  }
}

?>

You can test this tool with list_members_test.php, which is set to read the members of this list: https://twitter.com/twitter/lists/twitter-engineering.

list_members_test.php

<?php

require('list_members.php');
list_members('twitter','twitter-engineering','list_members',1);

?>

One other point. This will read public lists from any account, and can also read a private list owned by the account whose OAuth tokens are used for making the API request.

The previous post touched on some issues of protected accounts that should be pursued in more detail. Twitter’s rules for developers have two basic principles that apply here: Don’t surprise the user, and Respect user privacy. Both certainly apply to revealing data from a protected account.

What becomes clear if you experiment with the code in the user_tweet.php tool is that it is possible to see the last tweet of a protected account. This happens when you request a protected account with OAuth tokens from that same account. This can occur in two ways. You can be using single-user OAuth and then call user_tweet() with that user’s user_id or screen_name. Or you can have a multi-user login system, and call user_tweet() with the tokens for the same user you are asking about.

The important point is that just because you get back a $response->status element in the API response, you can’t assume it is OK to display it or store it in a database. You must always check the $response->protected element first. If that is not empty, or has a value of true, then the account is protected, and you should ignore any tweet data delivered by the API.

I understand why Twitter coded the response for protected accounts the way they did. Their assumption is that when you authorize with a user’s tokens, it is OK to give you all their data. It is your responsibility to not accidentally reveal this data. The safe rule is: If the account is protected, don’t look at the tweets.

Installation note: All of the Twitter API tools are built around a set of common library files that must be downloaded and installed before running the code shown here.

For my first tool I’m going to keep it simple. This one should demonstrate my goals for these tools. They should be useful, single purpose, simple to include in existing apps, and informative about the quirks and issues that may not be documented for the API request being used.

This tool gets the data on a user’s last tweet, and returns it as an array. All of the tools are delivered as a function in a file with the same name. This allows you to use this code from any script with a require(), and then call the function. For example, this tool is in a file named user_tweet.php with a user_tweet() function. Let’s look at the tool script first, then we can test it out and examine the possible results.

user_tweet.php

<?php
// Return an array of data from a user's last tweet
// Copyright (c) 2014 Adam Green. All rights reserved. 
// Contact info: http://140dev.com, @140dev, adam@140dev.com
// Released as open source under MIT license

// $type is a string with either 'screen_name' or 'user_id'
// $value is the matching account value 
// For example a calling script would use user_tweet('screen_name','140dev')
function user_tweet($type, $value) {
	
  // Connect to API with OAuth
  require('oauth_lib.php');
  $connection = get_connection();
 
  // Get the account profile for this user
  $connection->request('GET', $connection->url('1.1/users/show'), 
    array($type => $value));

  if($connection->response['code'] == 200) {
    $response = json_decode($connection->response['response']);

  if (empty($response->protected)) {
    // If an account is not protected, 
    // the protected element is blank
    $protected = 0;
			
    // The details of the last tweet are in $response->status
    $tweet_id = $response->status->id_str;
    $created_at = $response->status->created_at;
    $tweet_text = $response->status->text;
  } else {
    // If an account is protected, 
    // $response->status is not included
    $protected = 1;
    $tweet_id = 0;
    $created_at = '';
    $tweet_text = '';
  }
		
  $results = array('code' => $connection->response['code'],
    'protected' => $protected,
    'tweet_id' => $tweet_id,
    'created_at' => $created_at,
    'tweet_text' => $tweet_text);
		
  } else {
    $results = array('code' => $connection->response['code'],
      'response' => $connection->response['response']);
  }

  return $results;
}
?>

Each tool in this series will include a matching test script to show how the tool can be called, and the type of results to expect.

user_tweet_test.php

<?php

require('user_tweet.php');
print_r(user_tweet('screen_name','justinbieber'));

?>

You can run this test script from a web browser or the command line of a telnet or SSH client. I don’t include any HTML formatting, so the web browser output may be sloppy. Here is a sample run from the command line:

php user_tweet_test.php
Array
(
    [code] => 200
    [protected] => 0
    [tweet_id] => 433469700228411392
    [created_at] => Wed Feb 12 05:16:44 +0000 2014
    [tweet_text] => Love both these people. Go see @KevinHart4real's movies. Thanks guys.  http://t.co/5tkATy6Urg
)

Notice that the output includes the API response code. Your code should check this code element before using any of the other data. If the code is not 200, you have an API error, and the returned array will also contain the full error message in a response element for debugging.

This tool also handles the issue of a protected account. If the account requested is protected, the last tweet is not included in the results, and the array has empty values for the tweet data. Your code that calls user_tweet() should check the protected element before trying to use the results. Here is a sample run with a protected account:

Array
(
    [code] => 200
    [protected] => 1
    [tweet_id] => 0
    [created_at] => 
    [tweet_text] => 
)

Feedback on this and and any other tools can be posted on the Google Group.

The previous post listed the set of common library files needed to use the Twitter API tools. Installing this package to get ready to use the tools is pretty easy:

• Download the zipped set of files and unzip them on your local machine.
• Create a web accessible directory on your server. You’ll want this to be available to the web so you can test the web versions of the tools.
• Upload the unzipped set of files into this new directory.
• Edit config.php to include the MySQL connection options for a database. You can use an existing database, but I’d recommend creating a new one. There is no preset schema yet. Each tool will include the SQL statements needed to create the tables it will need.
• Edit config.php to include a complete set of OAuth tokens. If you are not sure how to create a set of tokens, you should read my free ebook on the subject. You’ll want the app that provides the tokens to be set for read, write, and direct message permissions to be able to run all the tools that are built.

That’s it. You are now ready to run the tools. As of writing this post there are no tools, but I’ll be cranking them out pretty fast. Hopefully, others will contribute tools as well, once they are familiar with the coding style I’m trying to achieve.

To make sure you have everything configured correctly, the set of common files includes two simple test programs. You should run them now to be sure. If you have any problems, this Google Group is ready for tech support questions.

test_db.php

<?php

// Connect to database using db_lib.php
require('db_lib.php');
$db = new db();

// Test connection with a simple query
$result = $db->select("SHOW DATABASES");
while($row = mysqli_fetch_assoc($result)) {
	print $row['Database'] . "\n";
}

?>

Test_db.php’s purpose is to make sure you can make a connection to MySQL. You can run it at the command line of your telnet or SSH client with:

php test_db.php

It will output a list of databases available to your MySQL user login. If it fails, you’ll see an error message from db_lib.php, and a copy of the MySQL error should be added to error_log.txt in the same directory as the code.

test_oauth.php

<?php
	
// Connect to API with OAuth
require('oauth_lib.php');
$connection = get_connection();

// Get the account profile for this user
$connection->request('GET', $connection->url('1.1/users/show'), 
	array('screen_name' => '140dev'));

print "Response code: " . $connection->response['code'] . "\n";

print "Response data: " . $connection->response['response'];

?>

Test_oauth.php is a basic proof that you can connect to the API through the OAuth tokens you placed in config.php. It uses the /users/show request, which will return the details of my @140dev account profile. If it fails, you’ll get the API’s error code and message. Running this test can again be done with your telnet or SSH client with:

php test_oauth.php

The most likely causes for an OAuth failure are: incorrect OAuth tokens, server clock that is more than 5 minutes incorrect, and incomplete installation of OAuth files. It is odd how often this happens, but if you get an error code of 0, it almost always means that cacert.pem was not uploaded with the rest of the files. Make sure you copied the complete set of files to your server.

If OAuth is still failing, use the Google Group for support. OAuth can be a pain to get started, but once it is configured correctly, it is reliable.

The collection of API tools will only need a few library files: db_lib.php, oauth_lib.php, and Matt Harris’ tmhOAuth library. These are all packaged together in a convenient zip file, ready for download. Here’s a summary of what you’ll find inside, and then we’ll look at the library code:

• cacert.pem – SSL certificate used by tmhOAuth library.
• config.php – Settings for database and OAuth connections.
• db_lib.php – My standard database library of functions for MySQL.
• license.txt – MIT license for my code, which is all open source, of course.
• oauth_lib.php – Simple library for creating an OAuth connection with tmhOAuth.
• test_db.php – Test for database connection code.
• test_oauth.php – Test for OAuth connection code.
• tmhOAuth.php – Matt Harris’ OAuth library.

config.php

<?php

// Database values for db_lib.php
$db_host = 'localhost';
$db_user = '*****';
$db_password = '*****';
$db_name = '*****';

// MySQL time zone setting to normalize dates
$time_zone = 'America/New_York';

// OAuth tokens for oauth_lib.php
$consumer_key = '*****';
$consumer_secret = '*****';
$user_token = '*****';
$user_secret = '*****';

?>

db_lib.php

<?php 
// General purpose database library for use by 140dev Twitter API tools 
// Copyright (c) 2014 Adam Green. All rights reserved.  
// Contact info: http://140dev.com, @140dev, adam@140dev.com 
// Released as open source under MIT license 
class db {   
  public $dbh;   
  public $error;   
  public $error_msg;          

  // Create a database connection for use by all functions in this class   
  function __construct() {     
    require('config.php');          
    if($this->dbh = mysqli_connect($db_host, 
      $db_user, $db_password, $db_name)) { 

      // Set every possible option to utf-8
      mysqli_query($this->dbh, 'SET NAMES "utf8"');
      mysqli_query($this->dbh, 'SET CHARACTER SET "utf8"');
      mysqli_query($this->dbh, 'SET character_set_results = "utf8",' .
        'character_set_client = "utf8", character_set_connection = "utf8",' .
        'character_set_database = "utf8", character_set_server = "utf8"');
    } else {
      // Log an error if the connection fails
      $this->error = true;
      $this->error_msg = 'Unable to connect to DB';
      $this->log_error('__construct','attempted connection to ' . $db_name);
    }

    date_default_timezone_set($time_zone);
  }

  // Call this after each DB request to test for and log errors
  // Supply the calling function name and query, so they can be logged
  private function error_test($function,$query) {

    // Record the last error state in the object, 
    // so code using objects of this class can read it
    if ($this->error_msg = mysqli_error($this->dbh)) {
        $this->log_error($function,$query);
        $this->error = true;
    } else {
        $this->error = false;
    }
    return $this->error;
  }

  // Write any errors into a text log
  // Include the date, calling script, function called, and query
  private function log_error($function,$query) {
    $fp = fopen('error_log.txt','a');
    fwrite($fp, date(DATE_RFC822) . ' | ' . 
      $_SERVER["SCRIPT_NAME"] . ' -> ' . $function . 
      ' | ' . $this->error_msg . ' | ' . $query . "\n");
    fclose($fp); 
  }

  // Create a standard data format for insertion of PHP dates into MySQL
  public function date($php_date) {
    return date('Y-m-d H:i:s', strtotime($php_date));	
  }

  // All text added to the DB should be cleaned with mysqli_real_escape_string
  // to block attempted SQL insertion exploits
  public function escape($str) {
    return mysqli_real_escape_string($this->dbh,$str);
  }

  // Test to see if a specific field value is already in the DB
  // Return false if no, true if yes
  public function in_table($table,$where) {
    $query = 'SELECT * FROM ' . $table . 
      ' WHERE ' . $where;
    $result = mysqli_query($this->dbh,$query);
    $this->error_test('in_table',$query); 
    return mysqli_num_rows($result) > 0;
  }

  // Perform a generic select and return a pointer to the result
  public function select($query) {
    $result = mysqli_query( $this->dbh, $query );
    $this->error_test("select",$query);
    return $result;
  }

  // Add a row to any table
  public function insert($table,$field_values) {
    $query = 'INSERT INTO ' . $table . ' SET ' . $field_values;
    mysqli_query($this->dbh,$query);
    $this->error_test('insert',$query);
  }

  // Update any row that matches a WHERE clause
  public function update($table,$field_values,$where) {
    $query = 'UPDATE ' . $table . ' SET ' . $field_values . 
      ' WHERE ' . $where;
    mysqli_query($this->dbh,$query);
   $this->error_test('update',$query);
  }  
}  
?>

oauth_lib.php

<?php
// General purpose OAuth library for use by 140dev Twitter API tools
// Copyright (c) 2014 Adam Green. All rights reserved. 
// Contact info: http://140dev.com, @140dev, adam@140dev.com
// Released as open source under MIT license

// Create an OAuth connection
function get_connection() {

	// Get OAuth tokens for engagement account
	require('config.php');
	require('tmhOAuth.php');
	
	// Create the connection
	// The OAuth tokens are kept in config.php
	$connection = new tmhOAuth(array(
		  'consumer_key'    => $consumer_key,
		  'consumer_secret' => $consumer_secret,
		  'user_token'      => $user_token,
		  'user_secret'     => $user_secret
	));
			
	return $connection;
}

?>

To get started with these tools, just download the package of libraries, unzip it, and copy the files to a web accesible directory on your server. The next post will cover installation and testing.

I already have two sets of open source Twitter API code: the streaming API framework, and my Engagement Programming book. Both of require a fair amount of setup, and the code is very interrelated. This new project will be a set of individual tools that can be run by themselves or called from any application that seems some Twitter integration.

All you’ll need to run these tools will be a common set of library files, which I’ll document in the next post. The tools will need to be able to connect to database and make API calls with OAuth. I’ll package these together in a single zip to make installation easy.

I already some tools planned out, such as getting all followers and posting tweets, but if you have any suggestions, please send them along by tweet or email, or post them here as comments.

The next post has download instructions for the common libraries used by all the tools.