The Twitter API provides two requests for user account info: users/show and users/lookup. On the surface the difference between the two seems obvious. Users/lookup returns data on 100 accounts at a time, while users/show only delivers a single user. The Twitter docs make the correct suggestion that users/lookup is best suited for gathering the account info based on a list of user_ids returned by requests like friends/ids and followers/ids. This leaves out an important difference between the possible requests.

Users/lookup only returns accounts that have not been deleted or suspended. If you pass 100 user_ids to this API call, and 10 of them are either deleted or suspended, you only get back data on 90 user accounts. It is up to you to compare the accounts returned with the accounts requested, and decide how to deal with any that are missing from the results. The problem with this is that you can’t tell which of the missing accounts are deleted or suspended. If an account has been suspended, it will almost certainly become active again in a few days. You may want to save the user_id and try again later to get its data.

Users/show, on the other hand, always returns something when a request is made. If the account is valid, you get the full profile. A 404 error code is returned for deleted accounts, and 403 is returned for suspended users. This makes users/show more informative, but rate limits and performance delays from making multiple requests makes it impractical as a replacement for users/lookup.

I’ve developed a set of best practices for handling this situation:

• When gathering data on a list of friends or followers, I first run users/lookup as my first pass. If any user_ids do not return results, I put them into a table along with the date/time of the first lookup. I then process this table’s contents with users/show. The user_ids that return a 404 are removed from the list. I repeat this processing every 24 hours. The suspended users will eventually reappear, and their ids can be then be removed from this list.
• I also maintain lists of users that need to be updated regularly for applications like tweet display or suggested follows. I add a date/time field to these user tables that is updated whenever I get a new copy of the account profile. Users/lookup is run against this list every 24 hours or so. If an account hasn’t been updated for 24 hours, I test it with users/show. Deleted accounts are removed, and suspended accounts are left in place, but flagged as suspended to keep them from being displayed by my apps.

You can find examples of this type of programming in the source code for my engagement programming book.

Rate limits are a constant concern when doing engagement programming with the REST API. I’ve settled on an incremental approach. Instead of building a rate accounting infrastructure that measures the remaining requests for each API call, I find it easier to write scripts that break high-usage tasks into manageable chunks that won’t exceed the limits. I then schedule a cronjob to repeat these scripts at a frequency that will stay below the rate limit. If for some reason I get back a 429 error code that signifies an exceeded rate limit, I have the script exit and let it try again later based on the cronjob schedule.

To support this code, I also record all API calls in an api_log database table that saves the account I was working with, the API request made, and the http code returned. A separate script checks this table and emails me if the number of rate limit errors over the last hour exceeds a pre-defined level.

This decoupled approach allows multiple scripts to overlap with the same API call. This is the reality of a complex system. Trying to be too much of a control freak, and coordinate all my scripts to prevent ever hitting a rate limit ends up in diminishing returns. You end up spending more time on the scaffolding surrounding your code, and less on actually getting work done.

Some people are afraid of ever triggering a rate limit error out of fear of suspension, but I have never had that problem. Remember, I back off as soon as I get the first error response. My code monitoring the api log also warns me if my system is getting overloaded. I can then reschedule the cronjobs at a lower rate, or have each script make fewer requests in each cycle.

I learned how to manage Twitter API rate limits with a minimum of work.

In yesterday’s post I recommended searching for the screen names of influential accounts as a way of getting high-quality leads for engagement. A related approach is to collect the timelines of key accounts, and extract the people they @mention. This is a good way to find the cool crowd in any subject area. If a leader engages directly with someone, they are worth checking out. You can find code for collecting the past timeline tweets for any account, and then updating this collection on a regular basis in my engagement book‘s source code.

Once you have a leader account’s tweets collected along with all their @mentions, you can run SQL queries to extract the most mentioned accounts, such as:
SELECT COUNT( * ) AS cnt, users.*
FROM tweet_mentions, users
WHERE tweet_mentions.source_user_id IN
(SELECT user_id FROM leaders)
AND tweet_mentions.target_user_id = users.user_id
GROUP BY tweet_mentions.target_user_id
ORDER BY cnt DESC

This query assumes that you have recorded the user ids of leader accounts in their own table.

A lot of my engagement programming for clients has involved finding good leads for them to build relationships with and gain as followers. Obviously, the search API is a good place to start, but just searching for keywords that match your engagement interests is not the best approach. For example, if I wanted to find high quality accounts that talk about food, I could run a search for keywords like food, restaurant, or cooking. Try this search and see how poor the quality of the tweets are. All you can really tell is that these users are probably hungry when they tweeted.

You can get much better results by using the leading accounts for your subject as the search keyword. Compare the results of the previous search with this search based on @FoodNetwork. Now we get people tweeting recipes, discussing cooking shows, and a good mix of celebrity chef accounts. The difference in quality is obvious. The rule is that people who tweet with the screen name of a leading account have a serious and informed interest in the subject. This group should be your starting point for an engagement campaign.

You can do your engagement directly from the search results, but if you want to store the results in a database for later analysis, you can start coding with this source code from my engagement programming book.

When you start programming with the Twitter API, the natural tendency is to identify users in your database and code by their screen name. It seems right, since that is how you identify them when using Twitter. I used to do that too until I found out that users in some communities change their screen names a lot more than expected. Twitter allows you to change an account’s screen name and still retain all tweets, friends, followers, etc. associated with the account. So if you’ve built a relational database of tweet data based on screen names and one of them changes, all your pointers for that user break. The only safe way to identify users is through their account’s user id. This is assigned when an account is created, and it never changes.

You can see a real-world database schema that follows this model of user ids as the linking value in the source code for my engagement programming book. The only place the screen name is recorded is in the users table. This field is updated with the current screen name every time my code refreshes the user records.

The only annoying aspect of programming with user ids, is that you will need to do a lookup in your database to convert the user id into a screen name so it can be displayed on web pages or in an app. For example, when I need to deliver tweets for display, I link the tweet to its user as follows:

SELECT tweets.*, users.screen_name
FROM tweets, users
WHERE tweets.user_id = users.user_id