Lots of people ask us to build databases of tweets, but they seem to miss the fact that along with the tweets you can also collect an amazing database of people. Data about the people who tweet is the proverbial low hanging fruit. The Twitter API gives you the complete profile of the author of each tweet it delivers. You don’t have to make an extra API call. Twitter is basically saying, “Here is a fresh set of data about this person, please take it and build something useful.” The Twitter Terms of Service has strict limits on the reselling of tweet text, but lets you do whatever you want with user profiles. These are strong signs that Twitter looks favorably on applications based on their users.

There are several ways a good Twitter consultant can help their clients understand the value of user data. My favorite technique is to make the marketing case that a tweet database is a great source of leads. Along with knowing what is being said, you know who is saying it. You also know everything else that user is saying. Excuse me for being crass, but the best way to describe this is that it is like email marketing, only you get to read the email of everyone you want to communicate with. That is a huge advantage.

Twitter lets you fly at 30,000 feet over the general landscape of discussion about your client’s product or market segment, and then zoom down and focus on a single individual. That is completely unprecedented. Even better, you can gather solid metrics about the influence of each user through values like follower count and frequency of mentions by others. Some of these values, like follower count, are readily available by looking at a user’s profile, but others require programming. That is where a Twitter consultant can add value.

My pitch is generally that while you can get influence measurements from tools like Klout, those are generic measurements of influence against all Twitter users and areas of interest. If you use the Twitter API and collect only tweets about a specific set of keywords, you can identify the most influential people for this area. I’ve written a detailed tutorial on this subject.

The best Twitter consultants make sure that they go beyond just building what the client asks for based on a limited knowledge of what is possible with Twitter data. By opening up the marketing benefits of a database of Twitter users, a whole new set of features are possible, and both the client and consultant profit.

Back in the day, when Twitter was just starting out, a lot of us recognized the performance art aspect of tweeting your day or taking pictures of a meal as you prepared it. But then we were actual people discovering our exhibitionist side. What do you call it when a character in a movie tweets without revealing they are a character as Natalie Portman’s character in Swan Queen does? Well, that is surely a performance. How about Sarah Palin, or Russian President Medvedev? The sense of reality is slipping fast.

Part of the sales process for Twitter consulting is convincing a new client that Twitter is more than just another way to broadcast their message. You have to show them that what appears to be a random stream of tweets is really a collection of highly qualified sales prospects. By aggregating Twitter users as well as their Tweets, you can extract a great set of sales leads along with their contact info. One way to quickly demonstrate the value of tweet aggregation is to deliver an Excel spreadsheet of sales prospects that meet the client’s needs.

When you aggregate tweets from the Twitter streaming API, it also returns the complete account profile for each user. You can data mine this collection of users to extract highly targeted lists of users, along with their geographical location and home page URL.

The free 140dev Twitter framework is an example of the code you will need to do the tweet aggregation. The schema for the MySQL database it creates shows you it has a table for all the aggregated tweets, which links to the list of tweeting users. Since all of this data is collected for a specific set of keywords, you can then extract personal details on the users who tweet these keywords the most with a simple SQL statement:

SELECT count(*) AS cnt, users.screen_name, users.name, users.location, users.url
FROM tweets, users
WHERE tweets.user_id = users.user_id
AND users.location != ''
AND users.url != ''
GROUP BY tweets.user_id
ORDER BY cnt DESC
LIMIT 1000

The 140dev framework’s example database collects tweets for the keyword “recipe”, so this query gives us the most active tweeters in the food world. Here are the results in phpMyAdmin:

You can then export the results from phpMyAdmin to an Excel spreadsheet, and email it to your client. This gives them solid data in a familiar form. Twitter doesn’t deliver email addresses, and doesn’t even collect phone numbers, but you do get each user’s home page URL. This can be used to gather other contact info, a task that is easily farmed out to people on freelance sites like Mechanical Turk.

So the next time you want to convince a client that Twitter is not just a bunch of kids talking to each other, you can just create a tweet aggregation database for the client’s industry keywords, let it collect data for a few days, and pull out a list of targeted users.

I went to the OpenCoffee meetup in Cambridge the other day. They all recognized the importance of Twitter, but don’t understand how to make money from it. We are exactly where we were in 1996 with the World Wide Web when I helped start Andover.net. Great point in the cycle.

So here is my quick 5 point pitch on how clients can benefit from integrating Twitter into business and marketing models. But first keep in mind that you don’t make money “from Twitter”, you make money “with Twitter”. Meaning that Twitter is a lever for improving your other efforts, but you don’t get cash handed to you directly by Twitter users on Twitter. Anyway, here’s my list:

1. Putting keyword targeted tweets on pages in the right way is great for SEO. Google loves tweets. This will increase the page’s search rank, getting a lot more first time visitors.

2. Datamining of tweets lets you find the right people to follow in Twitter for your market. This can be used very effectively to build a big follower list. This list becomes profitable when you tweet messages with URLs you want people to click on. Think of it as free Adwords.

3. Follower lists are also essential if you want to make people do something in the real world, like contribute money, or go to an event. Twitter will be huge in 2012 election.

4. If you have a database of tweets, you can datamine it for sales leads. You can give sales people the Twitter accounts and home page URLs of people who tweet a lot about the products the salesperson is selling. The best part is that the salesperson can see exactly what prospects say about their products and competitors before contacting them.

5. You can also datamine a tweet database for sentiment trends. This is valuable for PR and customer service. It gives you a real-time read on how effective the rest of your communication program is.

I’ve learned that follower count is greatly overrated as measurement of influence and popularity on Twitter. I now use the frequency a user is mentioned in tweets containing specific keywords instead. I’ve written a new tutorial on calculating and apply this idea of mention count to identify influential Twitter users.

One of the first things I work on with new clients is helping them to learn the vocabulary people use on Twitter. Clients who use the Web for marketing are very aware of the search terms that are most effective for ads and SEO, and they expect the same phrases to be used on Twitter. They need to learn that Twitter’s 140-character limit leads to a much more concise way of describing products and services.

Twitter users tend to create tags for commonly expressed ideas, so one of the first things I do after starting to aggregate tweets for the client’s keywords is produce a popular tag report that they can view online. When I add tweets to a database, I extract all the hash tags and save them in a separate table along with the id of the tweet where they were used. This lets me perform a SQL query like this to discover which tags are used most often in tweets with the tracked keywords:
SELECT count( * ) AS cnt, tag
FROM tag_tweets
GROUP BY tag
ORDER BY cnt DESC

As you’d expect, the tags distribute themselves in a long tail. For example, in one database I maintain that now has more than a million tweets there are over 33,000 unique tags recorded. The first hundred tags appear thousands of times. The next hundred appear only a few hundred times, and the vast majority of tags appear less than a dozen times out of 1,000,000 tweets. If you can identify the hundred most popular tags, you have have a much greater insight into how people actually converse on Twitter.

Twitter doesn’t produce data on usage of search.twitter.com, but it is safe to assume that when people search Twitter they look for the same tags they put into tweets. If a client can add the right tags to their tweets, they have a much greater chance of being found by people through Twitter search.

Since I also collect user data along with each tweet, I can extract the screen name of users who tweet most often with these popular tags as a way of creating a following list.

The amount of time it takes to manage an active Twitter account is one of the major hurdles in getting clients to use Twitter effectively. The best approach is to create an automated tweeting system to ensure a steady flow of product and corporate messages, and then supplement these tweets with hand written ones.

Automated tweeting code is easy to write. I create a MySQL table that contains the automated tweets, and a matching data entry page so that the client can add and edit these tweets on their own. Then I write a script that retrieves a single tweet from the database and posts it to the client’s account. Each tweet has a field for the last date and time it was posted which is filled in by the autotweeting script. The code that does the tweeting retrieves tweets from the database in reverse order by the last time it was tweeted, so the tweets will be cycled through from first to last. Eventually the system will work its way back to the first item in the list and repeat the tweets again, but if you have a hundred or more tweets in the table, and only autotweet five or six times a day, nobody will notice when you finally repeat the same tweets.

This autotweeting creates a baseline for an active account, and once the tweets are entered in the database there is no work for the client. But this isn’t enough. No matter how well the autotweets are written they will have an impersonal feel, because they can’t be addressed to a specific user. Even if they aren’t written as pure ad copy, they still read like broadcasting.

The next step is adding tweets that clearly come from a human. To do this I have the client read their Twitter stream every few hours and reply to one of the tweets that seem interesting. This reply should be directed to the @user who made the original tweet. Since this type of tweet couldn’t be done by a robot, at least not in a convincing way, anyone reading the client’s Twitter stream will see that they are actively engaged with their followers. Five or six of these human tweets every day in combination with the autotweets is enough to create an active account.

The goal is to create an interesting stream of tweets that not only will be followed back in response to a follow, but will be read by the followers when they appear. At the same time you need to reduce the amount of time the client needs to work at tweeting, since they have other business tasks. By automating the product messages, it lets the client craft these messages carefully, and replace them over time as their marketing approach changes.

All this sounds very premeditated. Shouldn’t Twitter just be about one human really engaging with another human on a deeply personal level? Sure. But in practice companies use Twitter to promote their business, and anyone who denies that is living in a fantasy world. Twitter sells promoted tweets, which are nothing by ads, so the idea of commercial messages on Twitter is nothing new. If I can help a client create an active account that gets followed with much less work on their part, that is a useful service.

One of the toughest challenges in doing Twitter API consulting is convincing clients that they do indeed have to use Twitter. At this point everyone knows about Twitter, and has a general idea of what it does, but I keep finding that clients expect to treat Twitter apps like a normal, brochure-style website. They think someone will build them a custom app or integrate Twitter functionality into their existing site, and then they can ignore it while it does its magic.

It just doesn’t work that way. Back in 2007 when social media marketing was reaching its initial peak, engagement with customers was the theme repeated over and over. That idea seems to have been lost with the mass market adoption of Twitter. Now businesses want to join in, but they don’t understand that Twitter is a retail environment, not wholesale. It isn’t a matter of just posting commercial tweets or following users in an automated manner. You need to treat people on Twitter as individuals. Even if it isn’t practical to actually engage with each user individually, the client needs to at least present the appearance of being a human who is interacting with other humans. This means reading and posting tweets by hand. Automation can play an important role, but unless the client actually uses Twitter they just won’t get it, and they won’t be able to get its benefits.

The big challenge is helping the client understand what Twitter communication is about. This is its own medium, with unique idioms and communication styles. The difficulty is that when a client tries to read tweets for the first time they are put off by the inanity and profanity that is so common. The general reaction is “How can anyone read this garbage?” They are right, there is a lot of garbage in the general Twitter stream.

What a consultant needs to do is make the valuable tweets stand out from the background noise. One of the first things I do when working with a new client is start aggregating tweets based on some appropriate keywords. After a week’s worth of tweets and user data are available in the database, I use various metrics of popularity to identify the key influencers in this particular space. Then I create a simple display page that only shows tweets from these influential users. This filters out the crap, and allows the client to see just how much valuable market intelligence there is on Twitter waiting to be acted on.

The other benefit of this initial set of aggregated tweets is that it allows the client to start learning the true vocabulary of their users on Twitter. Most clients are focused on keywords and phrases that are successful in SEO and Adwords marketing, but Twitter’s 140 character limit forces users to be much more concise and adopt different idioms from what they use to search Google. A common three or four word phrase used for searching may be condensed to a six letter hash tag. That is why I also do textual analysis of the tweet database to see which tags are used most frequently by people who tweet with the client’s keywords.

It generally takes a couple of weeks, but once clients start to grok the power of Twitter as a marketing and customer engagement tool, they can be much more active partners in any consulting project.