Since the announcement of the Twitter-Gnip partnership, there have been lots of news stories and blog posts stating that this is the end of the independent developer, because there is no more free Twitter data. This is completely wrong. You can get all the Twitter data you need, as long as you don’t want *all* the Twitter data. What Twitter is selling through Gnip is up to 50% of the full Firehose, which means 50% of all tweets. That is 50 million tweets a day at the present time. Twitter is also selling the entire Firehose to search engines, like Google and Bing.

Nobody is going to convince me that an independent consultant or a private corporation needs a copy of every single tweet. What these people need is all the tweets for a specific set of keywords or from specific users, and that is still free through the streaming API. Using the /statuses/filter request you can get all the tweets for up to 400 keywords and 5,000 users. All you have to do is decide which words or users you need to track when you make the request.

What Twitter won’t let you do is try and grab every single tweet, store them in a database, and then deliver them for selected keywords or users. That is the definition of a search engine. If you really have the bandwidth and server capacity needed to do this for 100,000,000 tweets each day, why in the world should Twitter deliver this to you for free and foot the bill for its side of the bandwidth and server capacity? That is just absurd. But then a lot of Web 2.0 was exactly that. Thankfully, it is now over.

This ability to get all the tweets, as long as you limit the keywords and users to a reasonable number was restated again today by John Kalucki of the Twitter API team. A developer asked on the API mailing list:

If I am using the statuses/filter streaming API, with a “track=” query
that is not overly broad, and my client never receives any “limit”
responses, can I assume that the results returned represent all the
results from the entire firehose?  In other words, in the absence of
“limit” response, is my visibility into the firehose 100%?

John responded:

Yes, where firehose is the stream of all public statuses, with some low-quality accounts removed.

From my usage of the streaming API, this is correct.

But what about even higher limits? Shouldn’t data be free? Maybe it should be in a perfect world, but in the real world bandwidth, servers, and labor costs aren’t. If you actually need more than the default level of access, you can request a higher level, which is often given for free. If you need all of Twitter’s data, you should share Twitter’s costs, because you better have a business model that supports your side of the costs.

And if you need source code for gathering tweets from the streaming API and storing them in a database, that is still free also.

It may be my age, but I’m confused when I read that Twitter is somehow evil by acting competitively towards their developers, or that there is no future in being a Twitter developer. I think what they mean is that Twitter is becoming increasingly competitive with developers who try to replace Twitter.com. But isn’t that obvious? Why do people think that Twitter.com doesn’t have the right to increase the functionality of their own site, and try to make it more attractive? Of course, Twitter will mimic the most successful applications of this type. Wouldn’t you?

That doesn’t mean there is no future in Twitter application development. It is just that people have forgotten what a platform is. Many years ago there were software companies that created products, like programming languages, operating systems, and databases that provided functionality for programmers to use in serving client’s needs. These programmers were called developers when they built stand-alone products, and consultants when they wrote code for specific clients. The major difference between then and now was that platform vendors were able to charge for their code. This naturally put their interests into alignment with outside programmers. Platform vendors needed programmers to write for their platform, so they were smart to leave room for others to make money too.

Then the Internet removed most of the platforms, and the programming languages all became free. Developers all tried to get rich by building and flipping code, and consultants devoted most of their time to building custom websites. We are entering an era of platforms again, specifically Twitter and Facebook, but programmers haven’t accepted that they are less likely to get rich by whipping up a cool application based on these platforms and giving it away. The one company that really gets the idea of being a platform is Apple, but they have institutional memory that goes back to before the Internet.

We need to shift our perspective and programmers have to actually find paying customers, who have needs to fill for their own businesses. No matter what Twitter does, there will always be lots of money to be made creating custom applications that solve real-world business problems. Programmers just have to start thinking of themselves as consultants, and find clients who need their code badly enough to pay for it. That doesn’t preclude making a lot of money, but it does mean that Twitter doesn’t have to be feared as a competitor.

Does anyone honestly think that Twitter will try to write and sell its own real estate applications, medical applications, time and billing applications, etc? The field is wide open. Just leave Twitter.com to Twitter, and they’ll leave you alone. They might even help you make money.

I had an unfortunate experience the other day when I tried helping someone out with a Twitter API question on a developer Q&A website. I won’t list the site’s name or URL, because I see no reason to give them even more traffic. What I want to do is point out some ways of using the API that will certainly backfire for any site that uses them, even though I know clients ask for these “features” all the time.

I tried replying to the user who asked the question, but first I was asked to register with the site. That’s reasonable, and keeps down spam. The registration form gave me the chance to register through my Twitter account. I accepted and was shown the standard Twitter OAuth login form that asked me to allow the site to “access and update” my account.

When you create a Twitter app for use with OAuth, you can give it Read & Write access to user accounts, or just read access, as I showed in a recent tutorial. This developers forum must have set up their app with Read & Write access. This seemed excessive, since I just needed to login. I wasn’t going to allow the site to change my Twitter account, or was I?

I wrote my reply on the site, and when I tried to save it, nothing happened. I tried again with no luck, and gave up. A few hours later, when I looked at my Twitter account, I saw that the developers site had modified my account to follow them on Twitter, and had posted a tweet in my account telling people about the site. These are exactly the type of behaviors that Twitter keeps warning against on the Twitter dev group. A Twitter application should never make any changes to a user’s account without explicit permission.

The temptation to do these types of things with the API is high, because they are so viral. The app’s owners get lots of followers, and all the app’s users deliver an “endorsement” to their own followers. But you should never give in to the temptation to build this into an application. Tricking people doesn’t help you in the long-run.

What was even funnier is how the site reacted when I discovered the problem. I immediately unfollowed them and deleted the tweet they had sent in my name. I also canceled the connection between that site and my Twitter account. At least that part of OAuth worked perfectly. I then tweeted a warning about this site’s behavior. Their response was to send back a tweet accusing me of lying. I replied that I had a copy of their tweet in my database, because I aggregate all of my tweets with my 140dev Twitter framework. They again accused me of making the whole thing up.

So let’s recap all the ways they managed to piss off a potential user of their site. They tweeted in my account without a warning or my permission, they followed themselves in my account without my permission, and finally they said I was lying when I had proof. I’ll keep this as one of those stories I tell clients when they ask me to cross the line as a Twitter consultant.

One of the oddest behaviors on Twitter is spammers who think they can persuade people by blasting the same message over and over again through multiple accounts. Yesterday I looked at the Twitter search results for the phrase “twitter api”. I found hundreds of copies of these two tweets from dozens of accounts:

The Twitter API is the one responsible why Twitter can easily know that you are automating your task http://adf.ly/BG6i

Using software that use Twitter API to do things is like inviting Twitter anti spam bots to look on your activites http://adf.ly/BG6i

I won’t even comment on the non-native English. Well, I guess I just did . The real point is that it doesn’t make any sense. URLs in tweets get a high click through rate, because they seem more honest than ads. But if someone sees hundreds of copies of the same tweet, will they really believe this is the action of a human or a bot?

The logic behind much of the Twitter spam is that users get many of their tweets through search, so if you put out tweets with likely search terms, your message will get in front of them. It makes sense, but if it is done in such a crude way, anyone searching will see that this is obviously a spam message. I think these sloppy versions of Twitter spam are done by email spammers, who are trying to apply the same techniques to Twitter. The difference is that you can’t see the emails arriving in everyone else’s mailbox, but you can see all the tweets from all the public accounts.

I know that often clients ask for this type of Twitter spam to be sent out on their behalf. A good Twitter consultant will explain that Twitter is a completely different medium from email, and it requires different techniques. I have nothing against commercial speech. I do it all the time for clients. I just don’t understand sloppy, badly executed communication plans that are bound to fail.

Part of the sales process for Twitter consulting is convincing a new client that Twitter is more than just another way to broadcast their message. You have to show them that what appears to be a random stream of tweets is really a collection of highly qualified sales prospects. By aggregating Twitter users as well as their Tweets, you can extract a great set of sales leads along with their contact info. One way to quickly demonstrate the value of tweet aggregation is to deliver an Excel spreadsheet of sales prospects that meet the client’s needs.

When you aggregate tweets from the Twitter streaming API, it also returns the complete account profile for each user. You can data mine this collection of users to extract highly targeted lists of users, along with their geographical location and home page URL.

The free 140dev Twitter framework is an example of the code you will need to do the tweet aggregation. The schema for the MySQL database it creates shows you it has a table for all the aggregated tweets, which links to the list of tweeting users. Since all of this data is collected for a specific set of keywords, you can then extract personal details on the users who tweet these keywords the most with a simple SQL statement:

SELECT count(*) AS cnt, users.screen_name, users.name, users.location, users.url
FROM tweets, users
WHERE tweets.user_id = users.user_id
AND users.location != ''
AND users.url != ''
GROUP BY tweets.user_id
ORDER BY cnt DESC
LIMIT 1000

The 140dev framework’s example database collects tweets for the keyword “recipe”, so this query gives us the most active tweeters in the food world. Here are the results in phpMyAdmin:

You can then export the results from phpMyAdmin to an Excel spreadsheet, and email it to your client. This gives them solid data in a familiar form. Twitter doesn’t deliver email addresses, and doesn’t even collect phone numbers, but you do get each user’s home page URL. This can be used to gather other contact info, a task that is easily farmed out to people on freelance sites like Mechanical Turk.

So the next time you want to convince a client that Twitter is not just a bunch of kids talking to each other, you can just create a tweet aggregation database for the client’s industry keywords, let it collect data for a few days, and pull out a list of targeted users.

I went to the OpenCoffee meetup in Cambridge the other day. They all recognized the importance of Twitter, but don’t understand how to make money from it. We are exactly where we were in 1996 with the World Wide Web when I helped start Andover.net. Great point in the cycle.

So here is my quick 5 point pitch on how clients can benefit from integrating Twitter into business and marketing models. But first keep in mind that you don’t make money “from Twitter”, you make money “with Twitter”. Meaning that Twitter is a lever for improving your other efforts, but you don’t get cash handed to you directly by Twitter users on Twitter. Anyway, here’s my list:

1. Putting keyword targeted tweets on pages in the right way is great for SEO. Google loves tweets. This will increase the page’s search rank, getting a lot more first time visitors.

2. Datamining of tweets lets you find the right people to follow in Twitter for your market. This can be used very effectively to build a big follower list. This list becomes profitable when you tweet messages with URLs you want people to click on. Think of it as free Adwords.

3. Follower lists are also essential if you want to make people do something in the real world, like contribute money, or go to an event. Twitter will be huge in 2012 election.

4. If you have a database of tweets, you can datamine it for sales leads. You can give sales people the Twitter accounts and home page URLs of people who tweet a lot about the products the salesperson is selling. The best part is that the salesperson can see exactly what prospects say about their products and competitors before contacting them.

5. You can also datamine a tweet database for sentiment trends. This is valuable for PR and customer service. It gives you a real-time read on how effective the rest of your communication program is.

I wrote a post last week about building a maintenance plan into any Twitter consulting agreement. One of the reasons I gave then was the continuing evolution of the Twitter API. It is a certainty that this code will continue to change, and many of these changes have the potential of breaking your code. This announcement on the Twitter development mailing list is a perfect example.

The most fundamental part of the API, the format of data returned for every tweet, is about to change in a way that will probably break Javascript apps that talk to the API, and could break apps in other languages as well. Adapting to this change won’t be hard, and Twitter warned developers that this is coming. But who warns the clients? How will someone who paid for Twitter to be integrated into their site handle this?

It isn’t possible for non-programmers to read all the messages on the developer’s mailing list and make the necessary changes. Consultants need to watch out for clients and they deserve to be paid for doing so. Otherwise, clients will decide that Twitter apps are unreliable, and that they soon fail. For their own good, consultants need to warn clients that this type of change is inevitable, and help them plan for it. If any client doesn’t agree, have them read this announcement.

It’s been six months since Fred Wilson advised Twitter developers to stop “filling holes” in Twitter. That blog post raised a huge outcry from developers that Twitter was changing the rules in the middle of the game. Well, yeah, they were, but the rules are going to keep changing all the time.

What I found shocking at the time wasn’t that Twitter was competing with its own independent developers. That happens every time a company gets large enough. I was amazed that nobody seemed to read the post carefully enough to see that Fred was also telling developers where to dig:

Verticals – We have some successes to point to here like Stocktwits for finance and Flixup for movies but this is a wide open opportunity in most verticals and we haven’t seen as much effort here as I’d have expected.

I agree with Fred. There is an entire industry of vertical Twitter applications waiting to be built. It may be that most Twitter developers are young enough to think that Web 2.0 is the only model. They need to realize that it was a fallacy. You can’t just build something cool that is totally generic, and then wait for the users to show up. Yes, that worked for Twitter, but it is the classic exception, not the rule.

There are so many possible vertical apps that can be built. Election politics are a great example. You can build a generic election app, and then customize it for every politician and political cause. Food is another open field. There are food truck apps, but that is a small segment of the food service industry. Sports, movies, travel, books, consumer electronics. They all need apps. The code may be generic inside, but you have to package it for each type of user, geographical location, socio-economic class, language, etc. That was what Fred was trying to tell everyone.

A good Twitter consultant needs to spend a lot of time educating new clients in the real value of Twitter and how to interact with users. Most people who aren’t regular users think that Twitter is just another broadcast advertising medium. It is easy to see why they think that when you look at how much blatant advertising and spam there is in the general tweet stream. So when a new client tries to use Twitter they usually write tweets that are copies of the text they use in Google ads.

A good way of explaining how to use Twitter effectively for marketing is to compare it to a party at a trade show or industry conference. Experienced trade show attendees know that this is where the most valuable connections are made, and long-term relationships are maintained. If a client thinks of Twitter as thousands of separate cocktail parties, all going on at the same time, they will see that using advertising copy in their tweets is not only ineffective, it is also insulting. I ask clients, “How would you react if someone entered a party, stood on a chair, and yelled out ‘Hot deals on new products. 50% off!’”.

The right way to network at a trade show party is to stand near a group involved in a public conversation that fits your interests, and gradually inject your own comments where appropriate. When you get a chance, introduce yourself to some of the participants in that conversation. Offer to help others with introductions where possible. Repeat this process at a few parties, and people will start recognizing you and bring you into other conversations. If you have something interesting to share, they’ll introduce you to their friends. Repeat this for a few events, and you’ll soon be a regular with a wide range of contacts. Your sales message will come across over time as a part of what you are about. There is nothing wrong with trying to sell, but it is rude to walk up to a stranger at a party and start with a sales pitch.

This analogy works well with clients, and if they are given a good tweet aggregation system that puts users in front of them who tweet with their keywords regularly, they soon understand the incredible value a Twitter marketing campaign represents.

Part of the education process that a Twitter consultant has to do with a new client is explaining that a Twitter app is not the same thing as a typical website. Web pages that are built on the Twitter API have to be monitored constantly, and the code has to be tweaked often. This is different from a brochure website with static pages, or even a website with new content being added through a CSM like WordPress.

The Twitter API is constantly evolving, which is a polite way of saying that it isn’t done yet, and may never be done. New features are always being added that will break your API code, such as the conversion to OAuth. Twitter HQ does a good job of warning people that changes are on their way, but they make no promises of backwards compatibility. Change happens, and it isn’t always pretty. Twitter still faces scaling challenges, and this means that API failures are still and probably will always be a fact of life.

With these issues in mind, I always try to build continual monitoring and priority upgrades into any consulting agreement. Clients have to realize that even the best written Twitter system will fail at times, and if they want to present a quality experience to their users they need to have someone watching for problems and upgrading the code in advance of any API “improvements.”

Twitter consultants who want to build it and forget it will leave a trail of unhappy clients behind.