I’ve been seeing a definite trend towards more and more Twitter staff responding to questions on the developer mailing lists. This is another in a series of positive signs for the future of the API. What I especially like is an apparent specialization by support staff, with some people seemingly assigned to specific areas of API functionality. Twitter’s investment in developer support proves they want the developer community to grow.

Now that I said something nice, I’d like to make a minor complaint. You can see a certain level of equivocation in the previous paragraph with words like “apparent” and “seemingly.” This uncertainty is based on the odd practice of the support staff to not identify themselves as working for Twitter in their response. Back in the day, it was traditional to use a signature like “Twitter Developer Advocate.” The new crowd rarely include any signature or other sign of their role, so you have to guess their status based on the context of their message. I’ve tried figuring out who is who, but many of them have perfected a tragically hip and ironic stance in their Twitter bios and tweets. I guess job titles are too 20th century. It would be great if everyone doing Twitter dev support at least had a signature that showed their messages are “official” in some way. But then I’m an old, crotchety, and totally unhip guy.

I still appreciate how much help we are all getting. Thanks, everyone at Twitter, even if I’m not sure who you are.

Before the IPO I predicted that devs would be viewed as an asset, not just a burden. That change is happening right on schedule. Check out this NY Times article. The money quote:

Twitter is counting on millions of websites to link to the service and encouraging legions of independent developers to find creative new uses for its platform, driving up activity and the number of advertisements that Twitter users see.

When was the last time you saw an article describing API devs this way? Hell, when was the last time you saw an article about API devs in any form?

All we need is parity. It is a partnership. Twitter provides the API, we provide new users. Everybody wins.

Since the announcement of the Twitter-Gnip partnership, there have been lots of news stories and blog posts stating that this is the end of the independent developer, because there is no more free Twitter data. This is completely wrong. You can get all the Twitter data you need, as long as you don’t want *all* the Twitter data. What Twitter is selling through Gnip is up to 50% of the full Firehose, which means 50% of all tweets. That is 50 million tweets a day at the present time. Twitter is also selling the entire Firehose to search engines, like Google and Bing.

Nobody is going to convince me that an independent consultant or a private corporation needs a copy of every single tweet. What these people need is all the tweets for a specific set of keywords or from specific users, and that is still free through the streaming API. Using the /statuses/filter request you can get all the tweets for up to 400 keywords and 5,000 users. All you have to do is decide which words or users you need to track when you make the request.

What Twitter won’t let you do is try and grab every single tweet, store them in a database, and then deliver them for selected keywords or users. That is the definition of a search engine. If you really have the bandwidth and server capacity needed to do this for 100,000,000 tweets each day, why in the world should Twitter deliver this to you for free and foot the bill for its side of the bandwidth and server capacity? That is just absurd. But then a lot of Web 2.0 was exactly that. Thankfully, it is now over.

This ability to get all the tweets, as long as you limit the keywords and users to a reasonable number was restated again today by John Kalucki of the Twitter API team. A developer asked on the API mailing list:

If I am using the statuses/filter streaming API, with a “track=” query
that is not overly broad, and my client never receives any “limit”
responses, can I assume that the results returned represent all the
results from the entire firehose?  In other words, in the absence of
“limit” response, is my visibility into the firehose 100%?

John responded:

Yes, where firehose is the stream of all public statuses, with some low-quality accounts removed.

From my usage of the streaming API, this is correct.

But what about even higher limits? Shouldn’t data be free? Maybe it should be in a perfect world, but in the real world bandwidth, servers, and labor costs aren’t. If you actually need more than the default level of access, you can request a higher level, which is often given for free. If you need all of Twitter’s data, you should share Twitter’s costs, because you better have a business model that supports your side of the costs.

And if you need source code for gathering tweets from the streaming API and storing them in a database, that is still free also.

It may be my age, but I’m confused when I read that Twitter is somehow evil by acting competitively towards their developers, or that there is no future in being a Twitter developer. I think what they mean is that Twitter is becoming increasingly competitive with developers who try to replace Twitter.com. But isn’t that obvious? Why do people think that Twitter.com doesn’t have the right to increase the functionality of their own site, and try to make it more attractive? Of course, Twitter will mimic the most successful applications of this type. Wouldn’t you?

That doesn’t mean there is no future in Twitter application development. It is just that people have forgotten what a platform is. Many years ago there were software companies that created products, like programming languages, operating systems, and databases that provided functionality for programmers to use in serving client’s needs. These programmers were called developers when they built stand-alone products, and consultants when they wrote code for specific clients. The major difference between then and now was that platform vendors were able to charge for their code. This naturally put their interests into alignment with outside programmers. Platform vendors needed programmers to write for their platform, so they were smart to leave room for others to make money too.

Then the Internet removed most of the platforms, and the programming languages all became free. Developers all tried to get rich by building and flipping code, and consultants devoted most of their time to building custom websites. We are entering an era of platforms again, specifically Twitter and Facebook, but programmers haven’t accepted that they are less likely to get rich by whipping up a cool application based on these platforms and giving it away. The one company that really gets the idea of being a platform is Apple, but they have institutional memory that goes back to before the Internet.

We need to shift our perspective and programmers have to actually find paying customers, who have needs to fill for their own businesses. No matter what Twitter does, there will always be lots of money to be made creating custom applications that solve real-world business problems. Programmers just have to start thinking of themselves as consultants, and find clients who need their code badly enough to pay for it. That doesn’t preclude making a lot of money, but it does mean that Twitter doesn’t have to be feared as a competitor.

Does anyone honestly think that Twitter will try to write and sell its own real estate applications, medical applications, time and billing applications, etc? The field is wide open. Just leave Twitter.com to Twitter, and they’ll leave you alone. They might even help you make money.

The smartest thing Twitter ever did was externalize their data through an API. It created an active third party community that supported Twitter when times were tough and they didn’t have the resources to develop clients for all the platforms that needed them. That developer community was based on the Web 2.0 business model, which meant many of these apps were angel or VC funded, silly as that now seems. Thankfully, the Web 2.0 model, or what I call Arrington capitalism, is finally passing. There will still be developers who think they can make hundreds of millions of dollars by creating the next great Twitter app, but they will be rare.

The next phase of development is starting for Twitter. That will involve independent consultants and corporate developers integrating Twitter into websites and marketing systems. Nobody will get rich from this, but it is critical for Twitter’s success. If Twitter thinks of their API as a strategic component of their growth, which they seem to do, they must now treat developers as a labor force. They need to make sure it is possible for developers to get paid for building Twitter apps.

This is a much older business model that goes back to the Eighties for PC software, and before that for larger systems. I have some experience with that world, and I plan on nagging Twitter HQ into showing more respect for this type of developer. I’ve been exchanging emails with Taylor Singletary (@episod) on this issue, and he’s been honest about how little resources are devoted to helping developers get whitelisted. I think this approach is based on the Web 2.0 model. If a VC funded company asks to “partner” with Twitter, they get plenty of attention. If a small-time developer asks, they stay at the back of the line and don’t even get a response. This isn’t Taylor’s fault. He needs a lot more help. If Twitter fails to provide it, they will waste all the work they have done in building the API in the first place.

I’ve been thinking about how to prepare the tweet aggregation code used for the dev tweets page. I know that I want to add multiple modules to the system, and make it easy to customize the appearance of the tweet list. This sounds a lot like the WordPress model with plugins and themes, so I’m going to adopt a similar code architecture. If I do this right, others will be able to add their own plugins and themes. I also want to make installation as drop dead simple as possible. Finally, I’m going to call the code base 140Dev. I already use this name for the company and site. Why bother coming up with a new product name, only to have to rename the site and company to match the product later? And so it begins…

I finally finished the last tweaks to the JS file. Doing Ajax from within a JS file HTML loaded by a plugin requires some interesting adjustments. I didn’t want to hard code the Ajax calls, so I had to capture the current URL of the plugin directory and save that in a hidden div within the page’s HTML so the JS code could use it. A little fussy, but it works!