One of my primary goals for this codebase has always been pedagogical. I thought of it as a set of training wheels for the streaming API. The number of moving parts is so great with the streaming API that getting data flowing from the API to a database using just the official docs is a huge challenge. I wanted to help people get up and running with the least amount of code possible, even if that meant a less then robust implementation. I assumed they would then build onto that simple base. Maybe I never explained that well, or maybe people didn’t want to hear it. Either way there has always been a demand for more features, as can be expected with software.
The dilemma is that the more a set of code does the less understandable it is. Solving production level problems adds too many corner cases and nested logic. I have versions of tweet collection systems in production that I have to spend 20-30 minutes reading each time I need to modify them. So how should I move the framework code forward without losing its benefits as a training tool?
The first step was introducing the latest version 0.30 of the code. This cleaned up a lot of performance and installation issues that have bothered people in the past. It also created a better db schema that is closer to what I use in production systems. Now that this is released, I want to start adding features, but I don’t want to create a bloated set of code that can’t be easily understood.
I can see two solutions. First I want to create the simplest possible implementation of tweet collection. This would be a single script that just got tweets and added them to a tweet table. This would be purely for illustrative purposes. People new to the streaming API could see the basic tasks more easily this way. Then I want to start publishing enhancements, such as storing keywords for collection in a database, or doing quality control on the users whose tweets are received. I want to present these enhancements separately as changes to the current framework code. I would not try to out them all together into a single version of the code. This will allow people to learn how the enhancements work, and then they can add the pieces they need to their own code base.
If you have any suggestions on this approach or can see a better way of solving the problem of more stuff without more complexity, please let me know.
