I've just posted the following request for support on Twitter's developer site. You can find the request at the following URL: https://dev.twitter.com/discussions/10124
The changes to the Twitter API mean I can't continue to use my current implementation and scale my web application. In the introduction to the forthcoming API changes @sippey (Michael Sippey) points out that Twitter would like to gain an understanding of the types of applications which exist in the Twitter ecosystem and how we use Twitter's APIs.
I really appreciate that @sippey and the Twitter team have shared their vision for the ecosystem and the new APIs they are providing. It's great to get the visibility. So I thought I'd reciprocate and be as open as possible about my application, what it tries to do, and how it is implemented. This should allow Twitter and @sippey to understand my application better.
Is there a home for me, and my application within the Twitter ecosystem - or is it time to move on?
The Request for Help
I'm responding to @sippey 's request:
"To [...] gain an understanding of what types of applications are accessing the API in order to evolve it to meet the needs of developers, it's important to have visibility into the activity on the Twitter API and the applications using the platform."
I have a problem around the new search limitations and rate limits in API1.1 - are there alternative APIs to the one's I'm currently using which would allow my site to continue to grow?
Let me share visibility into my application, what it does, and how it interacts with the current twitter API.
I'm a new developer who is new to the Twitter ecosystem. I'm trying to develop an application which would allow users to combine their RSS fees with twitter, so when they read a news article, they can see the conversations around the article on Twitter. When the user reads the news story Twitter's conversation is displayed along site the new item. I like to think that this approach is more inviting to users as the conversation is displayed to the user in the same context are the original story. This should encourage the user to join in the active discussions currently happening on Twitter.
To power the site I use the Twitter search API. I call this API when the user selects a news story. The application then searches for comments relating to a specific news article. In an effort to reduce the load on search endpoint, the site is designed to only search for comments when the user opens a news article.
Additionally to mitigate against the IP specific rate limitation, which Twitter currently applies, the site uses the user's own web browser to issue the search request. This means that Twitter sees the search request as originating at the user's IP address on a per user basis and not as a lot of search requests from a signal server request.
The search results are then sent back to my application server where I temporally cache them. This means that I can reduce the number of future search requests. This further reduces load on Twitter, while still providing the key functionality my website needs.
The problem is, that with the future imposed rate limitations on search that this could effectively limit the number of search requests I make, which in turn limits the number of users I can support - preventing my application from scaling beyond 24 concurrent users.
Are there alternatives to the Search API which would allow me to obtain historic data about a given Twitter search term?
You can read more about my website, what it does, and what its goals are on my blog post here: http://www.mind-flip.com/theBlog/2012/07/12/introducing-voczie/
You can even see the early site / experiment here:
http://voczie.com (login via Twitter required)
How a story appears with existing comments:
Just a heads-up: I do know that my site doesn't comply with the new display rules and this is something I can fix, but I'm reluctant to invest the time to do this if the site itself won't scale, and therefore won't live beyond the introduction of API 1.1.
Thanks in advance to anyone who reads this and offers advice. It is all very greatly appreciated!
 Concurrent users
Given a rate limit of 720 requests per hour, or 12 per minute (720/60). Then assuming that a single user takes 2 minutes to read a news article before moving onto the next news item (and invoking another search request), or 1 request every 2 minutes per concurrent user. Gives a maximum of 24 (12 req/min * 2) concurrent users per minute.