Dublin, Ireland
+353 87 41 86 742

Twitter API Changes & A Request for Help

I've just posted the following request for support on Twitter's developer site. You can find the request at the following URL: https://dev.twitter.com/discussions/10124

The changes to the Twitter API mean I can't continue to use my current implementation and scale my web application. In the introduction to the forthcoming API changes @sippey (Michael Sippey) points out that Twitter would like to gain an understanding of the types of applications which exist in the Twitter ecosystem and how we use Twitter's APIs.

I really appreciate that @sippey and the Twitter team have shared their vision for the ecosystem and the new APIs they are providing. It's great to get the visibility. So I thought I'd reciprocate and be as open as possible about my application, what it tries to do, and how it is implemented. This should allow Twitter and @sippey to understand my application better.

Is there a home for me, and my application within the Twitter ecosystem - or is it time to move on?

The Request for Help

I'm responding to @sippey 's request:

"To [...] gain an understanding of what types of applications are accessing the API in order to evolve it to meet the needs of developers, it's important to have visibility into the activity on the Twitter API and the applications using the platform."

I have a problem around the new search limitations and rate limits in API1.1 - are there alternative APIs to the one's I'm currently using which would allow my site to continue to grow?

Let me share visibility into my application, what it does, and how it interacts with the current twitter API.

I'm a new developer who is new to the Twitter ecosystem. I'm trying to develop an application which would allow users to combine their RSS fees with twitter, so when they read a news article, they can see the conversations around the article on Twitter. When the user reads the news story Twitter's conversation is displayed along site the new item. I like to think that this approach is more inviting to users as the conversation is displayed to the user in the same context are the original story. This should encourage the user to join in the active discussions currently happening on Twitter.

To power the site I use the Twitter search API. I call this API when the user selects a news story. The application then searches for comments relating to a specific news article. In an effort to reduce the load on search endpoint, the site is designed to only search for comments when the user opens a news article.

Additionally to mitigate against the IP specific rate limitation, which Twitter currently applies, the site uses the user's own web browser to issue the search request. This means that Twitter sees the search request as originating at the user's IP address on a per user basis and not as a lot of search requests from a signal server request.

The search results are then sent back to my application server where I temporally cache them. This means that I can reduce the number of future search requests. This further reduces load on Twitter, while still providing the key functionality my website needs.

The problem is, that with the future imposed rate limitations on search that this could effectively limit the number of search requests I make, which in turn limits the number of users I can support - preventing my application from scaling beyond 24 concurrent users[1].

Are there alternatives to the Search API which would allow me to obtain historic data about a given Twitter search term?

You can read more about my website, what it does, and what its goals are on my blog post here: http://www.mind-flip.com/theBlog/2012/07/12/introducing-voczie/

You can even see the early site / experiment here:
http://voczie.com (login via Twitter required)
How a story appears with existing comments:
http://voczie.com/stry?f=4f10db235e77fb7a3c000004&s=502ebddf8829413530000076

Just a heads-up: I do know that my site doesn't comply with the new display rules and this is something I can fix, but I'm reluctant to invest the time to do this if the site itself won't scale, and therefore won't live beyond the introduction of API 1.1.

Thanks in advance to anyone who reads this and offers advice. It is all very greatly appreciated!

Footnotes
[1] Concurrent users
Given a rate limit of 720 requests per hour, or 12 per minute (720/60). Then assuming that a single user takes 2 minutes to read a news article before moving onto the next news item (and invoking another search request), or 1 request every 2 minutes per concurrent user. Gives a maximum of 24 (12 req/min * 2) concurrent users per minute.

2 Comments

  1. Facing the same issue here. My company has this website: http://www.itweetlive.com. we run user searches through JS calls. Problem is we can’t support even 15 users who want their RT and Mentions and searches without hitting the rate limit.
    And using stream wont help since we have over 10,000 active users (about 100 concurrent).

    • mcwoods

      Hey Ed,

      Thanks for getting into touch, I’ve posted a couple of replies to the forum [1] – but they appear currently be in moderation and have not been posted to the public thread yet.

      @cjmcguinness (Charles McGuinness) makes a great observation, currently any authenticated API calls are rate limited on a per user basis, not on a per application basis [2]. I just hope that Twitter maintains this policy with the new API1.1. Assuming that it does – then as long as the user has logged into Twitter first (on our sites), it should be possible to search twitter again without a significant rate limit.

      As the new search API is now authenticated I’ll need to re-architect my application to make the request from the server and not from the client. This will add some additional load onto the server and could increase my server cost / when scaling – but at least the functionality is still available. – And I don’t need to have all the code I do – which tries to detect fake search results being sent back to my server from the client web page….

      I guess, for now, that all we can do is wait and see what the exact details of API1.1 are.

      If you have an alternative solution or approach, it would be wonderful if you could share it?

      Many thanks,
      Chris

      [1] Twitter API Changes & A Request for help https://dev.twitter.com/discussions/10124
      [2] Twitter’s Current Rate Limiting Policy https://dev.twitter.com/docs/rate-limiting/faq#measurement

Leave a Reply