Tagtwitter

Generating a twitter OAuth access key – the semi-manual way

[UPDATE]
Apparently someone at Twitter was listening, or I’m going senile/blind. Let’s call it a combination of both.

Instead of following all the steps below, you could just login with the Twitter account you want to use on http://dev.twitter.com, register your application and then click ‘Edit Details’ on the application overview page at http://dev.twitter.com/apps. Next click the ‘Application detail’ button on the right, followed by the ‘My Access Token’ button in order to get your Access Token and Access Token Secret.

This makes the old post below rather obsolete. Clearly a case of me thinking everything is a nail and ruby is a hammer (don’t they usually say this about java coders?) 😉

[ORIGINAL POST]

OAuth is great! OAuth allows your application to use your user’s data without the need to ask for their password. So Twitter made the API much safer for their and your users. Hurray! Free pizza for everyone!

Unless of course you’re using the Twitter API for your own needs like running your own bot and don’t need access to other user’s data. In such cases a simple username/password combination is more than enough. I can understand however that the Twitter guys don’t really care that much about these exceptions(?). Most such uses for the API are probably rather spammy in nature.

!!! If you have a twitter app that uses the API to access external user’s data: look for another solution. This solution is ONLY meant when you ONLY need access to your own account(s) through the API.

Other Solutions

Mr Dallas Devries posted a solution here which involves requesting and scraping a one-time PIN.
But: I like to minimize the amount of calls I make to twitter’s API or pages to lessen my chances of meeting the fail whale. Also, as soon as the pin isn’t included in a div called anymore, this will fail.

However, mr Devries’ post was a starting point for my solution, so I’m much obliged to him posting his findings.

Authenticating with the Twitter API: old vs new

Acessing The Twitter API the old way:

require 'twitter'
httpauth = Twitter::HTTPAuth.new('my_account','my_secret_password')
client = Twitter::Base.new(httpauth)
client.update('Hurray!')

The OAuth way:

require 'twitter'
oauth = Twitter::OAuth.new('ve4whatafuzzksaMQKjoI', 'KliketyklikspQ6qYALcuNandsomemored8pQ6qYALIG7mbEQY')
oauth.authorize_from_access('123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis', 'fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh')
client = Twitter::Base.new(oauth)
client.update('Hurray!')

In the above case,
ve4whatafuzzksaMQKjoI is the ‘consumer key’ (sometimes also referred to as ‘consumer token’) and
KliketyklikspQ6qYALcuNandsomemored8pQ6qYALIG7mbEQY is the ‘consumer secret’. You’ll get these from Twitter when you register your app.

123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis is the ‘access token’ and
fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh is the ‘access secret’. This combination gives the registered application access to your account. I’ll show you how to obtain these by following the steps below.

(Basically you’ll need a bunch of keys and you’ll have to jump a bit through hoops to obtain them for your server/bot. )

How to get these keys

1. Surf to the twitter apps registration page

go to http://dev.twitter.com/apps to register your app. Login with your twitter account.

2. Register your application

Enter something for Application name, Description, website,… as I said: they make you jump through hoops.

If you plan on using the API to post tweets, Your application name and website will be used in the ‘5 minutes ago via…’ line below your tweet. You could use the this to point to a page with info about your bot, or maybe it’s useful for SEO purposes.

For application type I choose ‘browser’ and entered http://www.hadermann.be/callback as a ‘Callback URL’. This url returns a 404 error, which is ideal because after giving our account access to our ‘application’ (step 6), it will redirect to this url with an ‘oauth_token’ and ‘oauth_verifier’ in the url. We need to get these from the url. It doesn’t really matter what you enter here though, you could leave it blank because you need to explicitely specify it when generating a request token.

You probably want read&write access so set this at ‘Default Access type’.

3. Get your consumer key and consumer secret

On the next page, copy/paste your ‘consumer key’ and ‘consumer secret’. You’ll need these later on. You also need these as part of the authentication in your script later on:

oauth = Twitter::OAuth.new([consumer key], [consumer secret])
4. Obtain your request token

run the following in IRB to obtain your ‘request token’
Replace my fake consumer key and consumer secret with the one you obtained in step 3. And use something else instead http://www.hadermann.be/callback: although this will only give a 404, you shouldn’t trust me.

irb(main):001:0> require 'oauth'
irb(main):002:0> c = OAuth::Consumer.new('ve4whatafuzzksaMQKjoI',
'KliketyklikspQ6qYALcuNandsomemored8pQ6qYALIG7mbEQY',
{:site => 'http://twitter.com'})
irb(main):003:0> request_token = c.get_request_token(:oauth_callback => 'http://www.hadermann.be/callback')
irb(main):004:0> request_token.token
=> "UrperqaukeWsWt3IAlfbxzyBUFpwWIcWkHP94QH2C1"

This (UrperqaukeWsWt3IAlfbxzyBUFpwWIcWkHP94QH2C1) is the request token: Copy/paste this token, you will need this next.

5. Authorize your application

surf to https://api.twitter.com/oauth/authorize?oauth_token=[the above token], for example:

https://api.twitter.com/oauth/authorize?oauth_token=UrperqaukeWsWt3IAlfbxzyBUFpwWIcWkHP94QH2C1

This will bring you to the ‘An application would like to connect to your account’- screen on Twitter where you can grant access to the app you just registered. If you aren’t still logged in, you need to login first. Click ‘Allow’. Unless you don’t trust yourself.

6. Get your oauth_verifier from the redirected url

Your browser will be redirected to your callback url, with an oauth_token and oauth_verifier parameter appended. You’ll need the oauth_verifier.

In my case the browser redirected to:

http://www.hadermann.be/callback?oauth_token=UrperqaukeWsWt3IAlfbxzyBUFpwWIcWkHP94QH2C1&oauth_verifier=waoOhKo8orpaqvQe6rVi5fti4ejr8hPeZrTewyeag

Which returned a 404, giving me the chance to copy/paste my oauth_verifier: waoOhKo8orpaqvQe6rVi5fti4ejr8hPeZrTewyeag

7. Request an access token

Back to irb, use the oauth_verifier to request an access token, as follows:

irb(main):005:0> at = request_token.get_access_token(:oauth_verifier => 'waoOhKo8orpaqvQe6rVi5fti4ejr8hPeZrTewyeag')
irb(main):006:0> at.params[:oauth_token]
=> "123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis"
irb(main):007:0> at.params[:oauth_token_secret]
=> "fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh"

We’re there!

123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis is the access token.
fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh is the access secret.

Try it!

Try the following to post an update:

require 'twitter'
oauth = Twitter::OAuth.new('ve4whatafuzzksaMQKjoI', 'KliketyklikspQ6qYALcuNandsomemored8pQ6qYALIG7mbEQY')
oauth.authorize_from_access('123-owhfmeyAgfozdyt5hDeprSevsWmPo5rVeroGfsthis', 'fGiinCdqtehMeehiddenymDeAsasaawgGeryye8amh')
client = Twitter::Base.new(oauth)
client.update('Cowabunga!')

Now you can go to your twitter page and delete the tweet if you want to.

Twitter gem – undefined method `stringify_keys’

Have you been getting the following errors when running the Twitter gem lately ?


/usr/local/lib/ruby/gems/1.8/gems/httparty-0.4.3/lib/httparty/response.rb:15:in `send': undefined method `stringify_keys' for # (NoMethodError)
from /usr/local/lib/ruby/gems/1.8/gems/httparty-0.4.3/lib/httparty/response.rb:15:in `method_missing'
from /usr/local/lib/ruby/gems/1.8/gems/mash-0.0.3/lib/mash.rb:131:in `deep_update'
from /usr/local/lib/ruby/gems/1.8/gems/mash-0.0.3/lib/mash.rb:50:in `initialize'
from /usr/local/lib/ruby/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:101:in `new'
from /usr/local/lib/ruby/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:101:in `fetch'
from test.rb:26

It’s because Twitter has been sending back plain text errors that are treated as a string instead of json and can’t be properly ‘Mashed’ by the Twitter gem. Also check http://github.com/jnunemaker/twitter/issues#issue/6.

Without diving into the bowels of the Twitter gem or HTTParty, you could ‘begin…rescue’ this error and try again in 5 minutes. I fixed it by overriding the offending code to return nil and checking for a nil response as follows:

module Twitter
  class Search
    def fetch(force=false)

      if @fetch.nil? || force
        query = @query.dup
        query[:q] = query[:q].join(' ')
        query[:format] = 'json' #This line is the hack and whole reason we're monkey-patching at all.

        response = self.class.get('http://search.twitter.com/search', :query => query, :format => :json)

        #Our patch: response should be a Hash. If it isnt, return nil.
        return nil if response.class != Hash

        @fetch = Mash.new(response)
      end

      @fetch
    end

  end
end

(adapted from http://github.com/jnunemaker/twitter/issues#issue/9)

If you have a better solution: speak up!

Twitter traffic might not be what it seems

Are you using bit.ly stats to measure interest in the links you post on twitter?

I’ve been hearing for a while about people claiming to get the majority of their traffic originating from twitter these days.

Now, I’ve been playing with the twitter ruby gem recently, doing various experiments which I’ll not go into detail here because they could be regarded as spamming… if I’d conduct them on a large scale, that is.
It’s scary to see people actually engaging with @replies crafted with some regular expressions and eliza-like trickery on status updates found using the twitter api. I’m wondering how Twitter is going to contain the coming spam-flood.

When posting links I used bit.ly as url shortener, since this one seems to be the de-facto standard on twitter. A nice thing about bit.ly is that it shows some basic stats about the redirects it performs for your shortened links.

To my surprise, most links posted almost immediately resulted in several visitors. Now, seeing that I was posting the links together with some information concerning what the link is about, I concluded that the people who were actually clicking the links should be very targeted visitors.
This felt a bit like free adwords, and I suddenly started to understand why everyone was raving about getting traffic from twitter.

How wrong I was! (and I think several 1000 online marketers with me)

On the destination site I used a traffic logging solution that works by including a little javascript snippet in your pages. It seemed that somehow all visitors disappeared after the bit.ly redirect and before getting to the site, because I was hardly seeing any visitors there. So I started investigating what was happening: by looking at the logfiles of the destination site, and by making my own ‘shortened’ urls by doing redirects using a very short domain name I own. This way, I could check the apache access_log before the redirects.

Most user agents turned out to be bots without a doubt. Here’s an excerpt of user-agents awk’ed from apache’s access_log for a time period of about one hour, right after posting some links:

AideRSS 2.0 (postrank.com)
Java/1.6.0_13
Java/1.6.0_14
libwww-perl/5.816
MLBot (www.metadatalabs.com/mlbot)
Mozilla/4.0 (compatible;MSIE 5.01; Windows -NT 5.0 - real-url.org)
Mozilla/5.0 (compatible; Twitturls; +http://twitturls.com)
Mozilla/5.0 (compatible; Viralheat Bot/1.0; +http://www.viralheat.com/)
Mozilla/5.0 (Danger hiptop 4.6; U; rv:1.7.12) Gecko/20050920
Mozilla/5.0 (X11; U; Linux i686; en-us; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.04 (jaunty) Firefox/3.5
OpenCalaisSemanticProxy
PycURL/7.18.2
PycURL/7.19.3
Python-urllib/1.17
Twingly Recon
twitmatic
Twitturly / v0.6
Wget/1.10.2 (Red Hat modified)
Wget/1.11.1 (Red Hat modified)

Of the few user-agents that seem ‘real’ at first, half are originating from an ip-address used by Amazon EC2. And I doubt people are setting op proxies on there.

Oh yeah, Googlebot (the real deal, from a legit google owned address) is sucking up posted links like fresh oysters.
I guess google is trying to make sure in advance to never be beaten by twitter in the ‘realtime search’ department. Actually, I think it’d be almost stupid NOT to post any new pages/posts/websites on Twitter, it must be one of the fastest ways to get a Googlebot visit.

Same experiment with a real, established twitter account

Now, because I was posting the url’s either as ‘status’ messages or directed @people, on a test-account with hardly any (human) followers, I checked again using the twitter accounts from a commercial site I’m involved with. These accounts all have between 500 and 1000 targeted (I think) followers. I checked the destination access_logs and also added ‘my’ redirect after the bit.ly redirect: same results, although seemingly a bit higher real visitor/bot ratio.

Btw: one of these account was ‘punished’ with a 1 week lock recently because the same (1 one!) status update was sent that was sent right before using another account. They got an email explaining the lock because the account didn’t act according to their TOS. I can’t find anything in their TOS about it, can you?
I don’t think Twitter is on the right track punishing a legit account, knowing the trickery I had been doing with it’s api went totally unpunished. I might be wrong though, I often am.

On the other hand: this commercial site reported targeted traffic and actual signups from visitors coming from Twitter. The ones that are really real visitors are also very targeted. I’m just not sure if the amount of work involved could hold up against an adwords campaign.

Reposting the same link over and over again helps

On thing I noticed: It helps to keep on reposting the same links with regular intervals.
I guess most people only look at their first page when checking out recent posts of the ones they’re following, or don’t look too far back when performing a search.

Now, this probably isn’t according to the twitter TOS. Actually, it might be spamming but no-one is obligated to follow anyone else of course.

This way, I was getting more real visitors and less bots. To my surprise (when my programmer’s hat is on) there were still repeated visits from the same bots coming from the same ip-addresses. Did they expect to find something else when visiting for a 2nd or 3rd time? (actually,this gave me an idea: you can’t change a link once it’s posted, but you can change where it redirects to)
Most bots were smart enough not to follow the same link again though.

Are you successful in getting real visitors from Twitter?
Are you only relying on bit.ly to provide traffic stats?