April 22, 2008

Google Search REST API

More than one year after Google discontinued the SOAP Search API, it finally got a proper replacement. The AJAX Search API can now be used from any Web application, not just in JavaScript. The other two Google AJAX APIs for feeds and translations were updated for non-AJAX use, as well.

"For Flash developers, and those developers that have a need to access the AJAX Search API from other Non-Javascript environments, the API exposes a simple RESTful interface. In all cases, the method supported is GET and the response format is a JSON encoded result set with embedded status codes."

"Using the APIs from your Flash or Server Side framework couldn't be simpler. If you know how to make an http request, and how to process a JSON response, you are in business," says Mark Lucovsky. Here's a simple example for web search:
http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Earth%20Day

There are some differences between the old SOAP API and the REST one.

PROs:
- the new API doesn't require a key
- there's no limitation for the number of queries
- it's much easier to use
- you can use the REST API for web search, but also for image search, news search, video search, local search, blog search and book search.

CONs:
- you need to send "a valid and accurate http referer header"
- you can only get up to 8 results in a single call and you can't go beyond the first 32 results
- the terms of use are pretty restrictive: for example, you need to attribute the results to Google and you are not allowed to change the order of search results.

It's interesting to notice that Yahoo's search APIs are more developer-friendly and, although they require an application ID and have some usage limitations (5,000 queries per IP per day), they offer more features and they are more flexible, by also including XML output. Another important difference is that Yahoo doesn't require "a valid and accurate http referer header".

Philipp Lenssen suggests that it's much easier to just screenscrape the results, but search engines could change their code or block your requests.

Update. Check this excellent interview with Mark Lucovsky, who mentions that the API has been available for almost two years, but it wasn't officially documented:

32 comments:

  1. is it really that difficult to create a 'valid and accurate http referer'? how much work is that? 30 seconds? more? less?

    as for Yahoo having a 'more developer-friendly' api - one could argue that. for instance, would you rather develop in python or cobol? that's your choice with search you can search either with google or yahoo.

    also, the obvious. the lesser search engine _has_ to provide more liberal usage terms, else nobody would even bother at all, would they?

    :)

    ReplyDelete
  2. Hi, I've just launched my new Job Search site JobGeni.com (beta) - The Spider that aggregates the best job boards on the web with the new Google AJAX Feed API.
    No nightly spidering, no big servers, just realtime rss feed search! have a look:
    http://www.jobgeni.com/

    ReplyDelete
  3. What about the Live Search APIs? They are SOAP only and require a key, but they allow 25K calls a day and up to 50 results per page - a lot more than what Yahoo offers, and come with less strings attached than Google's.
    And you get image search and phonebook search too.
    They are not to easy to use, but there is sample code in Flash out there that is not rocket science

    ReplyDelete
  4. actually, i don't think a referer is needed. cut'n'paste of the examples from http://code.google.com/apis/ajaxsearch/documentation/reference.html#_intro_fonje into a new tab works.

    ReplyDelete
  5. > but search engines could change
    > their code or block your requests

    That's a valid downside to screenscraping. Just a side-note: even APIs sometimes change their output, and if you're unlucky, that may break your code. I remember that Yahoo's REST API changed their output one day by adding a seemingly harmless XML-namespace declaration on top. However, these XMLNS thingies happen to confuse the PHP XML DOM parser so much that it won't be able to work with it anymore (unless you adjust it with a workaround).

    ReplyDelete
  6. Do you have the search api for sponsored links.
    (e.g. link for string layer is http://www.google.com/sponsoredlinks?q=lawyer&hl=en&um=1&ie=UTF-8)
    ?

    ReplyDelete
  7. I think http://www.gigawidget has started an app w/ the new AJAX API from google.

    ReplyDelete
  8. Does anybody know if google provided API for sponsored link?...It seems curret search API response doesn't include sponsored link.

    ReplyDelete
  9. i think referer spells referrer :)

    ReplyDelete
  10. @Tristan: there are historical reasons to spell it referer (e.g. NCSA HTTPD, Apache and the HTTP protocol all use this spelling).

    Of course, you're correct that "proper English" does indicate that it's spelled "referrer."

    -farkinga
    http://www.rtfa.net

    ReplyDelete
  11. does anyone know how to solve the 403 forbidden error in google?

    ReplyDelete
  12. lesser search engine _has_ to provide more liberal usage terms, else nobody would even bother at all, would they?

    ReplyDelete
  13. I just found this REST tool. It's quite handy run REST queries directly from Eclipse:
    http://http4e.roussev.org/

    ReplyDelete
  14. What do you think about the Bing API? Better than Google API and Yahoo! API? I tried to use it, but the 2 things I wanted to use are not available: Currency Conversion and MSN Autos. Google doesn't support currency conversion as well, so I gonna try Yahoo! now...

    ReplyDelete
  15. Is it possible to get the rest output in xml format.
    if anybody having idea please guid eme immidiatly.

    ReplyDelete
  16. Is there a way I can fetch as many results as I want using Google REST APIs?
    As it works with normal HTTP like(just append another parameter num=100 at the end)
    http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Earth%20Day&num=100, then it results 100 websites as a result, but this doesn't work with REST APIs, any solution?

    ReplyDelete
  17. Hi, I am noticing that I am getting different results using the REST API vs. the Google website, specifically, recently indexed documents in Google show up when searching the site but not when using the REST API. Is there a workaround?
    Thanks,

    ReplyDelete
  18. the SOAP API was working for me until just a few days ago :( (at seekng.com).. oh well, i use Yahoo, YouTube, Flickr, and MSN APIs, so not totally harmed at the moment

    ReplyDelete
  19. But the service is not the same like SOAP?

    ReplyDelete
  20. I know this is an old post, but I'm looking for more information. I'm developing a web application for a client of ours (a public figure in our country) where we'll monitor the mentions of this person on social sites (Twitter, Facebook, etc) and we'll also like to use this API to monitor his mentions on Google search results (such as News and Blogs mentions). Probably our process will run every 30 minutes with specify search terms regarding this person. Is that under the allowed features that can be used to? Would developing something like that get me banned from using the service? If I do, what other options could I have?

    ReplyDelete
  21. You right, i thing yahoo boss and other application would be developer friendly than GASA (Google Ajax Search Api) but sometimes, I just have a dream if theres some (one or application) can gather data from GASA and Yapi.

    ReplyDelete
  22. I fnid the API really difficult to use. instead you can use the following url:
    url = "http://www.google.com/ie?q=" + searchHTMLterm + "&num="+count.ToString()+"&hl=en&gl=uk&start="+startfrom.ToString()+"&sa=N&filter=0";


    I do believe there is a limit of how many requests you can send within 5 minutes or so (not sure, but you get a short term temp ban if you send too many within a short time frame)

    The API is far too limiting to me and it doesn't return the same results as you would get through searching via the web interface.

    pete

    ReplyDelete
  23. I'm trying to figure out the where to send this url request to get back search results. I've used rest before in other applications but i guess I'm confused as where to find the docs for building air apps using as 3.

    ReplyDelete
  24. does anyone know how to solve the 403 forbidden error in google?

    ReplyDelete
  25. i try to use google search api but found error

    ReplyDelete
  26. any new feedback. i seems working now. anyone using in a live system?

    ReplyDelete
  27. Does anybody know what is the parameter to get search list from video.google.com?

    https://ajax.googleapis.com/ajax/services/search/video?v=1.0&q=earth%20day
    This always returns video from youtube. i want data from google video.

    Am i using wrong api? or it just a parameter to pass to get google videos?

    ReplyDelete
  28. Restrict the results to Google Video by adding site:video.google.con to the query.

    ReplyDelete
  29. I will appriciate if you can give an example.

    I tried this
    https://ajax.googleapis.com/ajax/services/search/video?v=1.0&q=earth%20day&site=video.google.com

    but still it is fetching data from youtube only. i think "site" is not a valid parameter for this.

    ReplyDelete
  30. It's not a parameter, it's part of the query. site: is a Google operator.

    https://ajax.googleapis.com/ajax/services/search/video?v=1.0&q=earth%20day%20site:video.google.com

    ReplyDelete
  31. Thank you Alex.
    It's working now.
    Also thank you for the explaination.

    ReplyDelete
  32. I must know how to retrieve result with dates range...

    ReplyDelete

Note: Only a member of this blog may post a comment.