April 22, 2008

Google Search REST API

More than one year after Google discontinued the SOAP Search API, it finally got a proper replacement. The AJAX Search API can now be used from any Web application, not just in JavaScript. The other two Google AJAX APIs for feeds and translations were updated for non-AJAX use, as well.

"For Flash developers, and those developers that have a need to access the AJAX Search API from other Non-Javascript environments, the API exposes a simple RESTful interface. In all cases, the method supported is GET and the response format is a JSON encoded result set with embedded status codes."

"Using the APIs from your Flash or Server Side framework couldn't be simpler. If you know how to make an http request, and how to process a JSON response, you are in business," says Mark Lucovsky. Here's a simple example for web search:
http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Earth%20Day

There are some differences between the old SOAP API and the REST one.

PROs:
- the new API doesn't require a key
- there's no limitation for the number of queries
- it's much easier to use
- you can use the REST API for web search, but also for image search, news search, video search, local search, blog search and book search.

CONs:
- you need to send "a valid and accurate http referer header"
- you can only get up to 8 results in a single call and you can't go beyond the first 32 results
- the terms of use are pretty restrictive: for example, you need to attribute the results to Google and you are not allowed to change the order of search results.

It's interesting to notice that Yahoo's search APIs are more developer-friendly and, although they require an application ID and have some usage limitations (5,000 queries per IP per day), they offer more features and they are more flexible, by also including XML output. Another important difference is that Yahoo doesn't require "a valid and accurate http referer header".

Philipp Lenssen suggests that it's much easier to just screenscrape the results, but search engines could change their code or block your requests.

Update. Check this excellent interview with Mark Lucovsky, who mentions that the API has been available for almost two years, but it wasn't officially documented:

at

32 comments:

  1. PeterApril 23, 2008 at 1:26 AM

    is it really that difficult to create a 'valid and accurate http referer'? how much work is that? 30 seconds? more? less?

    as for Yahoo having a 'more developer-friendly' api - one could argue that. for instance, would you rather develop in python or cobol? that's your choice with search you can search either with google or yahoo.

    also, the obvious. the lesser search engine _has_ to provide more liberal usage terms, else nobody would even bother at all, would they?

    :)

    ReplyDelete
  2. VidalApril 23, 2008 at 6:30 AM

    Hi, I've just launched my new Job Search site JobGeni.com (beta) - The Spider that aggregates the best job boards on the web with the new Google AJAX Feed API.
    No nightly spidering, no big servers, just realtime rss feed search! have a look:
    http://www.jobgeni.com/

    ReplyDelete
  3. AnonymousApril 23, 2008 at 7:59 AM

    What about the Live Search APIs? They are SOAP only and require a key, but they allow 25K calls a day and up to 50 results per page - a lot more than what Yahoo offers, and come with less strings attached than Google's.
    And you get image search and phonebook search too.
    They are not to easy to use, but there is sample code in Flash out there that is not rocket science

    ReplyDelete
  4. AnonymousApril 23, 2008 at 8:26 AM

    actually, i don't think a referer is needed. cut'n'paste of the examples from http://code.google.com/apis/ajaxsearch/documentation/reference.html#_intro_fonje into a new tab works.

    ReplyDelete
  5. Philipp LenssenApril 23, 2008 at 10:12 AM

    > but search engines could change
    > their code or block your requests

    That's a valid downside to screenscraping. Just a side-note: even APIs sometimes change their output, and if you're unlucky, that may break your code. I remember that Yahoo's REST API changed their output one day by adding a seemingly harmless XML-namespace declaration on top. However, these XMLNS thingies happen to confuse the PHP XML DOM parser so much that it won't be able to work with it anymore (unless you adjust it with a workaround).

    ReplyDelete
  6. sanyastormJuly 7, 2008 at 10:20 AM

    Do you have the search api for sponsored links.
    (e.g. link for string layer is http://www.google.com/sponsoredlinks?q=lawyer&hl=en&um=1&ie=UTF-8)
    ?

    ReplyDelete
  7. AnonymousOctober 21, 2008 at 1:19 PM

    I think http://www.gigawidget has started an app w/ the new AJAX API from google.

    ReplyDelete
  8. AnonymousJanuary 13, 2009 at 10:19 AM

    Does anybody know if google provided API for sponsored link?...It seems curret search API response doesn't include sponsored link.

    ReplyDelete
  9. LoupioteFebruary 5, 2009 at 6:20 PM

    i think referer spells referrer :)

    ReplyDelete
  10. AnonymousFebruary 10, 2009 at 8:08 AM

    @Tristan: there are historical reasons to spell it referer (e.g. NCSA HTTPD, Apache and the HTTP protocol all use this spelling).

    Of course, you're correct that "proper English" does indicate that it's spelled "referrer."

    -farkinga
    http://www.rtfa.net

    ReplyDelete
  11. AnonymousFebruary 16, 2009 at 2:24 PM

    does anyone know how to solve the 403 forbidden error in google?

    ReplyDelete
  12. karta isicApril 28, 2009 at 12:39 AM

    lesser search engine _has_ to provide more liberal usage terms, else nobody would even bother at all, would they?

    ReplyDelete
  13. AnonymousJune 14, 2009 at 2:38 PM

    I just found this REST tool. It's quite handy run REST queries directly from Eclipse:
    http://http4e.roussev.org/

    ReplyDelete
  14. multimoltiJuly 13, 2009 at 11:42 PM

    What do you think about the Bing API? Better than Google API and Yahoo! API? I tried to use it, but the 2 things I wanted to use are not available: Currency Conversion and MSN Autos. Google doesn't support currency conversion as well, so I gonna try Yahoo! now...

    ReplyDelete
  15. Raja SubramanianJuly 28, 2009 at 6:02 AM

    Is it possible to get the rest output in xml format.
    if anybody having idea please guid eme immidiatly.

    ReplyDelete
  16. nishalspaceAugust 1, 2009 at 6:36 AM

    Is there a way I can fetch as many results as I want using Google REST APIs?
    As it works with normal HTTP like(just append another parameter num=100 at the end)
    http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Earth%20Day&num=100, then it results 100 websites as a result, but this doesn't work with REST APIs, any solution?

    ReplyDelete
  17. HaniSeptember 18, 2009 at 6:40 AM

    Hi, I am noticing that I am getting different results using the REST API vs. the Google website, specifically, recently indexed documents in Google show up when searching the site but not when using the REST API. Is there a workaround?
    Thanks,

    ReplyDelete
  18. alienSeptember 22, 2009 at 4:33 PM

    the SOAP API was working for me until just a few days ago :( (at seekng.com).. oh well, i use Yahoo, YouTube, Flickr, and MSN APIs, so not totally harmed at the moment

    ReplyDelete
  19. Video chatOctober 12, 2009 at 2:23 AM

    But the service is not the same like SOAP?

    ReplyDelete
  20. pulsorockFebruary 12, 2010 at 7:57 AM

    I know this is an old post, but I'm looking for more information. I'm developing a web application for a client of ours (a public figure in our country) where we'll monitor the mentions of this person on social sites (Twitter, Facebook, etc) and we'll also like to use this API to monitor his mentions on Google search results (such as News and Blogs mentions). Probably our process will run every 30 minutes with specify search terms regarding this person. Is that under the allowed features that can be used to? Would developing something like that get me banned from using the service? If I do, what other options could I have?

    ReplyDelete
  21. DhannyFebruary 16, 2010 at 4:48 AM

    You right, i thing yahoo boss and other application would be developer friendly than GASA (Google Ajax Search Api) but sometimes, I just have a dream if theres some (one or application) can gather data from GASA and Yapi.

    ReplyDelete
  22. custom web developmentMay 9, 2010 at 4:58 PM

    I fnid the API really difficult to use. instead you can use the following url:
    url = "http://www.google.com/ie?q=" + searchHTMLterm + "&num="+count.ToString()+"&hl=en&gl=uk&start="+startfrom.ToString()+"&sa=N&filter=0";


    I do believe there is a limit of how many requests you can send within 5 minutes or so (not sure, but you get a short term temp ban if you send too many within a short time frame)

    The API is far too limiting to me and it doesn't return the same results as you would get through searching via the web interface.

    pete

    ReplyDelete
  23. UnknownMay 27, 2010 at 7:26 AM

    I'm trying to figure out the where to send this url request to get back search results. I've used rest before in other applications but i guess I'm confused as where to find the docs for building air apps using as 3.

    ReplyDelete
  24. BeepdfSeptember 20, 2010 at 5:24 AM

    does anyone know how to solve the 403 forbidden error in google?

    ReplyDelete
  25. azload pdfOctober 25, 2010 at 7:33 PM

    i try to use google search api but found error

    ReplyDelete
  26. AnonymousJune 13, 2011 at 12:09 PM

    any new feedback. i seems working now. anyone using in a live system?

    ReplyDelete
  27. AnonymousDecember 21, 2011 at 9:16 AM

    Does anybody know what is the parameter to get search list from video.google.com?

    https://ajax.googleapis.com/ajax/services/search/video?v=1.0&q=earth%20day
    This always returns video from youtube. i want data from google video.

    Am i using wrong api? or it just a parameter to pass to get google videos?

    ReplyDelete
  28. Alex ChituDecember 21, 2011 at 9:20 AM

    Restrict the results to Google Video by adding site:video.google.con to the query.

    ReplyDelete
  29. AnonymousDecember 22, 2011 at 8:45 AM

    I will appriciate if you can give an example.

    I tried this
    https://ajax.googleapis.com/ajax/services/search/video?v=1.0&q=earth%20day&site=video.google.com

    but still it is fetching data from youtube only. i think "site" is not a valid parameter for this.

    ReplyDelete
  30. Alex ChituDecember 22, 2011 at 1:26 PM

    It's not a parameter, it's part of the query. site: is a Google operator.

    https://ajax.googleapis.com/ajax/services/search/video?v=1.0&q=earth%20day%20site:video.google.com

    ReplyDelete
  31. AnonymousDecember 22, 2011 at 11:23 PM

    Thank you Alex.
    It's working now.
    Also thank you for the explaination.

    ReplyDelete
  32. SamuelJune 13, 2012 at 11:06 AM

    I must know how to retrieve result with dates range...

    ReplyDelete

Note: Only a member of this blog may post a comment.