programming zona

Web Scraping: Gathering "Related Searches" Keyword Data From Google Search Results

You might first want to read Basic Web
Scraping: Pulling Out Data From Google Search Results.

In the other Basic Web Scraping post we created a Simple Web Scraper that retrieved
Google’s Search Result Pages for some keyword and scraped the “Number
of Pages” indexed for the keyword. That was a good starting point in learning
web scraping, that’s why in this post we’re going to extend that
scraper to return one more information, “Related Keywords”.

Actually, Related Searches is a bunch of keywords that Google displays at the
bottom of search result pages. These are not displayed all the keywords that
you search for, but only for keywords that are somewhat broad. These “related
Searches” are the keywords that Google has evaluated to be related with
the one searched for.

To scrape this, we’d first need to analyze the HTML code of Google Search
Result. We need to find out where the Related Searches block is in the code,
typically it’d like:

<h2 class=r>Searches related to:<b> computer programming</b></h2><table 
  border=0 cellpadding=0 cellspacing=0 style="margin-top:6px"><tr 
  style="font-size:84%"><td style="padding:0 30px 6px 0" 
  valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+schools&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=1">computer 
  programming <b>schools</b></a></td><td>&nbsp;</td><td 
  style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+careers&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=2">computer 
  programming <b>careers</b></a></td><td>&nbsp;</td><td 
  style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+languages&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=3">computer 
  programming <b>languages</b></a></td><td>&nbsp;</td><td 
  style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+c%2B%2B&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=4">computer 
  programming <b>c++</b></a></td><td>&nbsp;</td></tr><tr 
  style="font-size:84%"><td style="padding:0 30px 6px 0" 
  valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+information&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=5">computer 
  programming <b>information</b></a></td><td>&nbsp;</td><td 
  style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=basic+computer+programming&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=6"><b>basic</b> 
  computer programming</a></td><td>&nbsp;</td><td 
  style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+for+dummies&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=7">computer 
  programming <b>for dummies</b></a></td><td>&nbsp;</td><td 
  style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=game+programming&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=8"><b>game</b> 
  programming</a></td><td>&nbsp;</td></tr></table>

Remember our goal here is to find out certain footprint or consistent block
in the code (which always will be on every page we request), out of which the
needed data (Related Searches) can be pulled out.

Here, we can safely assume it to be:

<h2 class=r>Searches related to:

and

</td></tr></table>

So if we do a search for the above block in the code, we can easily pull the
required information.

Now, we can follow the following steps to scrape the required information:

Retrieve the required page.

Look for the consistent part of code that uniquely encloses or identifies
the required information. Scrape that block.

Pull the required data out.

Following is the code listing:


<html>

<head>

<title>Google Result Scraper</title>

</head>



<body>

<p align="center" style="font-size:500%"><font color="#0000FF">G</font><font color="#FF0000">o</font><font color="#FFFF00">o</font><font color="#0000FF">g</font><font color="#00FF00">l</font><font color="#FF0000">e</font> 


  <font size="2"><br />Result Scraper</font></p>



<?php

$s=$_GET['s'];




if(isset($s))

{

    //*******F I R S T   P A R T*********

    //Find the number of pages indexed for the searched term


    //*From previous part

    echo "<p><i>Search for $s</i></p>";



    $s=urlencode($s);




    $main_data=file_get_contents("http://www.google.com/search?hl=en&q=".$s."&btnG=Google+Search");

    

    //strip off HTML


    $data=strip_tags($main_data);

    //now $data only has text NO HTML



    //these have to ound out in the fetched data


    $find='Results 1 - 10 of about ';

    $find2=' for';



    //have text beginning from $find


    $data=strstr($data,$find);



    //find position of $find2

    //there might be many occurence


    //but it'd give position of the first one, 

    //which is what we want, anyway

    $pos=strpos($data,$find2);




    //take substring out, which'd be the number we want

    $search_number=substr($data,strlen($find), $pos-strlen($find));




    echo "Pages Indexed: $search_number";

    

    

    //********S E C O N D   P A R T*******


    //Find related searches

    

    echo "<h3>Related Keywords</h3>";



    //these have to found out in the fetched data


    $find='<h2 class=r>Searches related to:';

    $find2='</table>';

    $find3='<table border=0';




    //have text beginning from $find

    $data=strstr($main_data,$find);



    //find position of $find2


    //there might be many occurence

    //but it'd give position of the first one, 

    //which is what we want, anyway


    $pos=strpos($data,$find2);



    //take substring out, this is the block of data


    //that we'd fetch required information off of.

    $related_search_block=substr($data,strlen($find), $pos-strlen($find));


    

    //pull out the required data, stripping off "Searches related to.."

    $related_searches=strstr($related_search_block,$find3);


    //now we only have the keywords, of course formatted as is

    //with tables

    

    //strip off HTML, therefore taking off table and various


    //other formattings

    $related_searches=strip_tags($related_searches);

    //now we have the keywords (plain text) sepearted with 


    //each other using "&nbsp;'

    

    //explode it

    $keywords=explode('&nbsp;',$related_searches);


    //now we a nice array having all the keywords

    

    //print the keywords

    //nicely formatted using <ol> (ordered list)


    echo "<ol>";

    foreach($keywords as $keyword)

    {

        if($keyword!='' && $keyword!=' ')


        echo "<li>

                $keyword

             </li>";

    }

    echo "</ol>";    

}

else


{

?>



<form name="form1" id="form1" method="get" action="">

  <div align="center">

    <p> 

      <input name="s" type="text" id="s" size="50" />


      <input type="submit" name="Submit" value="Find Related Keywords" />

    </p>

  </div>

</form>

<p>&nbsp;</p>


<p>&nbsp;</p>

<p>&nbsp;</p>

<p>&nbsp;</p>

<p>

  <?php

}

?>


</p>

<p align="right"><font size="2">by <a href="http://learning-computer-programming.blogspot.com/">Learning 

  Computer Programming</a></font></p>


</body>

</html>

Although this script is not very useful as it is (after all it’s meant
for learning purpose) you can make it more useful by extending it. one thing
that’d be nice is to hav it search for related keywords and again take
each keyword as a base for further searches. This way it’d be able to
supply you with many keywords related to the one entered. Best thing is, the
keywords will be Googly Related!

Previous Posts:

Web
Scraping: Pulling Out Data From Google Search Results

Creating
a Simple Form Validation Script Using JavaScript

Deigning
a Simple HTML Menu-Bar Using CSS

Blogger:
Method to Add ‘Link to This Post’ Widget on Your Posts

New
Working Method of Making Blogger Title Tags SEO Friendly

Basic Web Scraping: Scraping Google Search Results

[UPDATE (22-AUG-2009): THIS IS THE NEW WORKING VERSION.]

Today we are going to discuss a bit advanced topic, not in the sense that it’d
be difficult to understand (I always try to make things easier anyway) but that
you won’t find an apparent use of it. What we are going to do today is
what is called Web Scraping. By the way web scraping means retrieving data from
web and pulling out useful information out of it for our use. Of course this
wouldn’t be the next best web scraper rather it would la a basic foundation
on how simple a web scraper can be.

OK let’s kick off guys!

As is obvious we are going to scrape Google’s Web Search Results to retrieve
the number of pages indexed for a search term.

Scarping Google Results for Number of Pages Indexed

To retrieve results for a search term we need the URL, for this fire up your
favorite Browser and browse to the Search Engine’s (Google, or whatever)
homepage, type in any search query and hit enter.

OK now look at the address bar, in my case I looked like below, your’
should be similar or whatever:

http://www.google.com/search?hl=en&q=learning+c&btnG=Google+Search

On inspection you can see pour search term in the URL which is ‘URL Encoded’
(changes some character such as spaces to codes). There we have it, you can
place any search keyword (urlencoded, very simple with PHP’s in-built
function) and fetch that page. But how in a script, you might ask. Because that
is what we need.

Well using the following function:

file_get_contents();

[UPDATE: WE'LL BE USING THE FOLLOWING USER-DEFINED FUNCTION INSTEAD. READ COMMENTS FOR MORE INFORMATION:


function my_fetch($url,$user_agent='Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)')


{

    $ch = curl_init();

    curl_setopt ($ch, CURLOPT_URL, $url);

    curl_setopt ($ch, CURLOPT_USERAGENT, $user_agent);


    curl_setopt ($ch, CURLOPT_HEADER, 0);

    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

    curl_setopt ($ch, CURLOPT_REFERER, 'http://www.google.com/');


    $result = curl_exec ($ch);

    curl_close ($ch);

    return $result;

}

]

If you have been following this blog for sometime, you might remember we once
used it in my Creating
a Simple Shout Box in PHP post to fetch contents from a local file.
Yeah its beauty is that it can fetch remote (HTTP) files too.

$data = file_get_contents("http://www.google.com/search?hl=en&q=learning+c&btnG=Google+Search");

[UPDATE: NOW USING:

$data = my_fetch("http://www.google.com/search?hl=en&q=learning+c&btnG=Google+Search");

]

Above code will fetch the Google Search Results for the keyword we searched
for in the browser, $data will contain the HTML source.

Since we have to scrape the total number of pages indexed for a particular
search term (displayed as “Results 1 - 10 of about XXXX …”)
we would find some text near that number(XXXX in this case). In this case that
text is simply “Results 1 - 10 of about”, its also unique throughout
the page hence if we could find it in the code returned we can easily find the
needed data. One more thing we can ease off searching by first stripping off
HTML from the code returned so that only text remains. This part can be implemented
as below:

$data=my_fetch("http://www.google.com/search?hl=en&q=".$s."&btnG=Google+Search"); //strip off HTML $data=strip_tags($data); $find='Results 1 - 10 of about '; $find2=' for'; //have text beginning from $find $data=strstr($data,$find); //find position of $find2 $pos=strpos($data,$find2); //take substring out, which'd be the number we want $search_number=substr($data,strlen($find), $pos-strlen($find));

Here is the complete code:


<html>

<head>

<title>Google Result Scraper</title>

</head>



<body>

<p align="center" style="font-size: 500%"><font color="#0000FF">G</font><font


     color="#FF0000">o</font><font color="#FFFF00">o</font><font

     color="#0000FF">g</font><font color="#00FF00">l</font><font

     color="#FF0000">e</font><font size="2"><br />


Result Scraper</font></p>



<?php

function my_fetch($url,$user_agent='Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)')


{

$ch = curl_init();

curl_setopt ($ch, CURLOPT_URL, $url);

curl_setopt ($ch, CURLOPT_USERAGENT, $user_agent);


curl_setopt ($ch, CURLOPT_HEADER, 0);

curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt ($ch, CURLOPT_REFERER, 'http://www.google.com/');


$result = curl_exec ($ch);

curl_close ($ch);

return $result;

}



$s = $_GET['s'];


if (isset($s))

{

echo "<p><i>Search for $s</i></p>";

    $s = urlencode($s);


    $data = my_fetch("http://www.google.com/search?hl=en&q=" . $s . "&btnG=Google+Search");

    //strip off HTML

    $data = strip_tags($data);


    //now $data only has text NO HTML

    //these have to found out in the fetched data

    $find = 'Results 1 - 10 of about ';


    $find2 = ' for';

    //have text beginning from $find

    $data = strstr($data, $find);


    //find position of $find2

    //there might be many occurence

    //but it'd give position of the first one,


    //which is what we want, anyway

    $pos = strpos($data, $find2);



//take substring out, which'd be the number we want


$search_number=substr($data,strlen($find), $pos-strlen($find));




echo "Pages Indexed: $search_number";

}

else

{

    ?>



<form name="form1" id="form1" method="get" action="">


<div align="center">

<p>  <input name="s" type="text" id="s" size="50" />

<input type="submit" name="Submit" value="Count" /></p>


</div>

</form>

<p>&nbsp;</p>

<p>&nbsp;</p>

<p>&nbsp;</p>

<p>&nbsp;</p>


<p>

<?php

}

?>

</p>

<p align="right"><font size="2">by <a

     href="http://learning-computer-programming.blogspot.com/">Learning

Computer Programming</a></font></p>


</body>

</html>

Wow, our first scarper is completed. It has a nice interface, you type in search
phrase click ‘Count’ and there you are. It displays the number of
pages that contains that term same as on Google.

Have fun guys and do comment!

P.S.: You might want to read String
Manipulation Function in PHP I and String
Manipulation Function in PHP II if you are not much familiar with the
string manipulation functions we are using in the code above.

Previous Posts: