Writing a Simple Site Crawler

Introduction

Well, it’s been a long time since I’ve written a post on my own blog post, in fact over 6 months! A lot has happened in my life since then so if you know me and want to know more then drop me a tweet (http://twitter.com/mattstannard/) or e-mail matt@mattstannard.com.

The last thing I was blogging about was writing some PHP to ping your Facebook Wall and then read wall updates and reply to them automatically. I will continue
part two of this blog, but since then I’ve written numureous other scripts to Monitor Twitter and Crawl Websites so I thought I’d share some of this.

So, what is this script for and why would I write it? Well, I wanted something that would find the internal links on a website and download the HTML content so I could run some processing on it at a later date. Obviously, I could do this real time but I wanted something that I could work with offline should the need
arise hence the download to MySQL.

The Crawler

The first part of the page I wrote was actually a form so I could enter any URL and crawl it, however I am not going to include this – it’s easy enough to change and I don’t want to encourage people to randomly crawl others sites.

Once I had a URL to crawl, I used the PHP DOMDocument to pull the content from the website. This gave me the HTML and allowed me to use some built in methods to select all of the links on a page (again, you can modify the script to parse JavaScript etc. if needed). Once I had my base page, there were several challenges I had to overcome:

• Ensure the crawler didn’t go off site – I didn’t really want it indexing Facebook, or indeed any other external content it would fill up my laptop
• Ensure the crawler didn’t get stuck

For the first, I created a method to extracted the Base URL, if the URL was not relative to the root (i.e. it contained http://) I checked the domain matched the current sites URL. If it didn’t I didn’t index it.

For the second, I cheated slightly. I used MySQL and made the page being indexed a unique key. When a page was crawled, all links were put into MySQL with a crawled flag of “0″ (i.e. they need to be crawled). If I had already “seen” the URL then it wouldn’t be added again and thus the list of pages to crawl would get smaller.

Finally, I added a MAX_DEPTH constact to ensure that the cralwer only went to a depth of 10 pages, this would prevent it getting stuck in the “depths” of a site.

This actually meant that the logic of my recursive function was actually a lot more simplistic than I thought it would be:

• Take a URL to Crawl and my Current Depth
• Load the page and get all of the hyperlinks
• For each hyperlink on the page
  • Check its not a bookmark, javascript or link off site
  • Add it to the list of pages to index
• Next Link
• Select all uncrawled links in database and for each link
  • Set the crawl state to crawled
  • Increment Depth and Call Function Recursively
• Finish

This seemed to run perfectly which was great, I will be tweaking and reviewing its performance, perhaps multi-threading it in the future and as always any feedback or
questions are welcomed.

Code

set_time_limit(0);

function GetDomainFromURL($URL)
{
	$strURL = $URL;

	// We don't want http://
	$strURL = str_replace("http://","",$strURL);

	// Then we want the left of the first /
	if (!(strpos($strURL,"/") === false))
	{
		$strURL = substr($strURL,0,strpos($strURL,"/")-1);
	}

	$strURL = trim($strURL);
	return($strURL);

}

function CrawlPage($BaseURL,$iDepth,$resMySQL)
{
	$intSanity;

	$intSanity = 0;
	$strURL = $BaseURL;
	$strBase = "";

	if ($iDepth > MAX_DEPTH)
	{
		return(0);
	}

	echo str_repeat("  &nbsp",$iDepth);
	echo "Entered CrawlDomain
";

	// Get the base domain
	$strBase = GetDomainFromURL($BaseURL);

	// Define an XML Document and load the page into
	$doc = new DOMDocument();
	@$doc->loadHTMLFile($strURL);

	// Lets build up a list of pages that are linked
	foreach($doc->getElementsByTagName('a') as $link)
	{
		$href = $link->getAttribute('href');

		// echo "Found: " . $href . " (" . strpos($href,"http://" . $strBase) . ")
";

		// If the link starts with a / we want to add the top level domain
		// We don't want to crawl anything which is a bookmark (i.e. a #)
		if (substr($href,0,1) != "#" && (strpos($href,"javascript:") === false))
		{
			if (substr($href,0,1) == "/")
			{
				$href = "http://" . $strBase . $href;
				// echo "Link:" . $href . "
";
				$strSQL = "INSERT INTO tblcrawllist (SessionID,CrawlURL,FirstParent,PageContent,CrawlLevel,Crawled) VALUES ('" . session_id() . "','" . mysql_real_escape_string($href) . "','" . mysql_real_escape_string($strURL) . "','" . mysql_real_escape_string($doc->saveHTML()) . "',$iDepth,0)";
				mysql_query($strSQL,$resMySQL);	

			}
			elseif(!(strpos($href,"http://" . $strBase) === false))
			{
				// Add this to the list of found URLs for this domain
				// echo "Link:" . $href . "
";
				$strSQL = "INSERT INTO tblcrawllist (SessionID,CrawlURL,FirstParent,PageContent,CrawlLevel,Crawled) VALUES ('" . session_id() . "','" . mysql_real_escape_string($href) . "','" . mysql_real_escape_string($strURL) . "','" . mysql_real_escape_string($doc->saveHTML()) . "',$iDepth,0)";
				mysql_query($strSQL,$resMySQL);	

			}

		} // End Validate HREF is valid
	} // End ForEach Link

	// Now let's go and crawl the pages we found
	$strSQL = "SELECT SessionID,CrawlURL,CrawlLevel FROM tblcrawllist WHERE FirstParent='" .  mysql_real_escape_string($strURL) . "' AND Crawled='0'";
	$resSQL = mysql_query($strSQL,$resMySQL);

	while ($row = mysql_fetch_array($resSQL))
	{
		$strSQL = "UPDATE tblcrawllist SET Crawled='1' WHERE CrawlURL='" . mysql_real_escape_string($row[1]) . "'";
		mysql_query($strSQL);
		$newDepth = ($iDepth + 1);
		echo str_repeat("  &nbsp",$iDepth);
		echo $href . " -> (" . $newDepth . ")
";
		CrawlPage($row[1],$newDepth,$resMySQL);
	}

	return(1);
} // End GetDomainFromURL