How to create a simple website crawler in PHP

Today, let’s talk about how you can create a simple website crawler in PHP. A website crawler, also known as a web spider, web robot, or simply crawler, is a program or automated script that systematically browses the World Wide Web, typically for the purpose of web indexing. Website crawlers navigate through the internet by following links on web pages, and they collect information about each webpage that they visit.

The information collected by a web crawler may include the content of the page, its HTML code, the page’s metadata, such as the title and description, and any links found on the page. This data can be used for a variety of purposes, such as indexing web pages for search engines, monitoring changes to a website, or analyzing website traffic.

Website crawlers can be created using a variety of programming languages and tools. They can range from simple scripts that visit a few web pages to complex systems that crawl millions of pages and process large amounts of data. Some examples of popular web crawlers include Googlebot, Bingbot, and Yahoo! Slurp.

Steps to follow

Creating a website crawler using PHP involves several steps:

1. Define the URLs to be crawled: Start by defining the URLs you want to crawl. This can be a single URL or a list of URLs.
2. Send HTTP requests: Use PHP’s cURL library to send HTTP requests to the URLs you defined. This will retrieve the HTML content of the page.
3. Parse the HTML content: Once you have the HTML content, use a library like DOMDocument to parse the HTML and extract the relevant information.
4. Store the data: Store the data you extracted in a database or in a file. You can use MySQL or SQLite to store data in a database.
5. Crawl additional pages: If the page contains links to additional pages, follow those links and repeat the process.

Creating a simple website crawler in PHP

Here is a sample code to get you started:

// Define the URLs to be crawled
$urls = [
  'https://example.com/page1',
  'https://example.com/page2',
  'https://example.com/page3',
];

// Loop through the URLs
foreach ($urls as $url) {
  // Send HTTP request
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  $html = curl_exec($ch);
  curl_close($ch);

  // Parse the HTML content
  $dom = new DOMDocument();
  @$dom->loadHTML($html);

  // Extract relevant information
  $title = $dom->getElementsByTagName('title')->item(0)->nodeValue;
  $meta = $dom->getElementsByTagName('meta')->item(0)->getAttribute('content');

  // Store the data
  $conn = new mysqli('localhost', 'username', 'password', 'database');
  $sql = "INSERT INTO pages (url, title, meta) VALUES ('$url', '$title', '$meta')";
  $conn->query($sql);
  $conn->close();
}

Note that this is just a basic example and you will need to customize the code to fit your specific requirements. Additionally, make sure to check the website’s terms of service and obtain any necessary permissions before crawling their pages.

Creating from sitemap XML file

To make things even easier and less time-consuming, you can decide to use your website’s sitemap file instead of having to manually define the URLs to be crawled by your crawler.

Creating a website crawler using a sitemap URL involves a slightly different approach compared to crawling individual URLs. Here are the basic steps involved:

1. Parse the sitemap: Use a library like SimpleXML to parse the sitemap XML and extract the URLs to be crawled.
2. Send HTTP requests: Use PHP’s cURL library to send HTTP requests to the URLs you extracted from the sitemap. This will retrieve the HTML content of the pages.
3. Parse the HTML content: Once you have the HTML content, use a library like DOMDocument to parse the HTML and extract the relevant information.
4. Store the data: Store the data you extracted in a database or in a file. You can use MySQL or SQLite to store data in a database.

Here is a sample code to get you started:

// Define the sitemap URL
$sitemap_url = 'https://example.com/sitemap.xml';

// Parse the sitemap
$xml = simplexml_load_file($sitemap_url);
$urls = [];
foreach ($xml->url as $url) {
    $urls[] = (string) $url->loc;
}

// Loop through the URLs
foreach ($urls as $url) {
  // Send HTTP request
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  $html = curl_exec($ch);
  curl_close($ch);

  // Parse the HTML content
  $dom = new DOMDocument();
  @$dom->loadHTML($html);

  // Extract relevant information
  $title = $dom->getElementsByTagName('title')->item(0)->nodeValue;
  $meta = $dom->getElementsByTagName('meta')->item(0)->getAttribute('content');

  // Store the data
  $conn = new mysqli('localhost', 'username', 'password', 'database');
  $sql = "INSERT INTO pages (url, title, meta) VALUES ('$url', '$title', '$meta')";
  $conn->query($sql);
  $conn->close();
}

Note that this is just a basic example, and you will need to customize the code to fit your specific requirements. Additionally, make sure to check the website’s terms of service and obtain any necessary permissions before crawling their pages.