How to log api requests made by page via puppeteer

To log api requests made by the browser using Puppeteer, you can intercept network requests and log their details using Puppeteer's page.setRequestInterception(true) method along with handling request and response events. Here’s how you can achieve this:

1. Enable Request Interception: Use page.setRequestInterception(true) to intercept all requests.

2. Log Requests: Inside the request interception callback, log the details of each request and optionally log the response details as well.

Here’s a step-by-step example:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Enable request interception
  await page.setRequestInterception(true);

  // Listen for requests
  page.on('request', request => {
    // Filter only XMLHttpRequests (XHR) or fetch requests
    if (request.resourceType() === 'xhr' || request.resourceType() === 'fetch') {
      // Log request URL and method
      console.log(`Request: ${request.method()} ${request.url()}`);

      // Optionally log request headers
      console.log('Request headers:', request.headers());

      // Continue with the request
      request.continue();
    } else {
      // Ignore other requests like CSS, JS, images, etc.
      request.continue();
    }
  });

  // Listen for responses
  page.on('response', response => {
    // Log response status and headers
    console.log(`Response: ${response.status()} ${response.url()}`);
    console.log('Response headers:', response.headers());
  });

  // Navigate to a website
  await page.goto('https://example.com');

  // Perform actions on the page that trigger network requests
  // ...

  // Close the browser
  await browser.close();
})();

Explanation:

• page.setRequestInterception(true): Enables request interception on the page.

• page.on('request', ...): Event listener for intercepted requests. Inside the callback function:

  • request.method() and request.url(): Retrieve the HTTP method and URL of the request.
  • request.headers(): Retrieve and log request headers.
  • request.continue(): Resumes the request, allowing it to proceed.

• page.on('response', ...): Event listener for responses. Inside the callback function:

  • response.status() and response.url(): Retrieve the HTTP status code and URL of the response.
  • response.headers(): Retrieve and log response headers.

• Example Usage: Replace await page.goto('https://example.com'); with your actual navigation or actions on the page that trigger network requests. All requests made during this navigation will be intercepted and logged.