How to capture api response time and headers using puppeteer

To capture API request details, including response time, headers, and payload, using Puppeteer, you'll need to leverage the request interception mechanism and handle requests appropriately. Puppeteer itself doesn't directly expose API response times or headers, but you can calculate timings and access headers through intercepted requests. Here's a structured approach to achieve this:

1. Enable Request Interception: Set up Puppeteer to intercept network requests.

2. Capture Request Details: Extract relevant information from intercepted requests.

3. Calculate Response Time: Compute the response time based on request initiation and response completion.

Here's how you can implement this:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Enable request interception
  await page.setRequestInterception(true);

  // Array to store API request details
  const apiRequests = [];

  // Event listener for request interception
  page.on('request', interceptedRequest => {
    // Filter requests based on URL or other criteria
    if (interceptedRequest.url().includes('/api/')) {
      const requestStartTime = new Date();

      // Continue intercepting the request
      interceptedRequest.continue().then(() => {
        // Event listener for response interception
        interceptedRequest.response().then(response => {
          const requestEndTime = new Date();
          const requestDuration = requestEndTime - requestStartTime;

          // Extract response headers and payload
          const headers = response.headers();
          response.text().then(body => {
            apiRequests.push({
              url: interceptedRequest.url(),
              method: interceptedRequest.method(),
              status: response.status(),
              headers: headers,
              payload: body,
              responseTime: requestDuration,
            });
          });
        });
      });
    } else {
      // Continue all other requests normally
      interceptedRequest.continue();
    }
  });

  // Navigate to a website that triggers API requests
  await page.goto('https://example.com');

  // Wait for a few seconds to capture network requests
  await page.waitForTimeout(5000); // Adjust as needed

  // Display captured API requests
  console.log('Captured API Requests:', apiRequests);

  await browser.close();
})();

Explanation:

• Enable Request Interception: Use page.setRequestInterception(true) to intercept all network requests made by the page.

• Event Handling: Use page.on('request', ...) to listen for intercepted requests. Inside the callback function:

  • Check if the request URL matches your API endpoint criteria (e.g., /api/).
  • If it matches, record the start time (requestStartTime).
  • Continue intercepting the request using interceptedRequest.continue().

• Response Handling: Inside interceptedRequest.response().then(response => { ... }), wait for the response and then:

  • Calculate the request duration (requestDuration) as the difference between requestStartTime and current time (requestEndTime).
  • Extract response headers using response.headers().
  • Retrieve response payload (body) using response.text().

• Store API Requests: Store relevant details (URL, method, status, headers, payload, response time) in the apiRequests array.

• Navigation and Waiting: Use page.goto() to navigate to a website that triggers API requests. Adjust the URL to match the actual website.

  • Use page.waitForTimeout() to wait for sufficient time to capture network requests. Adjust the timeout value based on the expected response times and network conditions.

• Close Browser: Ensure to close the browser instance using browser.close() to release resources after completing the task.