PUPPETEER Tutorial Course

Introduction to Puppeteer

Overview of Puppeteer

Puppeteer is a Node.js library that provides a high-level API to control headless Chrome or Chromium browsers programmatically. It's maintained by the Google Chrome team and is widely used for web scraping, automated testing, and other browser automation tasks.

Key Features
  • Headless Browser Automation: Puppeteer runs in headless mode by default, meaning it doesn't have a graphical user interface (GUI), making it faster and more efficient for automation tasks. It can also be run in "headful" mode to visually observe what the browser is doing.
  • Web Scraping: Puppeteer is excellent for web scraping. It allows you to navigate pages, interact with elements, and extract data from websites. Its ability to execute JavaScript on pages ensures that you can interact with dynamic content that might be generated by JavaScript frameworks.
  • Automated Testing: Puppeteer is widely used in testing frameworks to automate browser testing. It can simulate user interactions like clicks, typing, and form submissions, making it a powerful tool for end-to-end testing.
  • PDF and Screenshot Generation: Puppeteer can take screenshots of web pages or generate PDFs. This is particularly useful for creating visual reports or capturing the state of a webpage at a specific time.
  • Advanced Interactions: Puppeteer can handle complex interactions like filling out forms, handling file uploads, waiting for specific elements to appear, and even intercepting network requests.
  • Cross-Browser Testing: While Puppeteer is primarily for Chrome/Chromium, it also has the capability to work with Firefox via the Puppeteer-Firefox project, making it a versatile tool for cross-browser testing.
  • Built-in Support for Chrome DevTools: Puppeteer provides a programmatic way to use the same debugging features that developers use in Chrome DevTools, such as inspecting elements, monitoring network activity, and more.
  • Scripted Navigation: Puppeteer allows you to automate navigation across multiple pages, making it easy to create scripts that interact with multi-page workflows.
Use Cases
  • End-to-End Testing: Automate the testing of user interactions and flows on websites, ensuring that your web application behaves as expected.
  • Web Scraping: Extract data from websites, even those with dynamic content generated by JavaScript.
  • Performance Monitoring: Measure page load times, render times, and other performance metrics by automating the collection of these data points.
  • Automated Form Submission: Automate the filling and submission of web forms, useful in scenarios like testing, bot creation, and data entry.
  • SEO and Accessibility Audits: Analyze pages for SEO and accessibility compliance by leveraging Puppeteer's ability to interact with and inspect the DOM.

Installation and Setup

Installation and Setup of Puppeteer in VS Code

Here's a step-by-step guide to installing and setting up Puppeteer in Visual Studio Code (VS Code):

1. Install Node.js

Before you can use Puppeteer, you need to have Node.js installed on your system. Puppeteer is a Node.js library, so it requires Node.js to run.

  • Download and install Node.js from the official website.
  • After installation, you can verify the installation by running the following commands in your terminal:
1node -v
2npm -v

These commands should return the installed versions of Node.js and npm (Node Package Manager).

2. Set Up a New Project
  1. Open Visual Studio Code.
  2. Create a new directory for your project, and open it in VS Code.
  3. Open the terminal in VS Code (Ctrl + \`) and initialize a new Node.js project by running:
npm init -y

This command creates a package.json file with default settings.

3. Install Puppeteer

Now that your Node.js project is set up, you can install Puppeteer.

  1. Run the following command in the terminal to install Puppeteer:
npm install puppeteer

This command will download and install Puppeteer, including a compatible version of Chromium.

4. Create a Script to Use Puppeteer
  1. In your project directory, create a new JavaScript file, e.g., index.js.
  2. Open the index.js file in VS Code and add the following code as a basic Puppeteer script:
1const puppeteer = require('puppeteer');
2
3(async () => {
4// Launch a new browser instance
5const browser = await puppeteer.launch();
6// Open a new page
7const page = await browser.newPage();
8// Navigate to a website
9await page.goto('https://example.com');
10// Take a screenshot
11await page.screenshot({ path: 'example.png' });
12// Close the browser
13await browser.close();
14})();

This script does the following:

  • Launches a new headless Chrome/Chromium browser.
  • Opens a new page.
  • navigates to https://example.com.
  • Takes a screenshot of the page and saves it as example.png.
  • Closes the browser.
5. Run the Puppeteer Script
  1. Save the index.js file.
  2. In the terminal, run the script by executing:
node index.js

After the script runs, you should see a new file called example.png in your project directory. This file is a screenshot of the webpage.

6. Explore More Puppeteer Features

Now that Puppeteer is set up, you can start exploring its features, such as:

  • Automating user interactions: Clicks, typing, form submissions, etc.
  • Web scraping: Extracting data from web pages.
  • Generating PDFs: Creating PDFs from web pages.
  • Testing: Writing end-to-end tests for your web applications.

You can refer to the official Puppeteer documentation for more advanced usage and examples.

Page Interactions

Page Navigation

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch();
6    const page = await browser.newPage();
7
8    // Navigate to the first URL
9    await page.goto('https://example.com');
10
11    // Perform an action on the first page, such as clicking a link
12    await page.click('a'); // This example assumes there is at least one link on the page
13
14    // Wait for navigation to complete
15    await page.waitForNavigation();
16
17    // Print the current URL
18    console.log('Current URL:', page.url());
19
20    // Close the browser
21    await browser.close();
22})();

Clicking and Typing

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch();
6    const page = await browser.newPage();
7
8    // Navigate to a webpage with form elements
9    await page.goto('https://example.com'); // Replace with the URL of a page with input fields
10
11    // Click on a button or link
12    await page.click('#button-id'); // Replace with the selector of the element you want to click
13
14    // Type text into an input field
15    await page.type('#input-id', 'Sample text'); // Replace with the selector of the input field and the text you want to type
16
17    // Submit a form (if applicable)
18    await page.click('#submit-button-id'); // Replace with the selector of the form submit button
19
20    // Wait for navigation or for an element to appear
21    await page.waitForNavigation(); // Use this if clicking or typing leads to navigation
22    // Or use this if you are waiting for a specific element
23    // await page.waitForSelector('#some-element-id');
24
25    // Optionally, take a screenshot to verify results
26    await page.screenshot({ path: 'result.png' });
27
28    // Close the browser
29    await browser.close();
30})();

Form Handling

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch();
6    const page = await browser.newPage();
7
8    // Navigate to the page with the form
9    await page.goto('https://example.com/form'); // Replace with the URL of the page with your form
10
11    // Fill out the form
12    await page.type('#name', 'John Doe'); // Replace #name with the selector for the name input field
13    await page.type('#email', '[email protected]'); // Replace #email with the selector for the email input field
14    await page.type('#message', 'Hello, this is a test message.'); // Replace #message with the selector for the message textarea
15
16    // Submit the form
17    await page.click('#submit'); // Replace #submit with the selector for the submit button
18
19    // Optionally, wait for navigation or confirmation
20    await page.waitForNavigation(); // Use if form submission causes navigation
21    // Or wait for an element indicating form submission success
22    // await page.waitForSelector('#success-message'); // Replace with a selector for success message
23
24    // Capture a screenshot of the result page
25    await page.screenshot({ path: 'form-submission-result.png' });
26
27    // Close the browser
28    await browser.close();
29})();

Network Interception

Intercepting Requests

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch();
6    const page = await browser.newPage();
7
8    // Intercept network requests
9    await page.setRequestInterception(true);
10
11    page.on('request', request => {
12        // Log intercepted requests
13        console.log('Intercepted request:', request.url());
14
15        // Modify request or response here
16        // For example, abort a request
17        // if (request.url().includes('example.com/api')) {
18        //     request.abort();
19        // } else {
20        //     request.continue();
21        // }
22
23        // To modify response, use request.respond()
24        if (request.url().includes('example.com/api')) {
25            request.respond({
26                status: 200,
27                contentType: 'application/json',
28                body: JSON.stringify({ message: 'This is a mocked response' })
29            });
30        } else {
31            request.continue(); // Continue with the original request if not intercepted
32        }
33    });
34
35    // Navigate to the page
36    await page.goto('https://example.com'); // Replace with your target URL
37
38    // Optionally, wait for specific elements or interactions
39    await page.waitForSelector('h1'); // Replace with a selector relevant to your page
40
41    // Take a screenshot of the page
42    await page.screenshot({ path: 'intercepted-request.png' });
43
44    // Close the browser
45    await browser.close();
46})();

Modifying Requests and Responses

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch();
6    const page = await browser.newPage();
7
8    // Enable request interception
9    await page.setRequestInterception(true);
10
11    // Intercept and modify requests and responses
12    page.on('request', request => {
13        // Log the URL of the intercepted request
14        console.log('Intercepted request:', request.url());
15
16        // Example: Modify a request URL
17        if (request.url().includes('example.com/api')) {
18            // Optionally, modify request headers
19            request.continue({
20                headers: {
21                    ...request.headers(),
22                    'X-Modified-Header': 'ModifiedValue'
23                }
24            });
25        } else {
26            request.continue(); // Continue with the original request
27        }
28    });
29
30    page.on('response', async response => {
31        // Log the URL of the intercepted response
32        console.log('Intercepted response:', response.url());
33
34        // Example: Modify a response body
35        if (response.url().includes('example.com/api')) {
36            const responseBody = await response.text();
37
38            // Modify the response body
39            const modifiedBody = responseBody.replace('original value', 'modified value');
40
41            // Respond with the modified body
42            await response.respond({
43                status: response.status(),
44                contentType: response.headers()['content-type'],
45                body: modifiedBody
46            });
47        }
48    });
49
50    // Navigate to the page
51    await page.goto('https://example.com'); // Replace with your target URL
52
53    // Optionally, wait for specific elements or interactions
54    await page.waitForSelector('h1'); // Replace with a selector relevant to your page
55
56    // Take a screenshot of the page
57    await page.screenshot({ path: 'modified-request-response.png' });
58
59    // Close the browser
60    await browser.close();
61})();

Spoofing User Agents

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch();
6    const page = await browser.newPage();
7
8    // Set a custom user agent
9    const customUserAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36';
10    await page.setUserAgent(customUserAgent);
11
12    // Optionally, set other browser properties like viewport or language
13    await page.setViewport({ width: 1280, height: 800 });
14    await page.setExtraHTTPHeaders({
15        'Accept-Language': 'en-US,en;q=0.9'
16    });
17
18    // Navigate to a webpage
19    await page.goto('https://www.whatismybrowser.com/'); // Replace with your target URL
20
21    // Optionally, wait for specific elements or interactions
22    await page.waitForSelector('h1'); // Replace with a selector relevant to your page
23
24    // Take a screenshot to verify the user agent change
25    await page.screenshot({ path: 'user-agent-spoofed.png' });
26
27    // Close the browser
28    await browser.close();
29})();

File Handling

Uploading Files

1const puppeteer = require('puppeteer');
2const path = require('path');
3
4(async () => {
5    // Launch a new browser instance
6    const browser = await puppeteer.launch();
7    const page = await browser.newPage();
8
9    // Navigate to the page with the file upload form
10    await page.goto('https://example.com/upload'); // Replace with the URL of the file upload form
11
12    // Define the path to the file you want to upload
13    const filePath = path.resolve(__dirname, 'file-to-upload.txt'); // Replace with the path to your file
14
15    // Select the file input element and upload the file
16    const inputUploadHandle = await page.$('input[type=file]'); // Replace with the selector for your file input
17    await inputUploadHandle.uploadFile(filePath);
18
19    // Optionally, submit the form or click a button to complete the upload
20    await page.click('#submit-button'); // Replace with the selector for the submit button
21
22    // Wait for navigation or confirmation
23    await page.waitForNavigation(); // Use if the upload causes a page navigation
24    // Or wait for an element indicating upload success
25    // await page.waitForSelector('#success-message'); // Replace with the selector for a success message
26
27    // Optionally, take a screenshot to verify the result
28    await page.screenshot({ path: 'upload-result.png' });
29
30    // Close the browser
31    await browser.close();
32})();

Downloading Files

1const puppeteer = require('puppeteer');
2const fs = require('fs');
3const path = require('path');
4
5(async () => {
6    // Launch a new browser instance
7    const browser = await puppeteer.launch();
8    const page = await browser.newPage();
9
10    // Set up a download behavior
11    const downloadPath = path.resolve(__dirname, 'downloads');
12    if (!fs.existsSync(downloadPath)) {
13        fs.mkdirSync(downloadPath);
14    }
15
16    // Use a custom download folder
17    await page._client.send('Page.setDownloadBehavior', {
18        behavior: 'allow',
19        downloadPath: downloadPath
20    });
21
22    // Navigate to the page with the download link
23    await page.goto('https://example.com/download'); // Replace with the URL of the download page
24
25    // Click the download link or button
26    await page.click('#download-button'); // Replace with the selector for your download link/button
27
28    // Wait for the download to complete
29    // You might need to adjust the wait time based on the expected download duration
30    await page.waitForTimeout(5000); // Adjust timeout as necessary
31
32    // Check if the file has been downloaded
33    const files = fs.readdirSync(downloadPath);
34    if (files.length > 0) {
35        console.log('Downloaded files:', files);
36
37        // Optionally, read and process the downloaded file
38        const downloadedFile = path.join(downloadPath, files[0]);
39        console.log('Downloaded file path:', downloadedFile);
40
41        // Example: Read file contents (if it's a text file)
42        const fileContents = fs.readFileSync(downloadedFile, 'utf8');
43        console.log('File contents:', fileContents);
44    } else {
45        console.log('No files downloaded.');
46    }
47
48    // Close the browser
49    await browser.close();
50})();

Screenshots and PDFs

Taking Screenshots

1const puppeteer = require('puppeteer');
2
3(async () => {
4// Launch a new browser instance
5const browser = await puppeteer.launch();
6const page = await browser.newPage();
7
8// Navigate to a URL
9await page.goto('https://example.com'); // Replace with the URL of the page you want to capture
10
11// Optionally, wait for a specific element to be loaded
12// await page.waitForSelector('#element-id'); // Replace with a selector if needed
13
14// Take a screenshot of the entire page
15await page.screenshot({ path: 'fullpage-screenshot.png', fullPage: true });
16
17// Take a screenshot of a specific element
18const element = await page.$('#element-id'); // Replace with the selector of the element you want to capture
19if (element) {
20  await element.screenshot({ path: 'element-screenshot.png' });
21}
22
23// Close the browser
24await browser.close();
25})();

Capturing Specific Elements

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch();
6    const page = await browser.newPage();
7
8    // Navigate to the webpage
9    await page.goto('https://example.com'); // Replace with your target URL
10
11    // Wait for the specific element to be present
12    await page.waitForSelector('#specific-element'); // Replace with the selector for your target element
13
14    // Select the element
15    const element = await page.$('#specific-element'); // Replace with the selector for your target element
16
17    if (element) {
18        // Capture a screenshot of the specific element
19        await element.screenshot({ path: 'element-screenshot.png' });
20        console.log('Screenshot of the specific element has been saved as element-screenshot.png');
21    } else {
22        console.log('Element not found.');
23    }
24
25    // Close the browser
26    await browser.close();
27})();

Saving Pages as PDF

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch();
6    const page = await browser.newPage();
7
8    // Navigate to the webpage
9    await page.goto('https://example.com', { waitUntil: 'networkidle2' }); // Replace with your target URL
10
11    // Define the path where the PDF will be saved
12    const pdfPath = 'page-snapshot.pdf';
13
14    // Save the page as a PDF
15    await page.pdf({
16        path: pdfPath,
17        format: 'A4', // Paper format (A4, Letter, etc.)
18        printBackground: true, // Include background graphics
19        margin: { // Page margins
20            top: '20mm',
21            right: '20mm',
22            bottom: '20mm',
23            left: '20mm'
24        }
25    });
26
27    console.log(`PDF saved as ${pdfPath}`);
28
29    // Close the browser
30    await browser.close();
31})();

Browser Options and Settings

Headless vs Headful Modes

Headless Mode runs the browser without a graphical user interface (GUI). This mode is typically used for automation tasks, such as web scraping, testing, and generating PDFs. Headful Mode runs the browser with a graphical user interface. This mode is useful for visual debugging and scenarios where you need to interact with the browser manually.
1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch(); // Headless mode by default
5    const page = await browser.newPage();
6    await page.goto('https://example.com');
7    await browser.close();
8})();

Opening DevTools

Using DevTools with Puppeteer can be extremely helpful for debugging and inspecting web pages. Puppeteer provides a way to interact with the Chrome DevTools Protocol directly, allowing you to use DevTools features programmatically.
1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch the browser in headful mode and open DevTools
5    const browser = await puppeteer.launch({ headless: false, devtools: true });
6    const page = await browser.newPage();
7    await page.goto('https://example.com');
8
9    // Optional: Wait for some time to manually inspect
10    await page.waitForTimeout(30000); // Wait for 30 seconds
11
12    await browser.close();
13})();

Using devtools protocol directly

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch({ headless: false });
6    const page = await browser.newPage();
7
8    // Access the DevTools Protocol
9    const client = await page.target().createCDPSession();
10
11    // Enable the network domain
12    await client.send('Network.enable');
13
14    // Listen for network requests
15    client.on('Network.requestWillBeSent', (event) => {
16        console.log('Request:', event.request.url);
17    });
18
19    // Navigate to a webpage
20    await page.goto('https://example.com');
21
22    // Perform actions and use DevTools features
23    await page.evaluate(() => {
24        // Example: Log the document title
25        console.log(document.title);
26    });
27
28    // Close the browser
29    await browser.close();
30})();

Capturing network requests

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch({ headless: false });
6    const page = await browser.newPage();
7
8    // Access the DevTools Protocol
9    const client = await page.target().createCDPSession();
10
11    // Enable the network domain
12    await client.send('Network.enable');
13
14    // Listen for network requests
15    client.on('Network.requestWillBeSent', (event) => {
16        console.log('Request URL:', event.request.url);
17    });
18
19    // Navigate to a webpage
20    await page.goto('https://example.com');
21
22    // Close the browser
23    await browser.close();
24})();

Emulate Devices

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch({ headless: false });
6    const page = await browser.newPage();
7
8    // Access the DevTools Protocol
9    const client = await page.target().createCDPSession();
10
11    // Emulate a device
12    await client.send('Emulation.setDeviceMetricsOverride', {
13        width: 360,
14        height: 640,
15        deviceScaleFactor: 2,
16        mobile: true
17    });
18
19    // Navigate to a webpage
20    await page.goto('https://example.com');
21
22    // Close the browser
23    await browser.close();
24})();

Incognito Mode

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch({ headless: false });
6
7    // Create a new incognito browser context
8    const context = await browser.createIncognitoBrowserContext();
9
10    // Open a new page in the incognito context
11    const page = await context.newPage();
12
13    // Navigate to a webpage
14    await page.goto('https://example.com');
15
16    // Perform actions within the incognito context
17    console.log('Title:', await page.title());
18
19    // Close the incognito context
20    await context.close();
21
22    // Close the browser
23    await browser.close();
24})();

Handling Popups and Alerts

Handling Dialogs

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch({ headless: false });
6    const page = await browser.newPage();
7
8    // Listen for dialog events
9    page.on('dialog', async dialog => {
10        console.log('Dialog message:', dialog.message());
11        
12        // Handling different types of dialogs
13        if (dialog.type() === 'alert') {
14            console.log('Handling alert dialog');
15            await dialog.dismiss(); // Dismiss the alert dialog
16        } else if (dialog.type() === 'confirm') {
17            console.log('Handling confirm dialog');
18            await dialog.accept(); // Accept the confirm dialog
19        } else if (dialog.type() === 'prompt') {
20            console.log('Handling prompt dialog');
21            await dialog.accept('Some input'); // Accept the prompt dialog with input
22        }
23    });
24
25    // Navigate to a page that triggers dialogs
26    await page.goto('https://example.com'); // Replace with your URL
27
28    // Example actions that trigger dialogs
29    await page.evaluate(() => {
30        // Trigger an alert dialog
31        alert('This is an alert');
32        
33        // Trigger a confirm dialog
34        confirm('This is a confirm dialog');
35        
36        // Trigger a prompt dialog
37        prompt('This is a prompt dialog', 'Default text');
38    });
39
40    // Wait to see dialog handling
41    await page.waitForTimeout(5000); // Adjust timeout as needed
42
43    // Close the browser
44    await browser.close();
45})();

Dismissing Alerts

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch({ headless: false });
6    const page = await browser.newPage();
7
8    // Listen for dialog events
9    page.on('dialog', async dialog => {
10        console.log('Dialog type:', dialog.type()); // Log the dialog type
11        console.log('Dialog message:', dialog.message()); // Log the dialog message
12        
13        // Check if the dialog is an alert
14        if (dialog.type() === 'alert') {
15            console.log('Dismissing alert dialog');
16            await dialog.dismiss(); // Dismiss the alert dialog
17        }
18    });
19
20    // Navigate to a page that triggers an alert dialog
21    await page.goto('https://example.com'); // Replace with your URL
22
23    // Example action that triggers an alert dialog
24    await page.evaluate(() => {
25        alert('This is an alert dialog');
26    });
27
28    // Wait to ensure the dialog is handled
29    await page.waitForTimeout(5000); // Adjust timeout as needed
30
31    // Close the browser
32    await browser.close();
33})();

Dealing with Confirmations

1const puppeteer = require('puppeteer');
2
3(async () => {
4    // Launch a new browser instance
5    const browser = await puppeteer.launch({ headless: false });
6    const page = await browser.newPage();
7
8    // Listen for dialog events
9    page.on('dialog', async dialog => {
10        console.log('Dialog type:', dialog.type()); // Log the dialog type
11        console.log('Dialog message:', dialog.message()); // Log the dialog message
12        
13        // Check if the dialog is a confirmation
14        if (dialog.type() === 'confirm') {
15            console.log('Handling confirm dialog');
16            await dialog.accept(); // Accept the confirm dialog
17            // Alternatively, you can use dialog.dismiss() to cancel the confirmation
18            // await dialog.dismiss(); // Uncomment this line to cancel the confirm dialog
19        }
20    });
21
22    // Navigate to a page that triggers a confirmation dialog
23    await page.goto('https://example.com'); // Replace with your URL
24
25    // Example action that triggers a confirmation dialog
26    await page.evaluate(() => {
27        confirm('This is a confirm dialog');
28    });
29
30    // Wait to ensure the dialog is handled
31    await page.waitForTimeout(5000); // Adjust timeout as needed
32
33    // Close the browser
34    await browser.close();
35})();

Automated Testing with Puppeteer

Unit Testing with Jest

Install dependencies.
npm install puppeteer jest --save-dev
Write Tests.
1const puppeteer = require('puppeteer');
2
3describe('Puppeteer Unit Tests', () => {
4    let browser;
5    let page;
6
7    beforeAll(async () => {
8        browser = await puppeteer.launch({ headless: true });
9        page = await browser.newPage();
10    });
11
12    afterAll(async () => {
13        await browser.close();
14    });
15
16    test('should display the correct title', async () => {
17        await page.goto('https://example.com'); // Replace with your target URL
18        const title = await page.title();
19        expect(title).toBe('Example Domain'); // Replace with expected title
20    });
21
22    test('should have a specific element', async () => {
23        await page.goto('https://example.com'); // Replace with your target URL
24        const element = await page.$('h1'); // Replace with your selector
25        expect(element).toBeTruthy();
26    });
27
28    test('should click a button and navigate', async () => {
29        await page.goto('https://example.com'); // Replace with your target URL
30        await page.click('a'); // Replace with your selector
31        await page.waitForNavigation();
32        const newTitle = await page.title();
33        expect(newTitle).toBe('Expected New Title'); // Replace with expected title
34    });
35});
Run tests.
npx jest

Unit Testing with mocha

Install dependencies.
npm install puppeteer mocha chai --save-dev
Write tests.
1const puppeteer = require('puppeteer');
2const { expect } = require('chai');
3
4describe('Puppeteer Unit Tests', function () {
5    let browser;
6    let page;
7
8    before(async function () {
9        browser = await puppeteer.launch({ headless: true });
10        page = await browser.newPage();
11    });
12
13    after(async function () {
14        await browser.close();
15    });
16
17    it('should display the correct title', async function () {
18        await page.goto('https://example.com'); // Replace with your target URL
19        const title = await page.title();
20        expect(title).to.equal('Example Domain'); // Replace with expected title
21    });
22
23    it('should have a specific element', async function () {
24        await page.goto('https://example.com'); // Replace with your target URL
25        const element = await page.$('h1'); // Replace with your selector
26        expect(element).to.not.be.null;
27    });
28
29    it('should click a button and navigate', async function () {
30        await page.goto('https://example.com'); // Replace with your target URL
31        await page.click('a'); // Replace with your selector
32        await page.waitForNavigation();
33        const newTitle = await page.title();
34        expect(newTitle).to.equal('Expected New Title'); // Replace with expected title
35    });
36});
Run tests.
npx mocha

Continuous Integration

Integrating Puppeteer with Continuous Integration (CI) systems allows you to automate end-to-end tests as part of your build process, ensuring that your web application remains functional with each change. Here's how you can set up Puppeteer with popular CI tools like GitHub Actions, GitLab CI, and Jenkins.

CI using Github actions
1name: CI
2
3on:
4  push:
5    branches:
6      - main
7  pull_request:
8    branches:
9      - main
10
11jobs:
12  build:
13    runs-on: ubuntu-latest
14
15    steps:
16      - name: Checkout code
17        uses: actions/checkout@v3
18
19      - name: Set up Node.js
20        uses: actions/setup-node@v3
21        with:
22          node-version: '14' # Replace with your Node.js version
23
24      - name: Install dependencies
25        run: npm install
26
27      - name: Run tests
28        run: npm test # Adjust if you're using a different command to run tests
29
30      - name: Generate test coverage report
31        run: npm run coverage # Adjust if you have a coverage script
CI using gitlab CI
1stages:
2  - test
3
4test:
5  image: node:14 # Replace with your Node.js image
6  stage: test
7  script:
8    - npm install
9    - npm test # Adjust if you're using a different command to run tests
10  artifacts:
11    paths:
12      - coverage/ # Adjust if you have coverage reports or other artifacts
CI using Jenkins.
1pipeline {
2    agent any
3
4    stages {
5        stage('Install Dependencies') {
6            steps {
7                script {
8                    sh 'npm install'
9                }
10            }
11        }
12
13        stage('Run Tests') {
14            steps {
15                script {
16                    sh 'npm test' // Adjust if you're using a different command to run tests
17                }
18            }
19        }
20
21        stage('Archive Test Results') {
22            steps {
23                archiveArtifacts artifacts: 'coverage/**' // Adjust if you have coverage reports or other artifacts
24            }
25        }
26    }
27}

Advanced Puppeteer Techniques

Performance Analysis

1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch({ headless: true });
5    const page = await browser.newPage();
6
7    // Enable performance metrics
8    await page.goto('https://example.com', { waitUntil: 'networkidle2' });
9
10    // Capture performance metrics
11    const metrics = await page.evaluate(() => {
12        return JSON.stringify(window.performance.timing);
13    });
14
15    console.log('Performance Metrics:', metrics);
16
17    // Optionally, capture more detailed performance data
18    const performanceData = await page.evaluate(() => {
19        return new Promise((resolve) => {
20            const data = {
21                performance: window.performance.getEntriesByType('navigation'),
22                resources: window.performance.getEntriesByType('resource'),
23            };
24            resolve(data);
25        });
26    });
27
28    console.log('Detailed Performance Data:', performanceData);
29
30    await browser.close();
31})();

Performance API

1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch({ headless: true });
5    const page = await browser.newPage();
6
7    // Start performance metrics collection
8    await page.goto('https://example.com', { waitUntil: 'networkidle2' });
9
10    // Capture and log navigation timing
11    const navigationTiming = await page.evaluate(() => {
12        return window.performance.getEntriesByType('navigation')[0];
13    });
14
15    console.log('Navigation Timing:', navigationTiming);
16
17    // Capture and log resource timing
18    const resourceTiming = await page.evaluate(() => {
19        return window.performance.getEntriesByType('resource');
20    });
21
22    console.log('Resource Timing:', resourceTiming);
23
24    await browser.close();
25})();

Measuring Performance After Interaction

1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch({ headless: true });
5    const page = await browser.newPage();
6
7    // Navigate to the page
8    await page.goto('https://example.com', { waitUntil: 'networkidle2' });
9
10    // Measure performance before interaction
11    const beforeInteraction = await page.evaluate(() => {
12        return {
13            navigationTiming: window.performance.getEntriesByType('navigation')[0],
14            resourceTiming: window.performance.getEntriesByType('resource'),
15        };
16    });
17
18    console.log('Performance Before Interaction:', beforeInteraction);
19
20    // Simulate user interaction
21    await page.click('button'); // Replace with your interaction
22
23    // Measure performance after interaction
24    await page.waitForTimeout(2000); // Wait for any potential delays or updates
25
26    const afterInteraction = await page.evaluate(() => {
27        return {
28            navigationTiming: window.performance.getEntriesByType('navigation')[0],
29            resourceTiming: window.performance.getEntriesByType('resource'),
30        };
31    });
32
33    console.log('Performance After Interaction:', afterInteraction);
34
35    await browser.close();
36})();

Lighthouse

1const puppeteer = require('puppeteer');
2const { exec } = require('child_process');
3
4(async () => {
5    const browser = await puppeteer.launch({ headless: true });
6    const page = await browser.newPage();
7
8    await page.goto('https://example.com');
9
10    // Save page URL to a file
11    const url = 'https://example.com';
12    require('fs').writeFileSync('url.txt', url);
13
14    await browser.close();
15
16    // Run Lighthouse CLI
17    exec('lighthouse url.txt --output html --output-path ./report.html', (error, stdout, stderr) => {
18        if (error) {
19            console.error(`Error: ${error.message}`);
20            return;
21        }
22        if (stderr) {
23            console.error(`Stderr: ${stderr}`);
24            return;
25        }
26        console.log(`Stdout: ${stdout}`);
27    });
28})();

Device Emulation

Emulating an iPhone
1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch({ headless: false });
5    const page = await browser.newPage();
6
7    // Emulate iPhone 6
8    const iPhone = puppeteer.devices['iPhone 6'];
9    await page.emulate(iPhone);
10
11    await page.goto('https://example.com'); // Replace with your target URL
12    await page.screenshot({ path: 'screenshot.png' }); // Optional: take a screenshot
13
14    await browser.close();
15})();
Custom device emulation
1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch({ headless: false });
5    const page = await browser.newPage();
6
7    // Define custom device settings
8    const customDevice = {
9        name: 'Custom Device',
10        userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3',
11        viewport: {
12            width: 768,
13            height: 1024,
14            deviceScaleFactor: 1,
15            isMobile: true,
16            hasTouch: true,
17            isLandscape: false
18        }
19    };
20    await page.emulate(customDevice);
21
22    await page.goto('https://example.com'); // Replace with your target URL
23    await page.screenshot({ path: 'custom-device-screenshot.png' }); // Optional: take a screenshot
24
25    await browser.close();
26})();

Emulation with mobile features

1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch({ headless: false });
5    const page = await browser.newPage();
6
7    // Emulate iPhone X
8    const iPhoneX = puppeteer.devices['iPhone X'];
9    await page.emulate(iPhoneX);
10
11    await page.goto('https://example.com'); // Replace with your target URL
12    await page.screenshot({ path: 'iphone-x-screenshot.png' }); // Optional: take a screenshot
13
14    await browser.close();
15})();

Network emulation

1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch({ headless: false });
5    const page = await browser.newPage();
6
7    // Emulate iPhone 6
8    const iPhone = puppeteer.devices['iPhone 6'];
9    await page.emulate(iPhone);
10
11    // Set network conditions to 3G
12    await page.emulateNetworkConditions({
13        offline: false,
14        downloadThroughput: (1024 * 1024) / 10, // 1 Mbps
15        uploadThroughput: (1024 * 1024) / 10, // 1 Mbps
16        latency: 300 // 300 ms
17    });
18
19    await page.goto('https://example.com'); // Replace with your target URL
20    await page.screenshot({ path: '3g-network-screenshot.png' }); // Optional: take a screenshot
21
22    await browser.close();
23})();

Testing responsiveness of websites

1const puppeteer = require('puppeteer');
2
3(async () => {
4    const browser = await puppeteer.launch({ headless: false });
5    const page = await browser.newPage();
6
7    // Emulate different viewport sizes
8    const viewports = [
9        { width: 375, height: 667 }, // iPhone 6/7/8
10        { width: 414, height: 736 }, // iPhone 6/7/8 Plus
11        { width: 768, height: 1024 }, // iPad
12        { width: 1280, height: 800 } // Desktop
13    ];
14
15    for (const viewport of viewports) {
16        await page.setViewport(viewport);
17        await page.goto('https://example.com'); // Replace with your target URL
18        await page.screenshot({ path: `viewport-${viewport.width}x${viewport.height}.png` }); // Optional: take a screenshot
19    }
20
21    await browser.close();
22})();