Introduction to Puppeteer
Overview of Puppeteer
Puppeteer is a Node.js library that provides a high-level API to control headless Chrome or Chromium browsers programmatically. It's maintained by the Google Chrome team and is widely used for web scraping, automated testing, and other browser automation tasks.
Key Features- Headless Browser Automation: Puppeteer runs in headless mode by default, meaning it doesn't have a graphical user interface (GUI), making it faster and more efficient for automation tasks. It can also be run in "headful" mode to visually observe what the browser is doing.
- Web Scraping: Puppeteer is excellent for web scraping. It allows you to navigate pages, interact with elements, and extract data from websites. Its ability to execute JavaScript on pages ensures that you can interact with dynamic content that might be generated by JavaScript frameworks.
- Automated Testing: Puppeteer is widely used in testing frameworks to automate browser testing. It can simulate user interactions like clicks, typing, and form submissions, making it a powerful tool for end-to-end testing.
- PDF and Screenshot Generation: Puppeteer can take screenshots of web pages or generate PDFs. This is particularly useful for creating visual reports or capturing the state of a webpage at a specific time.
- Advanced Interactions: Puppeteer can handle complex interactions like filling out forms, handling file uploads, waiting for specific elements to appear, and even intercepting network requests.
- Cross-Browser Testing: While Puppeteer is primarily for Chrome/Chromium, it also has the capability to work with Firefox via the Puppeteer-Firefox project, making it a versatile tool for cross-browser testing.
- Built-in Support for Chrome DevTools: Puppeteer provides a programmatic way to use the same debugging features that developers use in Chrome DevTools, such as inspecting elements, monitoring network activity, and more.
- Scripted Navigation: Puppeteer allows you to automate navigation across multiple pages, making it easy to create scripts that interact with multi-page workflows.
- End-to-End Testing: Automate the testing of user interactions and flows on websites, ensuring that your web application behaves as expected.
- Web Scraping: Extract data from websites, even those with dynamic content generated by JavaScript.
- Performance Monitoring: Measure page load times, render times, and other performance metrics by automating the collection of these data points.
- Automated Form Submission: Automate the filling and submission of web forms, useful in scenarios like testing, bot creation, and data entry.
- SEO and Accessibility Audits: Analyze pages for SEO and accessibility compliance by leveraging Puppeteer's ability to interact with and inspect the DOM.
Installation and Setup
Installation and Setup of Puppeteer in VS CodeHere's a step-by-step guide to installing and setting up Puppeteer in Visual Studio Code (VS Code):
1. Install Node.jsBefore you can use Puppeteer, you need to have Node.js installed on your system. Puppeteer is a Node.js library, so it requires Node.js to run.
- Download and install Node.js from the official website.
- After installation, you can verify the installation by running the following commands in your terminal:
1node -v
2npm -v
These commands should return the installed versions of Node.js and npm (Node Package Manager).
2. Set Up a New Project- Open Visual Studio Code.
- Create a new directory for your project, and open it in VS Code.
- Open the terminal in VS Code (
Ctrl + \`
) and initialize a new Node.js project by running:
npm init -y
This command creates a package.json
file with default settings.
Now that your Node.js project is set up, you can install Puppeteer.
- Run the following command in the terminal to install Puppeteer:
npm install puppeteer
This command will download and install Puppeteer, including a compatible version of Chromium.
4. Create a Script to Use Puppeteer- In your project directory, create a new JavaScript file, e.g.,
index.js
. - Open the
index.js
file in VS Code and add the following code as a basic Puppeteer script:
1const puppeteer = require('puppeteer');
2
3(async () => {
4// Launch a new browser instance
5const browser = await puppeteer.launch();
6// Open a new page
7const page = await browser.newPage();
8// Navigate to a website
9await page.goto('https://example.com');
10// Take a screenshot
11await page.screenshot({ path: 'example.png' });
12// Close the browser
13await browser.close();
14})();
This script does the following:
- Launches a new headless Chrome/Chromium browser.
- Opens a new page.
- navigates to
https://example.com
. - Takes a screenshot of the page and saves it as
example.png
. - Closes the browser.
- Save the
index.js
file. - In the terminal, run the script by executing:
node index.js
After the script runs, you should see a new file called example.png
in your project directory. This file is a screenshot of the webpage.
Now that Puppeteer is set up, you can start exploring its features, such as:
- Automating user interactions: Clicks, typing, form submissions, etc.
- Web scraping: Extracting data from web pages.
- Generating PDFs: Creating PDFs from web pages.
- Testing: Writing end-to-end tests for your web applications.
You can refer to the official Puppeteer documentation for more advanced usage and examples.
Page Interactions
Page Navigation
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch();
6 const page = await browser.newPage();
7
8 // Navigate to the first URL
9 await page.goto('https://example.com');
10
11 // Perform an action on the first page, such as clicking a link
12 await page.click('a'); // This example assumes there is at least one link on the page
13
14 // Wait for navigation to complete
15 await page.waitForNavigation();
16
17 // Print the current URL
18 console.log('Current URL:', page.url());
19
20 // Close the browser
21 await browser.close();
22})();
Clicking and Typing
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch();
6 const page = await browser.newPage();
7
8 // Navigate to a webpage with form elements
9 await page.goto('https://example.com'); // Replace with the URL of a page with input fields
10
11 // Click on a button or link
12 await page.click('#button-id'); // Replace with the selector of the element you want to click
13
14 // Type text into an input field
15 await page.type('#input-id', 'Sample text'); // Replace with the selector of the input field and the text you want to type
16
17 // Submit a form (if applicable)
18 await page.click('#submit-button-id'); // Replace with the selector of the form submit button
19
20 // Wait for navigation or for an element to appear
21 await page.waitForNavigation(); // Use this if clicking or typing leads to navigation
22 // Or use this if you are waiting for a specific element
23 // await page.waitForSelector('#some-element-id');
24
25 // Optionally, take a screenshot to verify results
26 await page.screenshot({ path: 'result.png' });
27
28 // Close the browser
29 await browser.close();
30})();
Form Handling
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch();
6 const page = await browser.newPage();
7
8 // Navigate to the page with the form
9 await page.goto('https://example.com/form'); // Replace with the URL of the page with your form
10
11 // Fill out the form
12 await page.type('#name', 'John Doe'); // Replace #name with the selector for the name input field
13 await page.type('#email', '[email protected]'); // Replace #email with the selector for the email input field
14 await page.type('#message', 'Hello, this is a test message.'); // Replace #message with the selector for the message textarea
15
16 // Submit the form
17 await page.click('#submit'); // Replace #submit with the selector for the submit button
18
19 // Optionally, wait for navigation or confirmation
20 await page.waitForNavigation(); // Use if form submission causes navigation
21 // Or wait for an element indicating form submission success
22 // await page.waitForSelector('#success-message'); // Replace with a selector for success message
23
24 // Capture a screenshot of the result page
25 await page.screenshot({ path: 'form-submission-result.png' });
26
27 // Close the browser
28 await browser.close();
29})();
Network Interception
Intercepting Requests
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch();
6 const page = await browser.newPage();
7
8 // Intercept network requests
9 await page.setRequestInterception(true);
10
11 page.on('request', request => {
12 // Log intercepted requests
13 console.log('Intercepted request:', request.url());
14
15 // Modify request or response here
16 // For example, abort a request
17 // if (request.url().includes('example.com/api')) {
18 // request.abort();
19 // } else {
20 // request.continue();
21 // }
22
23 // To modify response, use request.respond()
24 if (request.url().includes('example.com/api')) {
25 request.respond({
26 status: 200,
27 contentType: 'application/json',
28 body: JSON.stringify({ message: 'This is a mocked response' })
29 });
30 } else {
31 request.continue(); // Continue with the original request if not intercepted
32 }
33 });
34
35 // Navigate to the page
36 await page.goto('https://example.com'); // Replace with your target URL
37
38 // Optionally, wait for specific elements or interactions
39 await page.waitForSelector('h1'); // Replace with a selector relevant to your page
40
41 // Take a screenshot of the page
42 await page.screenshot({ path: 'intercepted-request.png' });
43
44 // Close the browser
45 await browser.close();
46})();
Modifying Requests and Responses
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch();
6 const page = await browser.newPage();
7
8 // Enable request interception
9 await page.setRequestInterception(true);
10
11 // Intercept and modify requests and responses
12 page.on('request', request => {
13 // Log the URL of the intercepted request
14 console.log('Intercepted request:', request.url());
15
16 // Example: Modify a request URL
17 if (request.url().includes('example.com/api')) {
18 // Optionally, modify request headers
19 request.continue({
20 headers: {
21 ...request.headers(),
22 'X-Modified-Header': 'ModifiedValue'
23 }
24 });
25 } else {
26 request.continue(); // Continue with the original request
27 }
28 });
29
30 page.on('response', async response => {
31 // Log the URL of the intercepted response
32 console.log('Intercepted response:', response.url());
33
34 // Example: Modify a response body
35 if (response.url().includes('example.com/api')) {
36 const responseBody = await response.text();
37
38 // Modify the response body
39 const modifiedBody = responseBody.replace('original value', 'modified value');
40
41 // Respond with the modified body
42 await response.respond({
43 status: response.status(),
44 contentType: response.headers()['content-type'],
45 body: modifiedBody
46 });
47 }
48 });
49
50 // Navigate to the page
51 await page.goto('https://example.com'); // Replace with your target URL
52
53 // Optionally, wait for specific elements or interactions
54 await page.waitForSelector('h1'); // Replace with a selector relevant to your page
55
56 // Take a screenshot of the page
57 await page.screenshot({ path: 'modified-request-response.png' });
58
59 // Close the browser
60 await browser.close();
61})();
Spoofing User Agents
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch();
6 const page = await browser.newPage();
7
8 // Set a custom user agent
9 const customUserAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36';
10 await page.setUserAgent(customUserAgent);
11
12 // Optionally, set other browser properties like viewport or language
13 await page.setViewport({ width: 1280, height: 800 });
14 await page.setExtraHTTPHeaders({
15 'Accept-Language': 'en-US,en;q=0.9'
16 });
17
18 // Navigate to a webpage
19 await page.goto('https://www.whatismybrowser.com/'); // Replace with your target URL
20
21 // Optionally, wait for specific elements or interactions
22 await page.waitForSelector('h1'); // Replace with a selector relevant to your page
23
24 // Take a screenshot to verify the user agent change
25 await page.screenshot({ path: 'user-agent-spoofed.png' });
26
27 // Close the browser
28 await browser.close();
29})();
File Handling
Uploading Files
1const puppeteer = require('puppeteer');
2const path = require('path');
3
4(async () => {
5 // Launch a new browser instance
6 const browser = await puppeteer.launch();
7 const page = await browser.newPage();
8
9 // Navigate to the page with the file upload form
10 await page.goto('https://example.com/upload'); // Replace with the URL of the file upload form
11
12 // Define the path to the file you want to upload
13 const filePath = path.resolve(__dirname, 'file-to-upload.txt'); // Replace with the path to your file
14
15 // Select the file input element and upload the file
16 const inputUploadHandle = await page.$('input[type=file]'); // Replace with the selector for your file input
17 await inputUploadHandle.uploadFile(filePath);
18
19 // Optionally, submit the form or click a button to complete the upload
20 await page.click('#submit-button'); // Replace with the selector for the submit button
21
22 // Wait for navigation or confirmation
23 await page.waitForNavigation(); // Use if the upload causes a page navigation
24 // Or wait for an element indicating upload success
25 // await page.waitForSelector('#success-message'); // Replace with the selector for a success message
26
27 // Optionally, take a screenshot to verify the result
28 await page.screenshot({ path: 'upload-result.png' });
29
30 // Close the browser
31 await browser.close();
32})();
Downloading Files
1const puppeteer = require('puppeteer');
2const fs = require('fs');
3const path = require('path');
4
5(async () => {
6 // Launch a new browser instance
7 const browser = await puppeteer.launch();
8 const page = await browser.newPage();
9
10 // Set up a download behavior
11 const downloadPath = path.resolve(__dirname, 'downloads');
12 if (!fs.existsSync(downloadPath)) {
13 fs.mkdirSync(downloadPath);
14 }
15
16 // Use a custom download folder
17 await page._client.send('Page.setDownloadBehavior', {
18 behavior: 'allow',
19 downloadPath: downloadPath
20 });
21
22 // Navigate to the page with the download link
23 await page.goto('https://example.com/download'); // Replace with the URL of the download page
24
25 // Click the download link or button
26 await page.click('#download-button'); // Replace with the selector for your download link/button
27
28 // Wait for the download to complete
29 // You might need to adjust the wait time based on the expected download duration
30 await page.waitForTimeout(5000); // Adjust timeout as necessary
31
32 // Check if the file has been downloaded
33 const files = fs.readdirSync(downloadPath);
34 if (files.length > 0) {
35 console.log('Downloaded files:', files);
36
37 // Optionally, read and process the downloaded file
38 const downloadedFile = path.join(downloadPath, files[0]);
39 console.log('Downloaded file path:', downloadedFile);
40
41 // Example: Read file contents (if it's a text file)
42 const fileContents = fs.readFileSync(downloadedFile, 'utf8');
43 console.log('File contents:', fileContents);
44 } else {
45 console.log('No files downloaded.');
46 }
47
48 // Close the browser
49 await browser.close();
50})();
Screenshots and PDFs
Taking Screenshots
1const puppeteer = require('puppeteer');
2
3(async () => {
4// Launch a new browser instance
5const browser = await puppeteer.launch();
6const page = await browser.newPage();
7
8// Navigate to a URL
9await page.goto('https://example.com'); // Replace with the URL of the page you want to capture
10
11// Optionally, wait for a specific element to be loaded
12// await page.waitForSelector('#element-id'); // Replace with a selector if needed
13
14// Take a screenshot of the entire page
15await page.screenshot({ path: 'fullpage-screenshot.png', fullPage: true });
16
17// Take a screenshot of a specific element
18const element = await page.$('#element-id'); // Replace with the selector of the element you want to capture
19if (element) {
20 await element.screenshot({ path: 'element-screenshot.png' });
21}
22
23// Close the browser
24await browser.close();
25})();
Capturing Specific Elements
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch();
6 const page = await browser.newPage();
7
8 // Navigate to the webpage
9 await page.goto('https://example.com'); // Replace with your target URL
10
11 // Wait for the specific element to be present
12 await page.waitForSelector('#specific-element'); // Replace with the selector for your target element
13
14 // Select the element
15 const element = await page.$('#specific-element'); // Replace with the selector for your target element
16
17 if (element) {
18 // Capture a screenshot of the specific element
19 await element.screenshot({ path: 'element-screenshot.png' });
20 console.log('Screenshot of the specific element has been saved as element-screenshot.png');
21 } else {
22 console.log('Element not found.');
23 }
24
25 // Close the browser
26 await browser.close();
27})();
Saving Pages as PDF
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch();
6 const page = await browser.newPage();
7
8 // Navigate to the webpage
9 await page.goto('https://example.com', { waitUntil: 'networkidle2' }); // Replace with your target URL
10
11 // Define the path where the PDF will be saved
12 const pdfPath = 'page-snapshot.pdf';
13
14 // Save the page as a PDF
15 await page.pdf({
16 path: pdfPath,
17 format: 'A4', // Paper format (A4, Letter, etc.)
18 printBackground: true, // Include background graphics
19 margin: { // Page margins
20 top: '20mm',
21 right: '20mm',
22 bottom: '20mm',
23 left: '20mm'
24 }
25 });
26
27 console.log(`PDF saved as ${pdfPath}`);
28
29 // Close the browser
30 await browser.close();
31})();
Browser Options and Settings
Headless vs Headful Modes
Headless Mode runs the browser without a graphical user interface (GUI). This mode is typically used for automation tasks, such as web scraping, testing, and generating PDFs. Headful Mode runs the browser with a graphical user interface. This mode is useful for visual debugging and scenarios where you need to interact with the browser manually.1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch(); // Headless mode by default
5 const page = await browser.newPage();
6 await page.goto('https://example.com');
7 await browser.close();
8})();
Opening DevTools
Using DevTools with Puppeteer can be extremely helpful for debugging and inspecting web pages. Puppeteer provides a way to interact with the Chrome DevTools Protocol directly, allowing you to use DevTools features programmatically.1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch the browser in headful mode and open DevTools
5 const browser = await puppeteer.launch({ headless: false, devtools: true });
6 const page = await browser.newPage();
7 await page.goto('https://example.com');
8
9 // Optional: Wait for some time to manually inspect
10 await page.waitForTimeout(30000); // Wait for 30 seconds
11
12 await browser.close();
13})();
Using devtools protocol directly
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch({ headless: false });
6 const page = await browser.newPage();
7
8 // Access the DevTools Protocol
9 const client = await page.target().createCDPSession();
10
11 // Enable the network domain
12 await client.send('Network.enable');
13
14 // Listen for network requests
15 client.on('Network.requestWillBeSent', (event) => {
16 console.log('Request:', event.request.url);
17 });
18
19 // Navigate to a webpage
20 await page.goto('https://example.com');
21
22 // Perform actions and use DevTools features
23 await page.evaluate(() => {
24 // Example: Log the document title
25 console.log(document.title);
26 });
27
28 // Close the browser
29 await browser.close();
30})();
Capturing network requests
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch({ headless: false });
6 const page = await browser.newPage();
7
8 // Access the DevTools Protocol
9 const client = await page.target().createCDPSession();
10
11 // Enable the network domain
12 await client.send('Network.enable');
13
14 // Listen for network requests
15 client.on('Network.requestWillBeSent', (event) => {
16 console.log('Request URL:', event.request.url);
17 });
18
19 // Navigate to a webpage
20 await page.goto('https://example.com');
21
22 // Close the browser
23 await browser.close();
24})();
Emulate Devices
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch({ headless: false });
6 const page = await browser.newPage();
7
8 // Access the DevTools Protocol
9 const client = await page.target().createCDPSession();
10
11 // Emulate a device
12 await client.send('Emulation.setDeviceMetricsOverride', {
13 width: 360,
14 height: 640,
15 deviceScaleFactor: 2,
16 mobile: true
17 });
18
19 // Navigate to a webpage
20 await page.goto('https://example.com');
21
22 // Close the browser
23 await browser.close();
24})();
Incognito Mode
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch({ headless: false });
6
7 // Create a new incognito browser context
8 const context = await browser.createIncognitoBrowserContext();
9
10 // Open a new page in the incognito context
11 const page = await context.newPage();
12
13 // Navigate to a webpage
14 await page.goto('https://example.com');
15
16 // Perform actions within the incognito context
17 console.log('Title:', await page.title());
18
19 // Close the incognito context
20 await context.close();
21
22 // Close the browser
23 await browser.close();
24})();
Handling Popups and Alerts
Handling Dialogs
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch({ headless: false });
6 const page = await browser.newPage();
7
8 // Listen for dialog events
9 page.on('dialog', async dialog => {
10 console.log('Dialog message:', dialog.message());
11
12 // Handling different types of dialogs
13 if (dialog.type() === 'alert') {
14 console.log('Handling alert dialog');
15 await dialog.dismiss(); // Dismiss the alert dialog
16 } else if (dialog.type() === 'confirm') {
17 console.log('Handling confirm dialog');
18 await dialog.accept(); // Accept the confirm dialog
19 } else if (dialog.type() === 'prompt') {
20 console.log('Handling prompt dialog');
21 await dialog.accept('Some input'); // Accept the prompt dialog with input
22 }
23 });
24
25 // Navigate to a page that triggers dialogs
26 await page.goto('https://example.com'); // Replace with your URL
27
28 // Example actions that trigger dialogs
29 await page.evaluate(() => {
30 // Trigger an alert dialog
31 alert('This is an alert');
32
33 // Trigger a confirm dialog
34 confirm('This is a confirm dialog');
35
36 // Trigger a prompt dialog
37 prompt('This is a prompt dialog', 'Default text');
38 });
39
40 // Wait to see dialog handling
41 await page.waitForTimeout(5000); // Adjust timeout as needed
42
43 // Close the browser
44 await browser.close();
45})();
Dismissing Alerts
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch({ headless: false });
6 const page = await browser.newPage();
7
8 // Listen for dialog events
9 page.on('dialog', async dialog => {
10 console.log('Dialog type:', dialog.type()); // Log the dialog type
11 console.log('Dialog message:', dialog.message()); // Log the dialog message
12
13 // Check if the dialog is an alert
14 if (dialog.type() === 'alert') {
15 console.log('Dismissing alert dialog');
16 await dialog.dismiss(); // Dismiss the alert dialog
17 }
18 });
19
20 // Navigate to a page that triggers an alert dialog
21 await page.goto('https://example.com'); // Replace with your URL
22
23 // Example action that triggers an alert dialog
24 await page.evaluate(() => {
25 alert('This is an alert dialog');
26 });
27
28 // Wait to ensure the dialog is handled
29 await page.waitForTimeout(5000); // Adjust timeout as needed
30
31 // Close the browser
32 await browser.close();
33})();
Dealing with Confirmations
1const puppeteer = require('puppeteer');
2
3(async () => {
4 // Launch a new browser instance
5 const browser = await puppeteer.launch({ headless: false });
6 const page = await browser.newPage();
7
8 // Listen for dialog events
9 page.on('dialog', async dialog => {
10 console.log('Dialog type:', dialog.type()); // Log the dialog type
11 console.log('Dialog message:', dialog.message()); // Log the dialog message
12
13 // Check if the dialog is a confirmation
14 if (dialog.type() === 'confirm') {
15 console.log('Handling confirm dialog');
16 await dialog.accept(); // Accept the confirm dialog
17 // Alternatively, you can use dialog.dismiss() to cancel the confirmation
18 // await dialog.dismiss(); // Uncomment this line to cancel the confirm dialog
19 }
20 });
21
22 // Navigate to a page that triggers a confirmation dialog
23 await page.goto('https://example.com'); // Replace with your URL
24
25 // Example action that triggers a confirmation dialog
26 await page.evaluate(() => {
27 confirm('This is a confirm dialog');
28 });
29
30 // Wait to ensure the dialog is handled
31 await page.waitForTimeout(5000); // Adjust timeout as needed
32
33 // Close the browser
34 await browser.close();
35})();
Automated Testing with Puppeteer
Unit Testing with Jest
Install dependencies.npm install puppeteer jest --save-dev
1const puppeteer = require('puppeteer');
2
3describe('Puppeteer Unit Tests', () => {
4 let browser;
5 let page;
6
7 beforeAll(async () => {
8 browser = await puppeteer.launch({ headless: true });
9 page = await browser.newPage();
10 });
11
12 afterAll(async () => {
13 await browser.close();
14 });
15
16 test('should display the correct title', async () => {
17 await page.goto('https://example.com'); // Replace with your target URL
18 const title = await page.title();
19 expect(title).toBe('Example Domain'); // Replace with expected title
20 });
21
22 test('should have a specific element', async () => {
23 await page.goto('https://example.com'); // Replace with your target URL
24 const element = await page.$('h1'); // Replace with your selector
25 expect(element).toBeTruthy();
26 });
27
28 test('should click a button and navigate', async () => {
29 await page.goto('https://example.com'); // Replace with your target URL
30 await page.click('a'); // Replace with your selector
31 await page.waitForNavigation();
32 const newTitle = await page.title();
33 expect(newTitle).toBe('Expected New Title'); // Replace with expected title
34 });
35});
npx jest
Unit Testing with mocha
Install dependencies.npm install puppeteer mocha chai --save-dev
1const puppeteer = require('puppeteer');
2const { expect } = require('chai');
3
4describe('Puppeteer Unit Tests', function () {
5 let browser;
6 let page;
7
8 before(async function () {
9 browser = await puppeteer.launch({ headless: true });
10 page = await browser.newPage();
11 });
12
13 after(async function () {
14 await browser.close();
15 });
16
17 it('should display the correct title', async function () {
18 await page.goto('https://example.com'); // Replace with your target URL
19 const title = await page.title();
20 expect(title).to.equal('Example Domain'); // Replace with expected title
21 });
22
23 it('should have a specific element', async function () {
24 await page.goto('https://example.com'); // Replace with your target URL
25 const element = await page.$('h1'); // Replace with your selector
26 expect(element).to.not.be.null;
27 });
28
29 it('should click a button and navigate', async function () {
30 await page.goto('https://example.com'); // Replace with your target URL
31 await page.click('a'); // Replace with your selector
32 await page.waitForNavigation();
33 const newTitle = await page.title();
34 expect(newTitle).to.equal('Expected New Title'); // Replace with expected title
35 });
36});
npx mocha
Continuous Integration
Integrating Puppeteer with Continuous Integration (CI) systems allows you to automate end-to-end tests as part of your build process, ensuring that your web application remains functional with each change. Here's how you can set up Puppeteer with popular CI tools like GitHub Actions, GitLab CI, and Jenkins.
CI using Github actions1name: CI
2
3on:
4 push:
5 branches:
6 - main
7 pull_request:
8 branches:
9 - main
10
11jobs:
12 build:
13 runs-on: ubuntu-latest
14
15 steps:
16 - name: Checkout code
17 uses: actions/checkout@v3
18
19 - name: Set up Node.js
20 uses: actions/setup-node@v3
21 with:
22 node-version: '14' # Replace with your Node.js version
23
24 - name: Install dependencies
25 run: npm install
26
27 - name: Run tests
28 run: npm test # Adjust if you're using a different command to run tests
29
30 - name: Generate test coverage report
31 run: npm run coverage # Adjust if you have a coverage script
1stages:
2 - test
3
4test:
5 image: node:14 # Replace with your Node.js image
6 stage: test
7 script:
8 - npm install
9 - npm test # Adjust if you're using a different command to run tests
10 artifacts:
11 paths:
12 - coverage/ # Adjust if you have coverage reports or other artifacts
1pipeline {
2 agent any
3
4 stages {
5 stage('Install Dependencies') {
6 steps {
7 script {
8 sh 'npm install'
9 }
10 }
11 }
12
13 stage('Run Tests') {
14 steps {
15 script {
16 sh 'npm test' // Adjust if you're using a different command to run tests
17 }
18 }
19 }
20
21 stage('Archive Test Results') {
22 steps {
23 archiveArtifacts artifacts: 'coverage/**' // Adjust if you have coverage reports or other artifacts
24 }
25 }
26 }
27}
Advanced Puppeteer Techniques
Performance Analysis
1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch({ headless: true });
5 const page = await browser.newPage();
6
7 // Enable performance metrics
8 await page.goto('https://example.com', { waitUntil: 'networkidle2' });
9
10 // Capture performance metrics
11 const metrics = await page.evaluate(() => {
12 return JSON.stringify(window.performance.timing);
13 });
14
15 console.log('Performance Metrics:', metrics);
16
17 // Optionally, capture more detailed performance data
18 const performanceData = await page.evaluate(() => {
19 return new Promise((resolve) => {
20 const data = {
21 performance: window.performance.getEntriesByType('navigation'),
22 resources: window.performance.getEntriesByType('resource'),
23 };
24 resolve(data);
25 });
26 });
27
28 console.log('Detailed Performance Data:', performanceData);
29
30 await browser.close();
31})();
Performance API
1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch({ headless: true });
5 const page = await browser.newPage();
6
7 // Start performance metrics collection
8 await page.goto('https://example.com', { waitUntil: 'networkidle2' });
9
10 // Capture and log navigation timing
11 const navigationTiming = await page.evaluate(() => {
12 return window.performance.getEntriesByType('navigation')[0];
13 });
14
15 console.log('Navigation Timing:', navigationTiming);
16
17 // Capture and log resource timing
18 const resourceTiming = await page.evaluate(() => {
19 return window.performance.getEntriesByType('resource');
20 });
21
22 console.log('Resource Timing:', resourceTiming);
23
24 await browser.close();
25})();
Measuring Performance After Interaction
1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch({ headless: true });
5 const page = await browser.newPage();
6
7 // Navigate to the page
8 await page.goto('https://example.com', { waitUntil: 'networkidle2' });
9
10 // Measure performance before interaction
11 const beforeInteraction = await page.evaluate(() => {
12 return {
13 navigationTiming: window.performance.getEntriesByType('navigation')[0],
14 resourceTiming: window.performance.getEntriesByType('resource'),
15 };
16 });
17
18 console.log('Performance Before Interaction:', beforeInteraction);
19
20 // Simulate user interaction
21 await page.click('button'); // Replace with your interaction
22
23 // Measure performance after interaction
24 await page.waitForTimeout(2000); // Wait for any potential delays or updates
25
26 const afterInteraction = await page.evaluate(() => {
27 return {
28 navigationTiming: window.performance.getEntriesByType('navigation')[0],
29 resourceTiming: window.performance.getEntriesByType('resource'),
30 };
31 });
32
33 console.log('Performance After Interaction:', afterInteraction);
34
35 await browser.close();
36})();
Lighthouse
1const puppeteer = require('puppeteer');
2const { exec } = require('child_process');
3
4(async () => {
5 const browser = await puppeteer.launch({ headless: true });
6 const page = await browser.newPage();
7
8 await page.goto('https://example.com');
9
10 // Save page URL to a file
11 const url = 'https://example.com';
12 require('fs').writeFileSync('url.txt', url);
13
14 await browser.close();
15
16 // Run Lighthouse CLI
17 exec('lighthouse url.txt --output html --output-path ./report.html', (error, stdout, stderr) => {
18 if (error) {
19 console.error(`Error: ${error.message}`);
20 return;
21 }
22 if (stderr) {
23 console.error(`Stderr: ${stderr}`);
24 return;
25 }
26 console.log(`Stdout: ${stdout}`);
27 });
28})();
Device Emulation
Emulating an iPhone1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch({ headless: false });
5 const page = await browser.newPage();
6
7 // Emulate iPhone 6
8 const iPhone = puppeteer.devices['iPhone 6'];
9 await page.emulate(iPhone);
10
11 await page.goto('https://example.com'); // Replace with your target URL
12 await page.screenshot({ path: 'screenshot.png' }); // Optional: take a screenshot
13
14 await browser.close();
15})();
1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch({ headless: false });
5 const page = await browser.newPage();
6
7 // Define custom device settings
8 const customDevice = {
9 name: 'Custom Device',
10 userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3',
11 viewport: {
12 width: 768,
13 height: 1024,
14 deviceScaleFactor: 1,
15 isMobile: true,
16 hasTouch: true,
17 isLandscape: false
18 }
19 };
20 await page.emulate(customDevice);
21
22 await page.goto('https://example.com'); // Replace with your target URL
23 await page.screenshot({ path: 'custom-device-screenshot.png' }); // Optional: take a screenshot
24
25 await browser.close();
26})();
Emulation with mobile features
1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch({ headless: false });
5 const page = await browser.newPage();
6
7 // Emulate iPhone X
8 const iPhoneX = puppeteer.devices['iPhone X'];
9 await page.emulate(iPhoneX);
10
11 await page.goto('https://example.com'); // Replace with your target URL
12 await page.screenshot({ path: 'iphone-x-screenshot.png' }); // Optional: take a screenshot
13
14 await browser.close();
15})();
Network emulation
1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch({ headless: false });
5 const page = await browser.newPage();
6
7 // Emulate iPhone 6
8 const iPhone = puppeteer.devices['iPhone 6'];
9 await page.emulate(iPhone);
10
11 // Set network conditions to 3G
12 await page.emulateNetworkConditions({
13 offline: false,
14 downloadThroughput: (1024 * 1024) / 10, // 1 Mbps
15 uploadThroughput: (1024 * 1024) / 10, // 1 Mbps
16 latency: 300 // 300 ms
17 });
18
19 await page.goto('https://example.com'); // Replace with your target URL
20 await page.screenshot({ path: '3g-network-screenshot.png' }); // Optional: take a screenshot
21
22 await browser.close();
23})();
Testing responsiveness of websites
1const puppeteer = require('puppeteer');
2
3(async () => {
4 const browser = await puppeteer.launch({ headless: false });
5 const page = await browser.newPage();
6
7 // Emulate different viewport sizes
8 const viewports = [
9 { width: 375, height: 667 }, // iPhone 6/7/8
10 { width: 414, height: 736 }, // iPhone 6/7/8 Plus
11 { width: 768, height: 1024 }, // iPad
12 { width: 1280, height: 800 } // Desktop
13 ];
14
15 for (const viewport of viewports) {
16 await page.setViewport(viewport);
17 await page.goto('https://example.com'); // Replace with your target URL
18 await page.screenshot({ path: `viewport-${viewport.width}x${viewport.height}.png` }); // Optional: take a screenshot
19 }
20
21 await browser.close();
22})();