Browser Automation Beyond Testing: Playwright, Puppeteer, and Selenium

Browser Automation Beyond Testing: Playwright, Puppeteer, and Selenium

Browser automation is one of those capabilities that starts with "we need E2E tests" and quickly expands into scraping data, generating PDFs, capturing screenshots, automating form submissions, and monitoring web applications. The three dominant tools -- Playwright, Puppeteer, and Selenium -- each occupy a different niche. This guide covers all three with practical examples that go well beyond clicking buttons in a test suite.

The Tools at a Glance

Playwright: The Modern Default

Playwright has become the default choice for new projects. It handles Chromium, Firefox, and WebKit with a single API, auto-waits for elements, and ships with a test runner. But its automation capabilities extend far beyond testing.

Setup

# Node.js
npm init playwright@latest

# Python
pip install playwright
playwright install

E2E Testing

Playwright's test runner handles parallel execution, retries, and reporting out of the box.

// tests/checkout.spec.ts
import { test, expect } from '@playwright/test';

test('complete checkout flow', async ({ page }) => {
  await page.goto('https://shop.example.com');
  await page.click('[data-testid="product-card"]');
  await page.click('button:has-text("Add to Cart")');
  await page.click('[data-testid="cart-icon"]');
  await page.click('button:has-text("Checkout")');

  // Fill shipping form
  await page.fill('#email', '[email protected]');
  await page.fill('#address', '123 Test St');
  await page.fill('#city', 'Portland');
  await page.selectOption('#state', 'OR');
  await page.fill('#zip', '97201');

  await page.click('button:has-text("Place Order")');
  await expect(page.locator('.order-confirmation')).toContainText('Order #');
});

PDF Generation

Rendering HTML to PDF is one of the most common non-testing use cases. Playwright produces high-quality PDFs from any webpage or local HTML.

import { chromium } from 'playwright';

async function generateInvoice(invoiceData: InvoiceData): Promise<Buffer> {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // Load your HTML template
  await page.goto(`file://${__dirname}/templates/invoice.html`);

  // Inject data into the template
  await page.evaluate((data) => {
    document.getElementById('customer-name')!.textContent = data.customerName;
    document.getElementById('invoice-number')!.textContent = data.invoiceNumber;
    const tbody = document.getElementById('line-items')!;
    data.items.forEach(item => {
      const row = document.createElement('tr');
      row.innerHTML = `<td>${item.name}</td><td>${item.qty}</td><td>$${item.price}</td>`;
      tbody.appendChild(row);
    });
  }, invoiceData);

  const pdf = await page.pdf({
    format: 'Letter',
    margin: { top: '1in', bottom: '1in', left: '0.75in', right: '0.75in' },
    printBackground: true,
  });

  await browser.close();
  return pdf;
}

Screenshots and Visual Monitoring

Capture full-page screenshots, element screenshots, or clip specific regions. This is useful for visual regression testing, monitoring dashboards, and generating social media preview images.

import { chromium } from 'playwright';

async function captureScreenshots(url: string) {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // Full page screenshot
  await page.goto(url, { waitUntil: 'networkidle' });
  await page.screenshot({ path: 'full-page.png', fullPage: true });

  // Specific element screenshot
  const hero = page.locator('.hero-section');
  await hero.screenshot({ path: 'hero.png' });

  // Multiple viewport sizes for responsive testing
  const viewports = [
    { width: 1920, height: 1080, name: 'desktop' },
    { width: 768, height: 1024, name: 'tablet' },
    { width: 375, height: 812, name: 'mobile' },
  ];

  for (const vp of viewports) {
    await page.setViewportSize({ width: vp.width, height: vp.height });
    await page.screenshot({ path: `screenshot-${vp.name}.png` });
  }

  await browser.close();
}

Web Scraping

Playwright handles JavaScript-rendered content, login flows, and infinite scroll -- problems that trip up HTTP-based scrapers.

import { chromium } from 'playwright';

async function scrapeJobListings(searchTerm: string) {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.goto('https://jobs.example.com');
  await page.fill('#search-input', searchTerm);
  await page.click('button[type="submit"]');
  await page.waitForSelector('.job-card');

  // Scroll to load all results
  let previousHeight = 0;
  while (true) {
    const currentHeight = await page.evaluate(() => document.body.scrollHeight);
    if (currentHeight === previousHeight) break;
    previousHeight = currentHeight;
    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    await page.waitForTimeout(1000);
  }

  const jobs = await page.$$eval('.job-card', cards =>
    cards.map(card => ({
      title: card.querySelector('.job-title')?.textContent?.trim(),
      company: card.querySelector('.company-name')?.textContent?.trim(),
      location: card.querySelector('.location')?.textContent?.trim(),
      link: card.querySelector('a')?.href,
    }))
  );

  await browser.close();
  return jobs;
}

Puppeteer: Chrome-Native Automation

Puppeteer talks directly to Chrome DevTools Protocol, giving you lower-level control. It is the right choice when you only need Chromium and want the thinnest possible abstraction.

PDF Generation with Network Interception

Puppeteer's network interception is useful for injecting authentication or modifying requests during PDF generation.

import puppeteer from 'puppeteer';

async function generateReportPDF(reportUrl: string, authToken: string) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Intercept requests to add auth headers
  await page.setRequestInterception(true);
  page.on('request', (request) => {
    request.continue({
      headers: {
        ...request.headers(),
        Authorization: `Bearer ${authToken}`,
      },
    });
  });

  await page.goto(reportUrl, { waitUntil: 'networkidle0' });

  // Wait for charts to render
  await page.waitForSelector('.chart-rendered', { timeout: 10000 });

  const pdf = await page.pdf({
    format: 'A4',
    landscape: true,
    margin: { top: '20mm', bottom: '20mm', left: '15mm', right: '15mm' },
  });

  await browser.close();
  return pdf;
}

Performance Tracing

Puppeteer exposes Chrome's tracing API, which is valuable for performance monitoring.

import puppeteer from 'puppeteer';

async function tracePageLoad(url: string) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.tracing.start({ path: 'trace.json', screenshots: true });
  await page.goto(url, { waitUntil: 'networkidle0' });
  await page.tracing.stop();

  // Collect performance metrics
  const metrics = await page.metrics();
  console.log('JS Heap Used:', metrics.JSHeapUsedSize);
  console.log('DOM Nodes:', metrics.Nodes);
  console.log('Layout Count:', metrics.LayoutCount);

  // Collect Web Vitals
  const vitals = await page.evaluate(() => {
    return new Promise((resolve) => {
      new PerformanceObserver((list) => {
        const entries = list.getEntries();
        resolve({
          lcp: entries.find(e => e.entryType === 'largest-contentful-paint')?.startTime,
          fcp: entries.find(e => e.name === 'first-contentful-paint')?.startTime,
        });
      }).observe({ type: 'largest-contentful-paint', buffered: true });
    });
  });

  await browser.close();
  return { metrics, vitals };
}

Selenium: The Legacy Workhorse

Selenium is the oldest browser automation framework and still has the widest browser support. It uses the WebDriver protocol, which is a W3C standard. Choose Selenium when you need to test on real Safari, IE, or legacy browsers, or when your team already has a large Selenium test suite.

Setup with WebDriver Manager

# Python
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

driver.get("https://example.com/login")
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "username")))
driver.find_element(By.ID, "username").send_keys("[email protected]")
driver.find_element(By.ID, "password").send_keys("secure-password")
driver.find_element(By.CSS_SELECTOR, "button[type='submit']").click()

WebDriverWait(driver, 10).until(EC.url_contains("/dashboard"))
print(f"Logged in. Current URL: {driver.current_url}")
driver.quit()

Selenium Grid for Parallel Execution

# docker-compose.yml for Selenium Grid
services:
  selenium-hub:
    image: selenium/hub:4
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

  chrome-node:
    image: selenium/node-chrome:4
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_MAX_SESSIONS=4
    deploy:
      replicas: 3

When to Use Which

Choose Playwright when:

• Starting a new project with no existing automation
• You need cross-browser testing (Chromium, Firefox, WebKit)
• PDF generation and screenshots are part of your workflow
• You want built-in auto-wait and test runner

Choose Puppeteer when:

• You only target Chrome/Chromium
• You need Chrome DevTools Protocol features (tracing, network interception)
• You want minimal abstraction over the browser
• You are building a Chrome extension or working with Chrome-specific APIs

Choose Selenium when:

• You must test on real Safari, IE, or other browsers via WebDriver
• Your team has existing Selenium infrastructure
• You need Selenium Grid for large-scale parallel execution
• Corporate policy or compliance requires WebDriver protocol

Running Browser Automation in CI

All three tools run headless in CI. Here is a GitHub Actions example for Playwright:

# .github/workflows/e2e.yml
name: E2E Tests
on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/

Practical Tips

Reuse browser contexts, not browsers. Launching a browser is expensive (hundreds of milliseconds). Creating a new context within an existing browser is cheap. For scraping or PDF generation, launch one browser and create multiple contexts.

Set reasonable timeouts. The default 30-second timeout is usually too long. If a page is not ready in 10 seconds, something is wrong. Fail fast.

Use stealth mode for scraping. Both Playwright and Puppeteer can be detected by bot-detection scripts. The playwright-extra and puppeteer-extra packages with stealth plugins help avoid detection, but respect robots.txt and rate limits.

Intercept unnecessary resources. When scraping, block images, fonts, and analytics scripts to speed up page loads dramatically:

await page.route('**/*.{png,jpg,jpeg,gif,svg,woff,woff2}', route => route.abort());
await page.route('**/analytics**', route => route.abort());

Browser automation is a Swiss Army knife. Once you have a tool like Playwright in your stack, you will find uses for it far beyond testing -- from generating receipts to monitoring competitor pricing to creating automated visual reports. Start with Playwright unless you have a specific reason not to.