crawl

 listing_res = await crawler.arun(url=listing_url)

this is by default sequentioal

change to concurrent:

        listing_pages = await crawler.arun_many(

            listing_urls,

            config={"concurrency": 5, "delay": 2}

        )

Yes—setting concurrency: 1 in your arun_many() config will effectively serialize all your fetches, making it behave just like calling arun() in a simple loop.

# this…

pages = await crawler.arun_many(

  urls,

  config={"concurrency": 1, "delay": 0}

)

# …is equivalent to:

pages = []

for url in urls:

    pages.append(await crawler.arun(url=url))

The only difference is that arun_many still batches everything for you (and can apply a shared retry/backoff strategy), but with concurrency=1 it won’t open more than one page at a time. If you also set delay to 0 (or omit it), you’ll get a straight one-after-another crawl just like repeated arun() cal

crawl4ai how it works in background

 https://github.com/unclecode/crawl4ai?tab=readme-ov-file

crawl4ai uses Playwright under the hood.

crawl4ai-setup installs Chromium browser inside your Docker container

Playwright is a high-level browser-automation library (maintained by Microsoft) that lets you drive Chromium, Firefox or WebKit programmatically—whether headless or headful. It speaks the browser’s DevTools/WebSocket protocol to do things like:

• Navigate to pages and wait for network-idle or specific elements

• Execute JavaScript in the page context (e.g. to render SPAs or extract data)

• Interact with the page (click, type, scroll, hover) just as a real user would

• Intercept and modify network requests (useful for blocking ads, throttling, or API-only crawling)

• Take screenshots, PDFs, trace performance, record videos, etc.

---------------------------

How crawl4ai leverages Playwright + Chromium

1. Browser Installation via crawl4ai-setup

   • When you run crawl4ai-setup, it installs a specific version of the Chromium binary into your Docker image. This ensures that Playwright has a matching browser executable available in the container (no external downloads at runtime).

2. Launching a Headless Browser

   
   const { chromium } = require('playwright');
   const browser = await chromium.launch({ headless: true });
   

   crawl4ai’s core modules call chromium.launch(), creating a fresh browser instance inside Docker.

3. Creating Contexts & Pages

   • It then opens new browser contexts (isolated sessions) and pages for each target URL:

     
     const context = await browser.newContext();
     const page    = await context.newPage();
     await page.goto(targetUrl, { waitUntil: 'networkidle' });
     

4. Rendering & Extraction

   • Because many modern sites render data only after running client-side JavaScript, Playwright ensures the DOM is fully hydrated before crawl4ai extracts HTML or runs page.evaluate() to scrape structured content.

5. Anti-Bot / Stealth Techniques

   • Playwright can mask headless footprints (e.g. user-agent, WebGL vendors, timezone spoofing). crawl4ai may use those capabilities to reduce blocking by sophisticated anti-scraping defenses.

6. Data Output

   • Once the page is loaded and any necessary interactions (clicking “Load More,” scrolling, form-filling) are complete, crawl4ai grabs the HTML or JSON payload and maps it into your result/ folder as a timestamped .json.

---

Why this matters

• Reliability: Unlike a simple HTTP fetch, Playwright can handle endless JavaScript-driven content, login flows, single-page apps, and even CAPTCHAs (with additional tooling).

• Reproducibility: By bundling a fixed-version Chromium in Docker, your builds never break when a new browser release slips in.

• Flexibility: You can script virtually any user interaction—so if a site requires clicking through modals or scrolling to load data, Playwright covers it.

In short, crawl4ai sits on top of Playwright + Chromium to give you a “real browser” crawl, packaged neatly inside Docker for consistent, robust scraping of today’s JavaScript-heavy websites.