01ecb296e5
This commit involves a number of changes to finalizers, all aimed towards better consistency and reliability. A big part of this has to do with v8::Inspector's ability to move objects across IsolatedWorlds. There has been a few previous efforts on this, the most significant being https://github.com/lightpanda-io/browser/pull/1901. To recap, a Zig instance can map to 0-N v8::Objects. Where N is the total number of IsolatedWorlds. Generally, IsolatedWorlds between origins are...isolated...but the v8::Inspector isn't bound by this. So a Zig instance cannot be tied to a Context/Identity/IsolatedWorld...it has to live until all references, possibly from different IsolatedWorlds, are released (or the page is reset). Finalizers could previously be managed via reference counting or explicitly toggling the instance as weak/strong. Now, only reference counting is supported. weak/strong can essentially be seen as an acquireRef (rc += 1) and releaseRef (rc -= 1). Explicit setting did make some things easier, like not having to worry so much about double-releasing (e.g. XHR abort being called multiple times), but it was only used in a few places AND it simply doesn't work with objects shared between IsolatedWorlds. It is never a boolean now, as 3 different IsolatedWorlds can each hold a reference. Temps and Globals are tracked on the Session. Previously, they were tracked on the Identity, but that makes no sense. If a Zig instance can outlive an Identity, then any of its Temp references can too. This hasn't been a problem because we've only seen MutationObserver and IntersectionObserver be used cross-origin, but the right CDP script can make this crash with a use-after-free (e.g. `MessageEvent.data` is released when the Identity is done, but `MessageEvent` is still referenced by a different IsolateWorld). Rather than deinit with a `comptime shutdown: bool`, there is now an explicit `releaseRef` and `deinit`. Bridge registration has been streamlined. Previously, types had to register their finalizer AND acquireRef/releaseRef/deinit had to be declared on the entire prototype chain, even if these methods just delegated to their proto. Finalizers are now automatically enabled if a type has a `acquireRef` function. If a type has an `acquireRef`, then it must have a `releaseRef` and a `deinit`. So if there's custom cleanup to do in `deinit`, then you also have to define `acquireRef` and `releaseRef` which will just delegate to the _proto. Furthermore these finalizer methods can be defined anywhere on the chain. Previously: ```zig const KeywboardEvent = struct { _proto: *Event, ... pub fn deinit(self: *KeyboardEvent, session: *Session) void { self._proto.deinit(session); } pub fn releaseRef(self: *KeyboardEvent, session: *Session) void { self._proto.releaseRef(session); } } ``` ```zig const KeyboardEvent = struct { _proto: *Event, ... // no deinit, releaseRef, acquireref } ``` Since the `KeyboardEvent` doesn't participate in finalization directly, it doesn't have to define anything. The bridge will detect the most specific place they are defined and call them there.
Lightpanda Browser
The headless browser built from scratch for AI agents and automation.
Not a Chromium fork. Not a WebKit patch. A new browser, written in Zig.
chromedp requesting 933 real web pages over the network on a AWS EC2 m5.large instance. See benchmark details.
Lightpanda is the open-source browser made for headless usage:
- Javascript execution
- Support of Web APIs (partial, WIP)
- Compatible with Playwright1 , Puppeteer, chromedp through CDP
Fast web automation for AI agents, LLM training, scraping and testing:
- Ultra-low memory footprint (9x less than Chrome)
- Exceptionally fast execution (11x faster than Chrome)
- Instant startup
Quick start
Install
Install from the nightly builds
You can download the last binary from the nightly builds for Linux x86_64 and MacOS aarch64.
For Linux
curl -L -o lightpanda https://github.com/lightpanda-io/browser/releases/download/nightly/lightpanda-x86_64-linux && \
chmod a+x ./lightpanda
For MacOS
curl -L -o lightpanda https://github.com/lightpanda-io/browser/releases/download/nightly/lightpanda-aarch64-macos && \
chmod a+x ./lightpanda
For Windows + WSL2
The Lightpanda browser is compatible to run on windows inside WSL. Follow the Linux instruction for installation from a WSL terminal. It is recommended to install clients like Puppeteer on the Windows host.
Install from Docker
Lightpanda provides official Docker
images for both Linux amd64 and
arm64 architectures.
The following command fetches the Docker image and starts a new container exposing Lightpanda's CDP server on port 9222.
docker run -d --name lightpanda -p 9222:9222 lightpanda/browser:nightly
Dump a URL
./lightpanda fetch --obey-robots --log-format pretty --log-level info https://demo-browser.lightpanda.io/campfire-commerce/
INFO telemetry : telemetry status . . . . . . . . . . . . . [+0ms]
disabled = false
INFO page : navigate . . . . . . . . . . . . . . . . . . . . [+6ms]
url = https://demo-browser.lightpanda.io/campfire-commerce/
method = GET
reason = address_bar
body = false
req_id = 1
INFO browser : executing script . . . . . . . . . . . . . . [+118ms]
src = https://demo-browser.lightpanda.io/campfire-commerce/script.js
kind = javascript
cacheable = true
INFO http : request complete . . . . . . . . . . . . . . . . [+140ms]
source = xhr
url = https://demo-browser.lightpanda.io/campfire-commerce/json/product.json
status = 200
len = 4770
INFO http : request complete . . . . . . . . . . . . . . . . [+141ms]
source = fetch
url = https://demo-browser.lightpanda.io/campfire-commerce/json/reviews.json
status = 200
len = 1615
<!DOCTYPE html>
Start a CDP server
./lightpanda serve --obey-robots --log-format pretty --log-level info --host 127.0.0.1 --port 9222
INFO telemetry : telemetry status . . . . . . . . . . . . . [+0ms]
disabled = false
INFO app : server running . . . . . . . . . . . . . . . . . [+0ms]
address = 127.0.0.1:9222
Once the CDP server started, you can run a Puppeteer script by configuring the
browserWSEndpoint.
'use strict'
import puppeteer from 'puppeteer-core';
// use browserWSEndpoint to pass the Lightpanda's CDP server address.
const browser = await puppeteer.connect({
browserWSEndpoint: "ws://127.0.0.1:9222",
});
// The rest of your script remains the same.
const context = await browser.createBrowserContext();
const page = await context.newPage();
// Dump all the links from the page.
await page.goto('https://demo-browser.lightpanda.io/amiibo/', {waitUntil: "networkidle0"});
const links = await page.evaluate(() => {
return Array.from(document.querySelectorAll('a')).map(row => {
return row.getAttribute('href');
});
});
console.log(links);
await page.close();
await context.close();
await browser.disconnect();
Telemetry
By default, Lightpanda collects and sends usage telemetry. This can be disabled by setting an environment variable LIGHTPANDA_DISABLE_TELEMETRY=true. You can read Lightpanda's privacy policy at: https://lightpanda.io/privacy-policy.
Status
Lightpanda is in Beta and currently a work in progress. Stability and coverage are improving and many websites now work. You may still encounter errors or crashes. Please open an issue with specifics if so.
Here are the key features we have implemented:
- HTTP loader (Libcurl)
- HTML parser (html5ever)
- DOM tree
- Javascript support (v8)
- DOM APIs
- Ajax
- XHR API
- Fetch API
- DOM dump
- CDP/websockets server
- Click
- Input form
- Cookies
- Custom HTTP headers
- Proxy support
- Network interception
- Respect
robots.txtwith option--obey-robots
NOTE: There are hundreds of Web APIs. Developing a browser (even just for headless mode) is a huge task. Coverage will increase over time.
Build from sources
Prerequisites
Lightpanda is written with Zig 0.15.2. You have to
install it with the right version in order to build the project.
Lightpanda also depends on v8, Libcurl and html5ever.
To be able to build the v8 engine, you have to install some libs:
For Debian/Ubuntu based Linux:
sudo apt install xz-utils ca-certificates \
pkg-config libglib2.0-dev \
clang make curl git
You also need to install Rust.
For systems with Nix, you can use the devShell:
nix develop
For MacOS, you need cmake and Rust.
brew install cmake
Build and run
You an build the entire browser with make build or make build-dev for debug
env.
But you can directly use the zig command: zig build run.
Embed v8 snapshot
Lighpanda uses v8 snapshot. By default, it is created on startup but you can embed it by using the following commands:
Generate the snapshot.
zig build snapshot_creator -- src/snapshot.bin
Build using the snapshot binary.
zig build -Dsnapshot_path=../../snapshot.bin
See #1279 for more details.
Test
Unit Tests
You can test Lightpanda by running make test.
End to end tests
To run end to end tests, you need to clone the demo
repository into ../demo dir.
You have to install the demo's node requirements
You also need to install Go > v1.24.
make end2end
Web Platform Tests
Lightpanda is tested against the standardized Web Platform Tests.
We use a fork including a custom
testharnessreport.js.
For reference, you can easily execute a WPT test case with your browser via wpt.live.
Configure WPT HTTP server
To run the test, you must clone the repository, configure the custom hosts and generate the
MANIFEST.json file.
Clone the repository with the fork branch.
git clone -b fork --depth=1 git@github.com:lightpanda-io/wpt.git
Enter into the wpt/ dir.
Install custom domains in your /etc/hosts
./wpt make-hosts-file | sudo tee -a /etc/hosts
Generate MANIFEST.json
./wpt manifest
Use the WPT's setup guide for details.
Run WPT test suite
An external Go runner is provided by
github.com/lightpanda-io/demo/
repository, located into wptrunner/ dir.
You need to clone the project first.
First start the WPT's HTTP server from your wpt/ clone dir.
./wpt serve
Run a Lightpanda browser
zig build run -- --insecure-disable-tls-host-verification
Then you can start the wptrunner from the Demo's clone dir:
cd wptrunner && go run .
Or one specific test:
cd wptrunner && go run . Node-childNodes.html
wptrunner command accepts --summary and --json options modifying output.
Also --concurrency define the concurrency limit.
⚠️ Running the whole test suite will take a long time. In this case,
it's useful to build in releaseFast mode to make tests faster.
zig build -Doptimize=ReleaseFast run
Contributing
Lightpanda accepts pull requests through GitHub.
You have to sign our CLA during the pull request process otherwise we're not able to accept your contributions.
Why?
Javascript execution is mandatory for the modern web
In the good old days, scraping a webpage was as easy as making an HTTP request, cURL-like. It’s not possible anymore, because Javascript is everywhere, like it or not:
- Ajax, Single Page App, infinite loading, “click to display”, instant search, etc.
- JS web frameworks: React, Vue, Angular & others
Chrome is not the right tool
If we need Javascript, why not use a real web browser? Take a huge desktop application, hack it, and run it on the server. Hundreds or thousands of instances of Chrome if you use it at scale. Are you sure it’s such a good idea?
- Heavy on RAM and CPU, expensive to run
- Hard to package, deploy and maintain at scale
- Bloated, lots of features are not useful in headless usage
Lightpanda is built for performance
If we want both Javascript and performance in a true headless browser, we need to start from scratch. Not another iteration of Chromium, really from a blank page. Crazy right? But that’s what we did:
- Not based on Chromium, Blink or WebKit
- Low-level system programming language (Zig) with optimisations in mind
- Opinionated: without graphical rendering
-
Playwright support disclaimer: Due to the nature of Playwright, a script that works with the current version of the browser may not function correctly with a future version. Playwright uses an intermediate JavaScript layer that selects an execution strategy based on the browser's available features. If Lightpanda adds a new Web API, Playwright may choose to execute different code for the same script. This new code path could attempt to use features that are not yet implemented. Lightpanda makes an effort to add compatibility tests, but we can't cover all scenarios. If you encounter an issue, please create a GitHub issue and include the last known working version of the script. ↩︎