From c68692d78ea7c7dc075a2e4f31a0dd6d32aad795 Mon Sep 17 00:00:00 2001 From: Pierre Tachoire Date: Tue, 10 Feb 2026 14:24:38 +0100 Subject: [PATCH] add `--obey_robots` into README examples --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 521c8685..3baa7b7f 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ docker run -d --name lightpanda -p 9222:9222 lightpanda/browser:nightly ### Dump a URL ```console -./lightpanda fetch --dump https://lightpanda.io +./lightpanda fetch --obey_robots --dump https://lightpanda.io ``` ```console info(browser): GET https://lightpanda.io/ http.Status.ok @@ -90,7 +90,7 @@ info(browser): eval remote https://api.website.lightpanda.io/js/script.js: TypeE ### Start a CDP server ```console -./lightpanda serve --host 127.0.0.1 --port 9222 +./lightpanda serve --obey_robots --host 127.0.0.1 --port 9222 ``` ```console info(websocket): starting blocking worker to listen on 127.0.0.1:9222 @@ -156,6 +156,7 @@ Here are the key features we have implemented: - [x] Custom HTTP headers - [x] Proxy support - [x] Network interception +- [x] Respect `robots.txt` with option `--obey_robots` NOTE: There are hundreds of Web APIs. Developing a browser (even just for headless mode) is a huge task. Coverage will increase over time.