Commit Graph

292 Commits

Author SHA1 Message Date
Karl Seguin
c96fb3c2f2 support CDP proxy override 2025-08-11 21:37:03 +08:00
Karl Seguin
3555680335 Working navigation events (clicks, form submission) 2025-08-11 21:37:01 +08:00
Karl Seguin
f65a39a3e3 Re-enable telemetry
Start work on supporting navigation events (clicks, form submission).
2025-08-11 21:37:00 +08:00
Karl Seguin
54ab1326e5 Switch XHR to new http client
get puppeteer/cdp.js working again

make test are all passing
2025-08-11 21:37:00 +08:00
Karl Seguin
b0fe5d60ab Initial work on integrating libcurl and making all http nonblocking 2025-08-11 21:36:56 +08:00
Pierre Tachoire
7aff90aec7 cdp: add comment for CDP_USER_AGENT 2025-08-04 14:40:44 +02:00
sjorsdonkers
478f3a5308 simplify statusText
Some checks failed
e2e-test / zig build release (push) Has been cancelled
zig-test / zig build dev (push) Has been cancelled
zig-test / zig test (push) Has been cancelled
e2e-test / demo-scripts (push) Has been cancelled
e2e-test / cdp-and-hyperfine-bench (push) Has been cancelled
e2e-test / perf-fmt (push) Has been cancelled
zig-test / browser fetch (push) Has been cancelled
zig-test / perf-fmt (push) Has been cancelled
nightly build / build-linux-x86_64 (push) Has been cancelled
nightly build / build-linux-aarch64 (push) Has been cancelled
nightly build / build-macos-aarch64 (push) Has been cancelled
nightly build / build-macos-x86_64 (push) Has been cancelled
wpt / web platform tests json output (push) Has been cancelled
wpt / perf-fmt (push) Has been cancelled
2025-07-29 09:53:54 +02:00
sjorsdonkers
b98edf3d76 CDP response statusText 2025-07-29 09:53:54 +02:00
Karl Seguin
eef5f3fec2 support null params to CDP DOM.getDocument 2025-07-17 19:05:17 +08:00
Karl Seguin
f199816fcd support depth parameter for DOM.getDocument 2025-07-17 14:17:33 +08:00
Karl Seguin
5e74e17b41 Merge pull request #888 from lightpanda-io/cdp_dom_requestChildNodes
Some checks failed
e2e-test / zig build release (push) Has been cancelled
e2e-test / demo-scripts (push) Has been cancelled
e2e-test / cdp-and-hyperfine-bench (push) Has been cancelled
e2e-test / perf-fmt (push) Has been cancelled
zig-test / zig build dev (push) Has been cancelled
zig-test / browser fetch (push) Has been cancelled
zig-test / zig test (push) Has been cancelled
zig-test / perf-fmt (push) Has been cancelled
Add support for CDP's DOM.requestChildNodes
2025-07-17 10:48:24 +08:00
Karl Seguin
98b041e84a requestChildNode cannot have a depth of 0 2025-07-17 10:36:20 +08:00
Karl Seguin
1a05fe6ae1 Merge pull request #887 from lightpanda-io/go_rod
Some checks failed
e2e-test / zig build release (push) Has been cancelled
e2e-test / demo-scripts (push) Has been cancelled
e2e-test / cdp-and-hyperfine-bench (push) Has been cancelled
e2e-test / perf-fmt (push) Has been cancelled
zig-test / zig build dev (push) Has been cancelled
zig-test / browser fetch (push) Has been cancelled
zig-test / zig test (push) Has been cancelled
zig-test / perf-fmt (push) Has been cancelled
nightly build / build-linux-x86_64 (push) Has been cancelled
nightly build / build-linux-aarch64 (push) Has been cancelled
nightly build / build-macos-aarch64 (push) Has been cancelled
nightly build / build-macos-x86_64 (push) Has been cancelled
wpt / web platform tests json output (push) Has been cancelled
wpt / perf-fmt (push) Has been cancelled
Noop CDP methods that go-rod requires
2025-07-16 20:01:03 +08:00
sjorsdonkers
16fcbf66ee http_proxy_before ?? comment 2025-07-16 11:20:00 +02:00
Karl Seguin
09ca0e6ef0 Add support for CDP's DOM.requestChildNodes
https://github.com/lightpanda-io/browser/issues/866
2025-07-14 15:13:01 +08:00
Karl Seguin
fae2b5acfa Noop CDP methods that go-rod requires
go-rod appears to stop processing when it receives an error, such as
UnknownMethod. Added placeholder handlers for Network.setUserAgentOverride and
Page.stopLoading.

Setting a custom user agent is something still being discussed, so no-oping it
seems reasonable. And, due to the currently synchronous nature of the initial
page load, no-oping stopLoading also seems reasonable.

https://github.com/lightpanda-io/browser/issues/867
2025-07-14 11:21:02 +08:00
Karl Seguin
1602932d72 Add a "pre" polyfill
This is always run, but only the full webcomponents polyfill, it's very
small and isn't intrusive. This introduces a layer of indirection so that,
if the full polyfill is loaded, its monkeypatched constructor will be called
2025-07-12 19:49:19 +08:00
Pierre Tachoire
2cdc9e9f5f cdp: use a polyfill loader per isolate 2025-07-07 16:31:54 -07:00
Pierre Tachoire
941dace7f9 enable conditionnal loading for polyfill 2025-07-07 16:31:53 -07:00
Karl Seguin
0b846b15b1 Merge pull request #789 from lightpanda-io/browsercontext-proxyServer
browser context proxyServer
2025-06-19 10:22:17 +08:00
sjorsdonkers
60f4eab759 handle no params 2025-06-18 10:07:37 +02:00
sjorsdonkers
d7656ea985 expires dashes and f64 2025-06-18 10:07:37 +02:00
sjorsdonkers
e402998577 JS may not set/get HttpOnly cookies 2025-06-18 10:07:37 +02:00
sjorsdonkers
073f75efa3 CDP Network cookie tests 2025-06-18 10:07:37 +02:00
sjorsdonkers
da414f7eb3 CDP.Storage cookies tests 2025-06-18 10:07:37 +02:00
sjorsdonkers
270b89830a Cleaning up crumbles 2025-06-18 10:07:37 +02:00
sjorsdonkers
74ce7ca416 refactor path / domain parsing 2025-06-18 10:07:37 +02:00
sjorsdonkers
3f4338cb51 wip 2025-06-18 10:07:37 +02:00
sjorsdonkers
30ee41fd0e Network.getCookies 2025-06-18 10:07:37 +02:00
sjorsdonkers
4965fec55c storage cookies 2025-06-18 10:07:37 +02:00
sjorsdonkers
18dff8455c lower case domain 2025-06-18 10:07:37 +02:00
sjorsdonkers
fe16f06aee clearRetainingCapacity 2025-06-18 10:07:37 +02:00
sjorsdonkers
48c1c05a93 setCookie 2025-06-18 10:07:37 +02:00
sjorsdonkers
38dee1166d setCookies 2025-06-18 10:07:37 +02:00
sjorsdonkers
0c6fc68eae deleteCookies 2025-06-18 10:07:37 +02:00
sjorsdonkers
6f5141d5fb browser context proxyServer 2025-06-17 18:43:12 +02:00
Karl Seguin
c28d87d59c Improve build and test speed
Test speed has been improved only slightly by tweaking a 2-second running tests.

Build has been improved by:
1 - moving logFunctionCallError out of js.Caller and to a standalone function
2 - removing some non-generic code from the generic portions of the logger

Caller.getter and Caller.setter have been removed in favor or calling
Caller.method. This wasn't previously possible - prior to our v8 upgrade, they
had different signatures.

Also removed a largely unused parser/str.zig file.
2025-06-16 19:50:13 +08:00
sjorsdonkers
0c0ddc10ee rename scope jscontext
Some checks failed
e2e-test / zig build release (push) Has been cancelled
zig-test / zig build dev (push) Has been cancelled
zig-test / zig test (push) Has been cancelled
e2e-test / puppeteer-perf (push) Has been cancelled
e2e-test / demo-scripts (push) Has been cancelled
e2e-test / cdp-and-hyperfine-bench (push) Has been cancelled
e2e-test / perf-fmt (push) Has been cancelled
zig-test / browser fetch (push) Has been cancelled
zig-test / perf-fmt (push) Has been cancelled
nightly build / build-linux-x86_64 (push) Has been cancelled
nightly build / build-linux-aarch64 (push) Has been cancelled
nightly build / build-macos-aarch64 (push) Has been cancelled
nightly build / build-macos-x86_64 (push) Has been cancelled
wpt / web platform tests json output (push) Has been cancelled
wpt / perf-fmt (push) Has been cancelled
2025-06-13 10:30:50 +02:00
Karl Seguin
9bd5ff69ef Merge pull request #779 from lightpanda-io/waitForNavigation
Some checks failed
e2e-test / zig build release (push) Has been cancelled
e2e-test / puppeteer-perf (push) Has been cancelled
e2e-test / demo-scripts (push) Has been cancelled
e2e-test / cdp-and-hyperfine-bench (push) Has been cancelled
e2e-test / perf-fmt (push) Has been cancelled
zig-test / zig build dev (push) Has been cancelled
zig-test / browser fetch (push) Has been cancelled
zig-test / zig test (push) Has been cancelled
zig-test / perf-fmt (push) Has been cancelled
Internal navigation should change the CDP loader-id
2025-06-13 09:38:33 +08:00
Karl Seguin
872a9d393d Internal navigation should change the CDP loader-id
This is one of the ways that puppeteer knows that navigation happened
and is needed to support `waitForNavigation` which compares the
existing loader-id with the new one, so it has to change.

Also, fix a crash that could happen if CDP disconnects while
connections are being aborted.
2025-06-12 19:11:26 +08:00
Karl Seguin
97c769e805 Rework internal navigation to prevent deadlocking
The mix of sync and async HTTP requests requires care to avoid deadlocks.
Previously, it was possible for async requests to use up all available HTTP
state objects duration a navigation flow (either directly, or via an internal
redirect (e.g. click, submit, ...)). This would block the navigation, which,
because everything is single thread, would block the I/O loop, resulting in a
deadlock.

The correct solution seems to be to remove all synchronous I/O. And I tried to
do that, but I ran into a wall with module-loading, which is initiated from V8.
V8 says "give me the source for this module", and I don't see a great way to
tell it: wait a bit.

So I went back to trying to make this work with the hybrid model, despite last
weeks failures to get it to work. I changed two things:

1 - The http client will only directly initiate an async request if there's
    at least 2 free state objects available (1 for the request, and leaving 1
    free for any synchronous requests)

2 - Delayed navigation retries until there's at least 1 free http state object
    available.

Commits from last week did help with this. First, we're now guaranteed to have
a single sync-request at a time (previously, we could have had 2). Secondly,
the async connection is now async end-to-end (previously, it could have blocked
on an empty state pool).

We could probably make this a bit more obviously by reserving 1 state object
for synchronous requests. But, since the long term solution is probably having
no synchronous requests, I'm happy with anything that lets me move past this
issue.
2025-06-12 12:34:51 +08:00
Karl Seguin
34b49498c9 Making sure that the optional parameters have defaults
Co-authored-by: Sjors <72333389+sjorsdonkers@users.noreply.github.com>
2025-06-12 09:46:01 +08:00
Karl Seguin
effd07d8c0 Add basic support for key events
Support CDP's Input.dispatchKeyEvent and DOM key events. Currently only
keydown is supported and expects every key to be a displayable character.

It turns out that manipulating the DOM via key events isn't great because the
behavior really depends on the cursor. So, to do this more accurately, we'd
have to introduce some concept of a cursor.

Personally, I don't think we'll run into many pages that are purposefully
using keyboard events. But driver (puppeteer/playwright) scripts might be
another issue.
2025-06-12 09:45:59 +08:00
sjorsdonkers
d9ce89ab31 import html_element
Some checks failed
e2e-test / zig build release (push) Has been cancelled
zig-test / zig build dev (push) Has been cancelled
zig-test / zig test (push) Has been cancelled
e2e-test / puppeteer-perf (push) Has been cancelled
e2e-test / demo-scripts (push) Has been cancelled
e2e-test / cdp-and-hyperfine-bench (push) Has been cancelled
e2e-test / perf-fmt (push) Has been cancelled
zig-test / browser fetch (push) Has been cancelled
zig-test / perf-fmt (push) Has been cancelled
nightly build / build-linux-x86_64 (push) Has been cancelled
nightly build / build-linux-aarch64 (push) Has been cancelled
nightly build / build-macos-aarch64 (push) Has been cancelled
nightly build / build-macos-x86_64 (push) Has been cancelled
wpt / web platform tests json output (push) Has been cancelled
wpt / perf-fmt (push) Has been cancelled
2025-06-10 11:28:28 +02:00
Karl Seguin
2b48902f1b Emit http_request_fail notification
CDP translate this into a Network.loadingFailed. This is necessary to make sure
every Network.requestWillBeSent is paired with either a Network.loadingFailed
or a Network.responseReceived.
2025-06-06 19:15:47 +08:00
Karl Seguin
305460dedb Merge pull request #768 from lightpanda-io/setExtraHTTPHeaders
Some checks failed
e2e-test / zig build release (push) Has been cancelled
e2e-test / puppeteer-perf (push) Has been cancelled
e2e-test / demo-scripts (push) Has been cancelled
e2e-test / cdp-and-hyperfine-bench (push) Has been cancelled
e2e-test / perf-fmt (push) Has been cancelled
zig-test / zig build dev (push) Has been cancelled
zig-test / browser fetch (push) Has been cancelled
zig-test / zig test (push) Has been cancelled
zig-test / perf-fmt (push) Has been cancelled
setExtraHTTPHeaders
2025-06-06 16:45:07 +08:00
sjorsdonkers
bacef41a3b extra header feedback 2025-06-06 10:33:15 +02:00
Karl Seguin
fdd1a778f3 Properly drain event loop when navigating between pages 2025-06-06 12:53:45 +08:00
sjorsdonkers
f1672dd6d2 setExtraHTTPHeaders 2025-06-05 16:42:29 +02:00
Karl Seguin
74b36d6d32 support form.submit()
Only supports application/x-www-form-urlencoded
2025-05-29 14:10:07 +08:00