2632 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Karl Seguin	56f1b6cc19	Make the CDP read buffer heap allocated & dynamic ... Rather than stack-allocating MAX_MESSAGE_SIZE upfront, we now allocate 32KB and grow the buffer as needed for larger messages, up to MAX_MESSAGE_SIZE. This will reduce memory usage for drivers that don't send huge payloads (like playwright does). While not implemented, this would also enable us to set the MAX_MESSAGE_SIZE at runtime (e.g. via a command line option).	2025-08-29 10:14:58 +08:00
Karl Seguin	67b479b5c8	Merge pull request #983 from lightpanda-io/sigint ... exit the browser on SIGINT signal	2025-08-29 10:10:49 +08:00
Pierre Tachoire	eac2140693	Merge pull request #986 from lightpanda-io/readme-interception ... README: check request interception	2025-08-28 17:19:32 +02:00
Pierre Tachoire	7a3f5de9c2	Merge pull request #985 from lightpanda-io/fulfill-content-type-len ... http: set content_type len on fulfill request	2025-08-28 17:19:23 +02:00
Pierre Tachoire	7005bf2481	README: check request interception	2025-08-28 17:18:42 +02:00
Pierre Tachoire	b80ee3342c	http: set content_type len on fulfill request	2025-08-28 16:28:41 +02:00
Pierre Tachoire	4c7b7b1e60	handle graceful shutdown	2025-08-28 12:44:16 +02:00
Pierre Tachoire	1a4a3608c8	exit the browser on SIGINT signal	2025-08-28 12:44:12 +02:00
Pierre Tachoire	6800d50339	Merge pull request #981 from lightpanda-io/page-navigate-event ... page: ensure page navigate events order	2025-08-27 18:23:22 +02:00
Pierre Tachoire	036f808ec6	page: ensure page navigate events order	2025-08-27 17:36:36 +02:00
Pierre Tachoire	7647ce9e6d	Merge pull request #960 from lightpanda-io/auth-challenge ... auth required interception	2025-08-27 15:34:51 +02:00
Karl Seguin	545d3f81ce	Merge pull request #977 from lightpanda-io/selector_by_ref ... Select is relatively large (64 bytes), pass it by ref	2025-08-27 19:37:36 +08:00
Pierre Tachoire	455615b9c1	Merge pull request #980 from lightpanda-io/update-docker-readme ... Update docker readme	2025-08-27 09:32:41 +02:00
Pierre Tachoire	d0e2a03da5	README: proxy support is ready	2025-08-27 09:30:43 +02:00
Pierre Tachoire	fa408e644c	cs fix	2025-08-27 09:26:10 +02:00
Pierre Tachoire	a22416584d	README: --privileged is not needed anymore	2025-08-27 09:25:51 +02:00
Karl Seguin	b8fc60df45	Merge pull request #975 from lightpanda-io/dynamic_script ... Support dynamic scripts which are added to the DOM before src is set	2025-08-27 05:59:28 +08:00
Karl Seguin	c6455cf02e	Select is relatively large (64 bytes), pass it by ref	2025-08-27 05:55:04 +08:00
Pierre Tachoire	2ac1d39367	Merge pull request #976 from lightpanda-io/webapi_file_placeholder ... The most basic File implementation.	2025-08-26 18:20:53 +02:00
Pierre Tachoire	041e014d68	Merge pull request #970 from lightpanda-io/remove_loop ... Remove the loop	2025-08-26 18:17:32 +02:00
Pierre Tachoire	5defb5c442	http: build headers when auth challenge fails	2025-08-26 18:05:45 +02:00
Pierre Tachoire	520a572bb4	http: add reset and tries for transfer	2025-08-26 18:05:45 +02:00
Pierre Tachoire	4c602256da	http: remove useless field	2025-08-26 18:05:45 +02:00
Pierre Tachoire	5a40cbd655	cdp: use enum for AuthChallengeResponse	2025-08-26 18:05:45 +02:00
Pierre Tachoire	a75f9dd48d	cdp: set default username/passwd for authChallengeResponse	2025-08-26 18:05:44 +02:00
Pierre Tachoire	6b47aa2446	cdp: add auth required interception process	2025-08-26 18:05:44 +02:00
Pierre Tachoire	a847a1faae	http: replace _forbidden with _auth_challenge struct	2025-08-26 18:05:44 +02:00
Pierre Tachoire	bb381e522c	http: add creds into request	2025-08-26 18:05:39 +02:00
Karl Seguin	6962cfb91a	Merge pull request #973 from lightpanda-io/no-body-response ... Handle response without body	2025-08-26 18:44:22 +08:00
Pierre Tachoire	302c50a90e	Merge pull request #964 from lightpanda-io/proxy-header ... http: refacto headerCallback and get proxy CONNECT request details	2025-08-26 10:53:43 +02:00
sjorsdonkers	e2d47e1c86	fix merge conflict	2025-08-26 10:12:07 +02:00
Pierre Tachoire	7d51da1efb	Merge pull request #974 from lightpanda-io/ignore_non_js_script_tags ... Removes the log for unknown script tags	2025-08-26 08:53:29 +02:00
Karl Seguin	c7674926c3	The most basic File implementation. ... Almost silly as-is, but handles this case: ``` if (input instanceof File) { throw Error('file not supported') } ``` as seen on reddit.	2025-08-26 13:25:30 +08:00
Karl Seguin	f0ca9728ae	Support dynamic scripts which are added to the DOM before src is set ... This should load the "src.js": ``` const s = document.createElement('script'); document.getElementsByTagName('body')[0].appendChild(s); s.src = "src.js" ``` Notice that src is set AFTER the element is added to the DOM. This PR enables the above, by 1 - skipping dynamically added scripts which don't have a src 2 - trying to load a script whenever `set_src` is called. (2) is safe because the ScriptManager already prevents scripts from being processed multiple times. Additionally, not only can the src be set after the script is added to the DOM, but onload and onerror can be set after the src: ``` s.src = "src.js" s.onload = ...; s.onerror = ...; ``` This PR also delays reading the onload/onerror callbacks until the script is done loading. This behavior is seen on reddit.	2025-08-26 13:10:55 +08:00
Karl Seguin	5fa8567801	Removes the log for unknown script tags ... Some sites have a lot of text/template or application/json, and it just adds noise to the logs.	2025-08-26 08:48:29 +08:00
sjorsdonkers	e5b1acb6e1	Handle response without body	2025-08-25 18:07:02 +02:00
Karl Seguin	8fdbaef4c7	Use posix.TCP.NODELAY now that it's available in MacOS also	2025-08-25 22:03:58 +08:00
Pierre Tachoire	7869159657	add e2e test through proxy	2025-08-25 14:18:15 +02:00
Pierre Tachoire	7046e18d7e	http: simplify header parsing	2025-08-25 14:18:14 +02:00
Pierre Tachoire	a7516061d0	http: move use_proxy from connection to client	2025-08-25 14:18:14 +02:00
Pierre Tachoire	e61d787ff0	http: move header done callback in its own func ... And call it only after the headers are parsed, either from data callback or end of the request.	2025-08-25 14:18:14 +02:00
Pierre Tachoire	25ad420f85	http: ajust header callback according to review	2025-08-25 14:18:14 +02:00
Pierre Tachoire	fcd49c000f	page: avoid crash on empty body	2025-08-25 14:18:13 +02:00
Pierre Tachoire	e2320ebe66	http: handle proxy's request header callback	2025-08-25 14:18:13 +02:00
Pierre Tachoire	5e78a26e3d	http: refacto http header parsing	2025-08-25 14:18:13 +02:00
Pierre Tachoire	159bd06a56	http: add use_proxy bool in connection	2025-08-25 14:18:12 +02:00
Pierre Tachoire	bc7e1e07f4	typo fix	2025-08-25 14:18:08 +02:00
Karl Seguin	ccc9618102	Merge pull request #971 from lightpanda-io/fix-send-error-json-format ... Fix sendError message's format	2025-08-25 19:05:47 +08:00
sjorsdonkers	0ad09cca9d	Fix sendError message's format	2025-08-25 12:51:47 +02:00
Karl Seguin	0959eea677	Remove the loop ... Previously, the IO loop was doing three things: 1 - Managing timeouts (either from scripts or for our own needs) 2 - Handling browser IO events (page/script/xhr) 3 - Handling CDP events (accept, read, write, timeout) With the libcurl merge, 1 was moved to an in-process scheduler and 2 was moved to libcurl's own event loop. That means the entire loop code, including the dependency on tigerbeetle-io existed for handling a single TCP client. Not only is that a lot of code, there was also friction between the two loops (the libcurl one and our IO loop), which would result in latency - while one loop is waiting for the events, any events on the other loop go un-processed. This PR removes our IO loop. To accomplish this: 1 - The main accept loop is blocking. This is simpler and works perfectly well, given we only allow 1 active connection. 2 - The client socket is passed to libcurl - yes, libcurl's loop can take arbitrary FDs and poll them along with its own. In addition to having one less dependency, the CDP code is quite a bit simpler, especially around shutdowns and writes. This also removes _some_ of the latency caused by the friction between page process and CDP processing. Specifically, when CDP now blocks for input, http page events (script loading, xhr, ...) will still be processed. There's still friction. For one, the reverse isn't true: when the page is waiting for events, CDP events aren't going to be processed. But the page.wait already have some sensitivity to this (e.g. the page.request_intercepted flag). Also, when CDP waits, while we will process network events, page timeouts are still not processed. Because of both these remaining issues, we still need to jump between the two loops - but being able to block on CDP (even for a short time) WITHOUT stopping the page's network I/O, should reduce some latency.	2025-08-25 17:27:28 +08:00