Playwright, when creating a new CDPSession, sends an
attachToBrowserTarget followed by another attachToTarget to re-attach
itself to the existing target.
see playwright/axtree.js from demo/ repository.
Most significantly, if removing from the multi fails, the connection
is added to a "dirty" list for the removal to be retried later. Looking at
the curl source code, remove fails on a recursive call, and we've struggled with
recursive calls before, so I _think_ this might be happening (it fails in other
cases, but I suspect if it _is_ happening, it's for this reason). The retry
happens _after_ `perform`, so it cannot fail for due to recursiveness. If it
fails at this point, we @panic. This is harsh, but it isn't easily recoverable
and before putting effort into it, I'd like to know that it's actually happening.
Fix potential use of undefined when a 401-407 request is received, but no
'WWW-Authenticate' or 'Proxy-Authenticate' header is received.
Don't call `curl_multi_remove_handle` on an easy that hasn't been added yet do
to error. Specifically, if `makeRequest` fails during setup, transfer_conn is
nulled so that `transfer.deinit()` doesn't try to remove the connection. And the
conn is removed from the `in_use` queue and made `available` again.
On Abort, if getting the private fails (extremely unlikely), we now still try
to remove the connection from the multi.
Added a few more fields to the famous "ScriptManager.Header recall" assertion.
This PR introduces a custom CDP domain 'LP' (Lightpanda) to expose browser-specific tools. The first method, 'LP.getMarkdown', allows retrieving a Markdown representation of the DOM or a specific node by its 'nodeId'. This is optimized for AI agents and LLM-based scraping tasks.
This field was recently added and is used to generate correct frameIds in CDP
messages. They remain the same during a navigation event, so calling them
page.id might cause surprises since navigation events create new pages, but
retain the original id. Hence, frame_id is more accurate and hopefully less
surprising.
(This is a small cleanup prior to doing some iframe navigation work).
https://github.com/lightpanda-io/browser/pull/1614 improved our shutdown
behavior so that microtasks associated with a context wouldn't fire after the
context was disposed of. This involved having context-specific microtasks,
pumping the message loop, and prevent re-entry.
The shutdown code in CDP already had much of this behavior built-in, but it has
now become redundant. Most importantly the CDP shutdown logic did not prevent
re-entry.
Removing this code fixes a flaky WPT crash. I didn't seem to be tied to a
specific test, but rather a cross-context/page use-after-free that was saw
prior to 1614. I could reproduce it reliably by running `/wasm/core/`.
I'll be honest, it isn't clear to me why _removing_ the CDP cleanup helps.
Running the message loop and microtask _before_ our normal shutdown might be
unnecessary, but why would it crash? I don't know, but the CDP path is slightly
different in that it also involves Inspector shutdown. So there's still
something about this flow I don't quite understand. And, at least for this case
the current flow seems "correct".
I think this code comes from some serialization tweak from when everything was
an std.Uri and by switch to [:0]const u8 everywhere not only was the tweak
unecessary, it was also wrong - possibly resulting in the generation of
invalid JSON.
After looking at a handful of websites, the # of Text and Commend nodes
that are small (<= 12 bytes) is _really_ high. Ranging from 85% to 98%. I
thought that was high, but a lot of it is indentation or a sentence that's
broken down into multiple nodes, eg:
<div><b>sale!</b> <span class=price>$1.99</span> buy now<div>
So what looks like 1 sentence to us, is actually 3 text nodes.
On a typical website, we should see thousands of fewer allocations in the
page arena for the text in text nodes.
Most importantly, this allows the Selector.List to be self-contained with
an arena from the ArenaPool. Selector.List can be both relatively large
and relatively common, so moving it off the page.arena is a nice win.
Also applied this to ChildNodes, which is much smaller but could also be
called often.
I was initially going to hook into the v8::Object's internal fields to store
the referencing v8::Object. So the v8::Object representing the Iterator
would store the v8::Object representing the NodeList inside of its internal
field - which the GC would trace/detect/respect. And that is probably the
fastest and most v8-ish solution, but I couldn't come up with an elegant
solution. The best I had was having a "afterCreate" callback which passed
the v8 object (this is similar to the old postAttach callback we had, but
used for a different purpose). However, since "acquireRef" was recently
added to events, re-using that was much simpler and worked well.
Follow up to https://github.com/lightpanda-io/browser/pull/1646
The encodeURL (renamed to ensureEncoded and exposed in this commit) already
handled already-encoded URLs, so this was largely a matter of exposing the
functionality.
The reason this isn't baked directly into Page.navigate is that, in some places
e.g. internal navigation, the URL is already know to be encoded. So it's up
to every caller to make sure they are passing a valid URL to navigate.
Depends on: https://github.com/lightpanda-io/zig-v8-fork/pull/152
We previously ran the message loop every 250ms. This commit changes it to run on
every tick (much more frequently). It also runs microtasks after draining the
message loop (since it can generate microtasks).
Also, we use to run microtasks after each script execution. Now we drain the
message Loop + microtasks.
We still only drain the microtasks when executing v8 callbacks.
As part of this change, we also adjust our wait time based on whether or not
there are pending background tasks in v8 in order to try to execute them (in
general) and in a timely manner.
The goal is to ensure that tasks v8 enqueued on the foreground thread are
executed promptly.
This change is particularly useful for calls to webassembly as compilation
happens in the background and eventually requires the message loop to be drained
to continue.
Previously, if a script did `await WebAssembly.instantiate(....)`, there was
a good chance we'd never finish the code - we'd wait too long to run the
message loop AND, after running it, we wouldn't necessarily resolve the promise.
Under some conditions, a microtask would be executed for a context that was
already deinit'd, resulting in various use-after-free.
The culprit appears to be WASM compilation being placed in the microtask queue
(by a user-script) and then resolved at some point in the future. We guard the
microtask queue by a context.shutting_down boolean, but v8 doesn't know anything
about this flag. The fact is that, microtasks are tied to an isolate, not a
context.
This commit introduces a number of changes:
1 - It follows 309f254c2c and stores the zig Context inside of an embedder field. This
ensures v8 doesn't consider this when GC'ing, which _could_ extend the
lifetime of the v8::Context beyond what we expect
2 - Most significantly, it introduces per-context microtasks queues. Each
context gets its own queue. This makes cleanup much simpler and reduces the
chance of microtasks outliving the context
3 - pumpMessageLoop is called on context.deinit, this helps to ensure that any
tasks v8 has for our context are processed (e.g. wasm compilation) before
shtudown
4 - The order of context shutdown is important, we notify the isolate of the
context destruction first, then pump the message loop and finally destroy
the context's message loop.
Depends on https://github.com/lightpanda-io/zig-v8-fork/pull/151
Missing:
- [ ] Navigation support within frames (in fact, as-is, any navigation done
inside a frame, will almost certainly break things
- [ ] Correct CDP support. I don't know how frames are supposed to be exposed
to CDP. Normal navigate events? Distinct CDP frame_ids?
- [ ] Cross-origin restrictions. The interaction between frames is supposed to
change depending on whether or not they're on the same origin
- [ ] Potentially handling src-less frames incorrectly. Might not really matter
Adds basic frame support. Initially explored adding a BrowsingContext and
embedding it in Page, with the goal of also having it embedded in a to-be
created Frame. But it turns out that 98% of Page _was_ BrowsingContext and
introducing a BrowsingContext as the primary interaction unit broke pretty much
_every_ single WebAPI. So Page was expanded:
- Added `_parent: ?*Page`, which is `null` for "root" page.
- Added `frame: ?*IFrame`, which is `null` for the "root" page. This is the
HTMLIFrameElement for frame-pages.
- Added a _type: enum{root, frame}, which is currently only used to improve
the logs
- Added a frames: std.ArrayList(*Page). This is a list of frames for the page.
Note that a "frame-page" can itself haven nested frames.
Besides the above, there were 3 "big" changes.
1 - Adding frames (dynamically, parsed) has to create a new page, start
navigation, track it (in the frames list). Part of this was just
piggybacking off of code that handles <script>
2 - The page "load" event blocks on the frame "load" event. This cascades.
when a page triggers it's load, it can do:
```zig
if (self._parent) |p| {
p.iframeLoaded(self);
}
```
Pages need to keep track of how many iframes they're waiting to load. When
all iframes (and all scripts) are loaded, it can then triggers its own load
event.
3 - Our JS execution expects 1 primary entered context (the pages). But we now
have multiple page contexts, and we need to be in the correct one based
on where javascript is being executed. There is no more an default entered
context. Creating a Local.Scope enters the context, and ls.deinit() exits
the context.
Our BrowsingContext currently supports 1 target. So we have a per-BC target_id.
Previously, our target had 1 "frame" - our page. So we often treated the
targetId as the frameId. But to work with frames, we need page-specific
frameIds and loaderIds.
This tries to clean up our ids (a little). frameIds are now ids derived from
a new incrementing page.id. This page.id has to be passed around (via http
Requests and through notifications) in order to properly generate messages with
a frameId.
Added a get-only `getStyle` which doesn't lazily create a new style if none
exists. This can be used in the (frequently used) `checkVisibility` to avoid
an allocation. Added a specialized getBoundingClientRectForVisible which
skips the checkVisibility check, since a few callers have already done their
own visibility check.
DOMRect is now off the heap. This avoids _a lot_ of allocation when a DOMRect
is only needed for internal calculation, e.g. in Document.elementFromPoint.
Compute and include the cookie size field (name.len + value.len)
in Storage.getCookies and Network.getCookies CDP responses,
matching Chrome's behavior.
Page.reset exists for 1 use case: multiple calls to the Page.navigate CDP
method. At an extreme, something like this in puppeteer:
```
await page.goto(baseURL + '/campfire-commerce/');
await page.goto(baseURL + '/campfire-commerce/');
```
Rather than handling this generically in Page, we now handle this case
specifically at the CDP layer. If the page isn't in its initial load state,
i.e. page._load_state != .waiting, then we reload the page from the session.
For reloading, my initial inclination was to do session.removePage then
session.createPage(). This behavior still seems potentially correct to me, but
compared to our `reset`, this would trigger extra notifications, namely:
self.notification.dispatch(.page_remove, .{});
and
self.notification.dispatch(.page_created, page);
Bacause of https://github.com/lightpanda-io/browser/pull/1265/ I guess that
could have side effects. So, to keep the behavior as close to the current
"reset", a new `session.replacePage()` has been added which behaves a lot like
removePage + createPage, but without the notifications being sent.
While I generally think this is just cleaner, this was largely driven by some
planning for frame support. The entity for a Frame will share a lot with the
Page (we'll extract that logic), so simplifying the Page, especially around
initialization, helps simplify frame support.
At a high level, this does for Events what was recently done for XHR, Fetch and
Observers. Events are self-contained in their own arena from the ArenaPool and
are registered with v8 to be finalized.
But events are more complicated than those other types. For one, events have
a prototype chain. (XHR also does, but it's always the top-level object that's
created, whereas it's valid to create a base Event or something that inherits
from Event). But the _real_ complication is that Events, unlike previous types,
can be created from Zig or from V8.
This is something that Fetch had to deal with too, because the Response is only
given to V8 on success. So in Fetch, there's a period of time where Zig is
solely responsible for the Response, until it's passed to v8. But with events
it's a lot more subtle.
There are 3 possibilities:
1 - An Event is created from v8. This is the simplest, and it simply becomes a
a weak reference for us. When v8 is done with it, the finalizer is called.
2 - An Event is created in Zig (e.g. window.load) and dispatched to v8. Again
we can rely on the v8 finalizer.
3 - An event is created in Zig, but not dispatched to v8 (e.g. there are no
listeners), Zig has to release the event.
(It's worth pointing out that one thing that still keeps this relatively
straightforward is that we never hold on to Events past some pretty clear point)
Now, it would seem that #3 is the only issue we have to deal with, and maybe
we can do something like:
```
if (event_manager.hasListener("load", capture)) {
try event_manager.dispatch(event);
} else {
event.deinit();
}
```
In fact, in many cases, we could use this to optimize not even creating the
event:
```
if (event_manager.hasListener("load, capture)) {
const event = try createEvent("load", capture);
try event_manager.dispatch(event);
}
```
And that's an optimization worth considering, but it isn't good enough to
properly manage memory. Do you see the issue? There could be a listener (so we
think v8 owns it), but we might never give the value to v8. Any failure between
hasListener and actually handing the value to v8 would result in a leak.
To solve this, the bridge will now set a _v8_handover flag (if present) once it
has created the finalizer_callback entry. So dispatching code now becomes:
```
const event = try createEvent("load", capture);
defer if (!event._v8_handover) event.deinit(false);
try event_manager.dispatch(event);
```
The v8 finalizer callback was also improved. Previously, we just embedded the
pointer to the zig object. In the v8 callback, we could cast that back to T
and call deinit. But, because of possible timing issues between when (if) v8
calls the finalizer, and our own cleanup, the code would check in the context to
see if the ptr was still valid. Wait, what? We're using the ptr to get the
context to see if the ptr is valid?
We now store a pointer to the FinalizerCallback which contains the context.
So instead of something stupid like:
```
// note, if the identity_map doesn't contain the value, then value is likely
// invalid, and value.page will segfault
value.page.js.identity_map.contains(@intFromPtr(value))
```
We do:
```
if (fc.ctx.finalizer_callbacks.contains(@intFromPtr(fc.value)) {
// fc.value is safe to use
}
```
The BrowserContext currently uses 3 arenas:
1 - Command-specific, which is like the call_arena, but for the processing of a
single CDP command
2 - Notification-specific, which is similar, but for the processing of a single
internal notification event
3 - Arena, which is just the session arena and lives for the duration of the
BrowseContext/Session
This is pretty coarse and can results in significant memory accumulation if a
browser context is re-used for multiple navigations.
This commit introduces 3 changes:
1 - Rather than referencing Session.arena, the BrowerContext.arena is now its
own arena. This doesn't really change anything, but it does help keep things
a bit better separated.
2 - Introduces a page_arena (not to be confused with Page.arena). This arena
exists for the duration of a 1 page, i.e. it's cleared when the
BrowserContext receives the page_created internal notification. The
`captured_responses` now uses this arena, which means captures only exist
for the duration of the current page. This appears to be consistent with
how chrome behaves (In fact, Chrome seems even more aggressive and doesn't
appear to make any guarantees around captured responses). CDP refers to this
lifetime as a "renderer" and has an experimental message, which we don't
support, `Network.configureDurableMessages` to control this.
3 - Isolated Worlds are now more self contained with an arena from the ArenaPool.
There are currently 2 places where the BrowserContext.arena is still used:
1 - the isolated_world list
2 - the custom headers
Although this could be long lived, I believe the above is ok. We should just
really think twice whenever we want to use it for anything else.
V8's inspector world is made up of 4 components: Inspector, Client, Channel and
Session. Currently, we treat all 4 components as a single unit which is tied to
the lifetime of CDP BrowserContext - or, loosely speaking, 1 "Inspector Unit"
per page / v8::Context.
According to https://web.archive.org/web/20210622022956/https://hyperandroid.com/2020/02/12/v8-inspector-from-an-embedder-standpoint/
and conversation with Gemini, it's more typical to have 1 inspector per isolate.
The general breakdown is the Inspector is the top-level manager, the Client is
our implementation which control how the Inspector works (its function we expose
that v8 calls into). These should be tied to the Isolate. Channels and Sessions
are more closely tied to Context, where the Channel is v8->zig and the Session
us zig->v8.
This PR does a few things
1 - It creates 1 Inspector and Client per Isolate (Env.js)
2 - It creates 1 Session/Channel per BrowserContext
3 - It merges v8::Session and v8::Channel into Inspector.Session
4 - It moves the Inspector instance directly into the Env
5 - BrowserContext interacts with the Inspector.Session, not the Inspector
4 is arguably unnecessary with respect to the main goal of this commit, but
the end-goal is to tighten the integration. Specifically, rather than CDP having
to inform the inspector that a context was created/destroyed, the Env which
manages Contexts directly (https://github.com/lightpanda-io/browser/pull/1432)
and which now has direct access to the Inspector, is now equipped to keep this
in sync.
The ExecutionWorld doesn't do anything meaningful. It doesn't map to, or
abstract any, v8 concepts. It creates a js.Context, destroys the context and
points to the context. Those all all things the Env can do (and it isn't like
the Env is over-burdened as-is).
Plus the benefit of going through the Env is that we can track/collect all
known Contexts for an isolate in 1 place (the Env), which can facilitate things
like context creation/deletion notifications.
We're seeing an assertion in Page.appendNew fail because the node has a parent.
According to html5ever, this shouldn't be possible (appendNew is only called
from the Parser). BUT, it's possible we're mutating the node in a way that
we shouldn't...maybe there's JavaScript executing as we're parsing which is
mutating the node.
In release, this will be more defensive. In debug, this still crashes. It's
possible this is valid (like I said, maybe there's JS interleaved which is
mutating the node), but if so, I'd like to know the exact scenario that produces
this case.
Avoids having to allocate small strings when going from v8 -> Zig. Also
added a discriminatory type, string.Global which uses the arena, rather than
the call_arena, if an allocation _is_ necessary. (This is similar to a feature
we had before, but was lost in zigdom). Strings from v8 that need to be
persisted, can be allocated directly v8 -> arena, rather than v8 -> call_arena
-> arena.
I think there are a lot of places where we should use string.String - where
strings are expected to be short (e.g. attribute names). But started with just
document.querySelector and querySelectorAll.
