Test speed has been improved only slightly by tweaking a 2-second running tests.
Build has been improved by:
1 - moving logFunctionCallError out of js.Caller and to a standalone function
2 - removing some non-generic code from the generic portions of the logger
Caller.getter and Caller.setter have been removed in favor or calling
Caller.method. This wasn't previously possible - prior to our v8 upgrade, they
had different signatures.
Also removed a largely unused parser/str.zig file.
This is one of the ways that puppeteer knows that navigation happened
and is needed to support `waitForNavigation` which compares the
existing loader-id with the new one, so it has to change.
Also, fix a crash that could happen if CDP disconnects while
connections are being aborted.
The mix of sync and async HTTP requests requires care to avoid deadlocks.
Previously, it was possible for async requests to use up all available HTTP
state objects duration a navigation flow (either directly, or via an internal
redirect (e.g. click, submit, ...)). This would block the navigation, which,
because everything is single thread, would block the I/O loop, resulting in a
deadlock.
The correct solution seems to be to remove all synchronous I/O. And I tried to
do that, but I ran into a wall with module-loading, which is initiated from V8.
V8 says "give me the source for this module", and I don't see a great way to
tell it: wait a bit.
So I went back to trying to make this work with the hybrid model, despite last
weeks failures to get it to work. I changed two things:
1 - The http client will only directly initiate an async request if there's
at least 2 free state objects available (1 for the request, and leaving 1
free for any synchronous requests)
2 - Delayed navigation retries until there's at least 1 free http state object
available.
Commits from last week did help with this. First, we're now guaranteed to have
a single sync-request at a time (previously, we could have had 2). Secondly,
the async connection is now async end-to-end (previously, it could have blocked
on an empty state pool).
We could probably make this a bit more obviously by reserving 1 state object
for synchronous requests. But, since the long term solution is probably having
no synchronous requests, I'm happy with anything that lets me move past this
issue.
Support CDP's Input.dispatchKeyEvent and DOM key events. Currently only
keydown is supported and expects every key to be a displayable character.
It turns out that manipulating the DOM via key events isn't great because the
behavior really depends on the cursor. So, to do this more accurately, we'd
have to introduce some concept of a cursor.
Personally, I don't think we'll run into many pages that are purposefully
using keyboard events. But driver (puppeteer/playwright) scripts might be
another issue.
CDP translate this into a Network.loadingFailed. This is necessary to make sure
every Network.requestWillBeSent is paired with either a Network.loadingFailed
or a Network.responseReceived.
Outputs in logfmt in release and a "pretty" print in debug mode. The format
along with the log level will become arguments to the binary at some point in
the future.
- Add 2 internal notifications
1 - http_request_start
2 - http_request_complete
- When Network.enable CDP message is received, browser context registers for
these 2 events (when Network.disable is called, it unregisters)
- On http_request_start, CDP will emit a Network.requestWillBeSent message.
This _does not_ include all the fields, but what we have appears to be enough
for puppeteer.waitForNetworkIdle.
- On http_request_complete, CDP will emit a Network.responseReceived message.
This _does not_ include all the fields, bu what we have appears to be enough
for puppeteer.waitForNetworkIdle.
We currently don't emit any other new events, including any network-specific
lifecycleEvent (i.e. Chrome will emit an networkIdle and networkAlmostIdle).
To support this, the following other things were done:
- CDP now has a `notification_arena` which is re-used between browser contexts.
Normally, CDP code runs based on a "cmd" which has its own message_arena, but
these notifications happen out-of-band, so we needed a new arena which is
valid for handling 1 notification.
- HTTP Client is notification-aware. The SessionState no longer includes the
*http.Client directly. It instead includes an http.RequestFactory which is
the combination fo the client + a specific configuration (i.e. *Notification).
This ensures that all requests made from that factory have the same settings.
- However, despite the above, _some_ requests do not appear to emit CDP events,
such as loading a <script src="X">. So the page still deals directly with the
*http.Client.
- Playwright and Puppeteer (but Playwright in particular) are very sensitive to
event ordering. These new events have introduced additional sensitivity.
The result sent to Page.navigate had to be moved to inside the navigate event
handler, which meant passing some cdp-specific data (the input.id) into the
NavigateOpts. This is the only way I found to keep both happy - the sequence
of events is closer (but still pretty far) from what Chrome does.
Some data has to exist specifically for the navigation of one page to another.
For example, if a hyperlink is clicked, the URL begins its life with the
original page, but is transferred to the new page. The page_arena cannot be used
for such data.
It's possible to use the session_arena, but it's lifetime is much longer and,
given enough navigation, could accumulate a lot of memory.
The new transfer_arena exists within the session, but only exists until the
next navigation.
While currently only used for the navigation URL, the main goal here is to have
a place to put the request body on form submission, which has a lifetime similar
to a click url.
While I'm at it, I promoted the existing session arena and the new transfer
arena to the browser, allowing better memory re-use between sessions.
