There is some risk to this change. The first is that I made a mistake. The
other is that one of the APIs that doesn't currently return an error changes
in the future.
CDP currently assumes that if we get a page-related notification (like a
request interception, or page lifecycle event), then we must have a session
and page.
But, Target.detachFromTarget can remove the session from the BrowserContext
while still having the page run (I wonder if we should stop the page at this
point??). So, remove these assumptions and make sure we have a page/session
in the handling of page events.
Most CDP drivers have a mechanism to wait for idle network, or an almost idle
network (sometimes called networkIdle2). These are events the browser must emit.
The page will now emit `networkIdle` when we are reasonably sure there's no more
network activity (this requires some slight changes to request interception,
since, I believe, intercepted requests should be considered).
`networkAlmostIdle` is currently _always_ emitted prior to emitting
`networkIdle`. We should tweak this but I can't, at a glance, think of a great
heuristic for when this should be emitted.
Further reducing bouncing between page and server for loop polling. If there is
a page, the page polls. If there isn't a page, the server polls. Simpler.
chromedb doesn't support duplicate header names. Although servers _will_ send
this (e.g. Cache-Control: public\r\nCache-Control: max-age=60\r\n), Chrome
seems to join them with a "\n". So we do the same.
A note on curl_easy_nextheader, which this code ultimately uses to iterate
and collect the headers. The documentation says:
Applications must copy the data if they want it to survive subsequent API
calls or the life-time of the easy handle.
As-is, I'd understand this to mean that a given header name/value is only
valid until any API call, including another call to curl_easy_nextheader. So,
from this comment, we _should_ be duping the name/value. But we don't. Why?
Because, despite the note in the documentation, this doesn't appear to be how
it actually works, nor does it really make sense. If it's just a linked list,
there's no reason curl_easy_nextheader should invalidate previous results. I'm
guessing this is just a general lack of guarantee libcurl is willing to make re
lifetimes.
https://github.com/lightpanda-io/browser/issues/966
Add response_data event, CDP now captures the full body so that it can respond
to the Network.getResponseBody. This isn't memory efficient, but I don't see
another way to do it. At least this way, it's only capturing/storing every
response body when (a) CDP is used and (b) Network.enabled is called. That is,
as opposed to baking this into Http/Client.zig, which would force the memory
consumption for all use-cases.
There's arguably some optimizations we could make for XHR requests, which also
dupe/own the response. As of now, the response is dupe'd separately for CDP
and XHR.
With networking enabled, CDP listens to this event and emits a
`Network.loadingFinished` event. This is event is used by puppeteer to know that
details about the response (i.e. the body) can be queries.
Added dummy handling for the Network.getResponseBody message. Returns an
empty body. Needed because we emit the loadingFinished event which signals
to drivers that they can ask for the body.
Stream (to json) the Transfer as a request and response object in the various
network interception-related events (e.g. Network.responseReceived).
Add a page.request_intercepted boolean flag for CDP to signal the page that
requests have been intercepted, allowing Page.wait to prioritize intercept
handling (or, at least, not block it).
On client.request(req) we now immediately wrap the request into a Transfer. This
results in less copying of the Request object. It also makes the transfer.uri
available, so CDP no longer needs to std.Uri(request.url) anymore.
The main advantage is that it's easier to manage resources. There was a use-
after free before due to the sensitive nature of the tranfer's lifetime. There
were also corner cases where some resources might not be freed. This is
hopefully fixed with the lifetime of Transfer being extended.
