Async HTTP request work by emitting a "Progress" object to a callback. This
object has a "done" flag which, when `true`, indicates that all data has been
emitting and no future "Progress" objects will be sent.
Callers like XHR buffer the response and wait for "done = true" to then process
the request.
The HTTP client relies on two important object pools: the connection and the
state (with all the buffers for reading/writing).
In its current implementation, the async flow does not release these pooled
objects until the final callback has returned. At best, this is inefficient:
we're keeping the connection and state objects checked out for longer than they
have to be. At worse, it can lead to a deadlock. If the calling code issues a
new request when done == true, we'll eventually run out of state objects in the
pool.
This commit now releases the state objects before emit the final "done" Progress
message. For this to work, this final message will always have null data and
an empty header object.
1 - Make log_level a runtime option (not a build-time)
2 - Make log_format a runtime option
3 - In Debug mode, allow for log scope filtering
Improve the general usability of scopes. Previously, the scope was more or less
based on the file that the log was in. Now they are more logically grouped.
Consider the case where you want to silence HTTP request information, previously
you'd have to filter out the `page`, `xhr` and `http_client` scopes, but that
would also elimiate other page, xhr and http_client logs. Now, you can just
filter out the `http` scope.
The Request object now exists on the heap, allowing it to outlive whatever is
making the request (e.g. the XHR object). We can now wait until all inflight IO
events are completed before clearing the memory.
This change fixes the crash observed in:
https://github.com/lightpanda-io/browser/issues/667
Outputs in logfmt in release and a "pretty" print in debug mode. The format
along with the log level will become arguments to the binary at some point in
the future.
This might not be specific to network notification, but the issue happens all
the time testing scenarios that rely on network notification, so it's hard
to ignore.
- Add 2 internal notifications
1 - http_request_start
2 - http_request_complete
- When Network.enable CDP message is received, browser context registers for
these 2 events (when Network.disable is called, it unregisters)
- On http_request_start, CDP will emit a Network.requestWillBeSent message.
This _does not_ include all the fields, but what we have appears to be enough
for puppeteer.waitForNetworkIdle.
- On http_request_complete, CDP will emit a Network.responseReceived message.
This _does not_ include all the fields, bu what we have appears to be enough
for puppeteer.waitForNetworkIdle.
We currently don't emit any other new events, including any network-specific
lifecycleEvent (i.e. Chrome will emit an networkIdle and networkAlmostIdle).
To support this, the following other things were done:
- CDP now has a `notification_arena` which is re-used between browser contexts.
Normally, CDP code runs based on a "cmd" which has its own message_arena, but
these notifications happen out-of-band, so we needed a new arena which is
valid for handling 1 notification.
- HTTP Client is notification-aware. The SessionState no longer includes the
*http.Client directly. It instead includes an http.RequestFactory which is
the combination fo the client + a specific configuration (i.e. *Notification).
This ensures that all requests made from that factory have the same settings.
- However, despite the above, _some_ requests do not appear to emit CDP events,
such as loading a <script src="X">. So the page still deals directly with the
*http.Client.
- Playwright and Puppeteer (but Playwright in particular) are very sensitive to
event ordering. These new events have introduced additional sensitivity.
The result sent to Page.navigate had to be moved to inside the navigate event
handler, which meant passing some cdp-specific data (the input.id) into the
NavigateOpts. This is the only way I found to keep both happy - the sequence
of events is closer (but still pretty far) from what Chrome does.
Synchronous body reader now exposes a peek() function to get the first few bytes
from the response body. This will be no less than 100 bytes (assuming the body
is that big), but could be more. Streaming API, via res.next() continues to work
as-is even if peek() is called.
Introduce Mime.sniff() that detects a few common types - the ones that we care
about right now - from the body content.
The HTTP response values _are_ mutable, but because we're using std.http.Header
the type is a `[]const u8`. This introduce a custom `Header` type where the
value is `[]u8`.
The goal is largely to allow more efficient value-comparison, by allowing
calling code to lower-case in-place. I specifically have the Mime parser in
mind:
25dcae7648/src/browser/mime.zig (L134)
Combine uri + rawuri into single struct.
Try to improve ownership around URIs and URI-like things.
- cookie & request can take *const std.Uri
(TODO: make them aware of the new URL struct?)
- Location (web api) should own its URL (web api URL)
- Window should own its Location
Most of these changes result in (a) a cleaner Page and (b) not having to carry
around 2 nullable objects (URI and rawuri).
When we only have 1 or 2 bytes missing from a chunk (i.e. the tailing \n or
\r\n), don't emit an empty chunk if we have more data available to process.
When set, this disables the host verification of all HTTP requests. Available
for both the fetch and serve mode.
Also introduced an App.Config, for future command line options which need to
be passed more deeply into the code.
