I don't know if FrameId is related to an <iframe>, and whether each Page has
1 implicit "frame". But, playwright seems to treat frameId and targetId as
interchangeable, and chrome seems to agree (at leas to some degree); chrome will
return a targetId and reuse that value for the frameId.
So the simplest solution is just to remove our concept of a frameId and use
targetId exclusively. This doesn't seem to cause any issues with puppeteer.
The getTargetInfo result must return a `targetInfo` key.
Here is an example returned by Chrome:
```json
{
"id": 16,
"result": {
"targetInfo": {
"targetId": "d93a1bbc-f906-4bbb-bb4d-a2285234b091",
"type": "browser",
"title": "",
"url": "",
"attached": true,
"canAccessOpener": false
}
}
}
```
Rely on inspector to send the result, otherwise we'll send 2 responses to the
same message (one ourselves and one from the inspector), which Playwright does
not like.
The TL;DR is that this commit enforces the use of correct IDs, introduces a
BrowserContext, and adds some CDP tests.
These are the ids we need to be aware of when talking about CDP:
- id
- browserContextId
- targetId
- sessionId
- loaderId
- frameId
The `id` is the only one that _should_ originate from the driver. It's attached
to most messages and it's how we maintain a request -> response flow: when
the server responds to a specific message, it echo's back the id from the
requested message. (As opposed to out-of-band events sent from the server which
won't have an `id`). When I say "id" from this point forward, I mean every id
except for this req->res id.
Every other id is created by the browser.
Prior to this commit, we didn't really check incoming ids from the driver. If
the driver said "attachToTarget" and included a targetId, we just assumed that
this was the current targetId. This was aided by the fact that we only used
hard-coded IDS. If _we_ only "create" a frameId of "FRAME-1", then it's tempting
to think the driver will only ever send a frameId of "FRAME-1".
The issue with this approach is that _if_ the browser and driver fall out of sync
and there's only ever 1 browserContextId, 1 sessionId and 1 frameId, it's not
impossible to imagine cases where we behave on the thing.
Imagine this flow:
- Driver asks for a new BrowserContext
- Browser says OK, your browserContextId is 1
- Driver, for whatever reason, says close browserContextId 2
- Browser says, OK, but it doesn't check the id and just closes the only
BrowserContext it knows about (which is 1)
By both re-using the same hard-coded ids, and not verifying that the ids sent
from the client correspond to the correct ids, any issues are going to be hard
to debug.
Currently LOADER_ID and FRAEM_ID are still hard-coded. Baby steps.
Previously, we could have multiple in-flight messages from the server to a
single client. This isn't safe and can lead to message interleaving. While
write / send are atomic, they are only atomic for the N bytes which they write,
which may not be the entire buffer. Consider this writeAll function:
```
pub fn writeAll(socket: socket_t, bytes: []const u8) !void {
var index: usize = 0;
while (index < bytes.len) {
index += try posix.write(socket, bytes[index..]);
}
}
```
If we're trying to send "abc123", this could take anywhere from 1 to 6 calls
to posix.write (it would take 6 calls, for example, if every call to
posix.write only wrote a single byte). Now if you're trying to write other data
to this same socket at the same time, messages _will_ get interleaved.
In order for this to work, the client now has a send_queue (doubly linked list).
When one message is sent, it sends the next.
In addition to the above change, the Client is now self-contained with respect
to its lifetime. This is necessary so that completions which come in AFTER our
concept of its lifetime ends, can still be processed. I think all types that
receive completions need to follow this model. This relies on the fact that
kqueue (which I know for a fact) and io_uring (which people seem to imply) handle
socket shutdown properly. It's still a bit messy because of timeout and not
wanting to wait until timeout to accept new connections, but needing to wait
until timeout to cleanup the client.
The self-contained nature of Client makes it difficult to test as a generic. I
removed Client(T). Tests now use real sockets. Some tests had to be removed
because they're too difficult to test over a real connection :(
./lightpanda serve --host ...
./lightpanda fetch https://...
Makes it easier to communicate / document which command has which options.
Internally added a "usage" command for displaying the usage - removing the need
for error.NoError :|
Create UUID v4.
Create prefixed ids. To support more of the CDP protocol, we need to remove the
hard-coded IDs (session, browser context, frame, loader, ...) and be able to
dynamically create them, i.e. creating a new BrowserContextId when
Target.createBrowserContext is called.
var frame_id = id.Incremental(u16, "FRM"){};
frame_id.next() == "FRM-1"
frame_id.next() == "FRM-2"
Generation is allocation-free (the returned string is only valid until the
next call to next()). This is not thread safe, each CDP instance will have its
own generator (for each id it needs to generate).
The generated IDs are different than what Chrome uses, i.e.
BROWSERSESSIONID597D9875C664CAC0. I looked at various drivers and none have
any expectations beyond a string. Shorter IDs will be more efficient. Also, the
ID can cheeply be converted to and from an integer, allowing for lookups via
AutoHashMap(u16) instead of StringHashMap.
In debug mode, it has a more user-friendly output:
level | the log messge | ms since last message | key=value key=value
In release mode, it logs using logfmt, which is supported by most log
ingestion frameworks.
Not being used anywhere right now, keeping this PR small with no impact on
existing code.
ADD CDP testing helpers (mock Browser, Session, Page and Client). These are
placeholders until tests are added which use them.
Added a couple CDP tests.
For the time being, given that we only allow 1 client at a time, I took a
shortcut to implement this. The server has an incrementing "current_client_id"
which is part of every completion. On completion callback, we just check if
its client_id is still equal to the server's current_client_id.
Move more logic into the reader. Avoid copying partial messages in
cases where we know that the buffer is large enough.
This is mostly groundwork for trying to add support for continuation
frames.
Adding HTTP & websocket awareness to the TCP server.
HTTP server handles `GET /json/version` and websocket upgrade requests.
Conceptually, websocket handling is the same code as before, but receiving
data will parse the websocket frames and writing data will wrap it in
a websocket frame.
The previous `Ctx` was split into a `Server` and a `Client`. This was
largely done to make it easy to write unit tests, since the `Client` is
a generic, all its dependencies (i.e. the server) can be mocked out. This
also makes it a bit nicer to know if there is or isn't a client (via the
server's client optional).
Added a MemoryPool for the Send object (I thought that was a nice touch!)
Removed MacOS hack on accept/conn completion usage.
Known issues:
- When framing an outgoing message, the entire message has to be duped. This
is no worse than how it was before, but it should be possible to eliminate
this in the future. Probably not part of this PR.
- Websocket parsing will reject continuation frames. I don't know of a single
client that will send a fragmented message (websocket has its own
message fragmentation), but we should probably still support this just in
case.
- I don't think the receive, timeout and close completions can safely be
re-used like we're doing. I believe they need to be associated with a specific
client socket.
- A new connection creates a new browser session. I think this is right (??),
but for the very first, we're throwing out a perfectly usable session. I'm
thinking this might be a change to how Browser/Sessions work.
- zig build test won't compile. This branch reproduces the issue with none
of these changes:
https://github.com/karlseguin/browser/tree/broken_test_build
(or, as a diff to main):
https://github.com/lightpanda-io/browser/compare/main...karlseguin:broken_test_build
