MCP server

Okinawa ships an optional in-engine MCP (Model Context Protocol) server. It lets an external agent connect to a running app over local HTTP and observe or drive it through the tools listed below. The public surface (OkMcpServer) deliberately exposes no HTTP or JSON types: all of that lives behind a pimpl, so consumers do not inherit those dependencies.

Tools

The server exposes these tools to a connected agent:

Key names accepted by press_key/press_keys: single letters and digits, space, up/down/left/right, escape, enter, tab, backspace, grave (or backtick), period, minus. grave toggles the engine console; while the console is open, injected printable keys feed its input line, so a command can be typed a key at a time (grave, then the letters with space/period, then enter).

Do not drive the console that way. Each key is one MCP call with its own round trip, so a fifteen-character line costs sixteen of them and takes about as many seconds — long enough that an agent stops reaching for the commands it most needs while debugging. Use console instead: one call, one line, and the command's own output comes back as the result.

Driving the pointer

press_key reaches anything bound to the keyboard, which in a game is most of it. An application with an interface of its own -- panels, menus, things dragged into place -- is a different matter: none of that is reachable from a keyboard, so without a pointer an agent working on one has to ask a person to make the gesture and send a picture back.

Say how far, and in what. "Move up" is not an instruction: up by how much, measured in what? Every number this tool takes is in the pixels get_state reports under window, origin at the top left -- the frame a screenshot is made of, so a position read off an image can be handed straight back.

That frame is the framebuffer, and on a screen with more pixels than points an interface is laid out in half as many: the tool converts, so a caller never has to know what display it is drawing on. What a caller does have to allow for is a picture that arrived scaled down on the way -- multiply by window.width over the image's own width first.

{"action": "move",  "x": 700, "y": 400}
{"action": "move",  "dx": 0, "dy": -120}
{"action": "click", "x": 30, "y": 9, "button": "left"}
{"action": "drag",  "x": 1100, "y": 500, "to_x": 1250, "to_y": 420, "steps": 10}
{"action": "wheel", "notches": 3}

Two things are deliberate and neither is free to change:

• A click holds its button for about four frames. An interface that never sees the button held does not see a click at all.
• A drag is delivered in steps, a frame apart. Half of what an interface does with a drag it does while the drag is happening; one jump from start to end is a gesture whose middle nothing sees.

What the application has to do

The engine keeps the injected pointer and says it has been used; applying it is the application's business, exactly like the injected keys. An application that draws its own interface with a UI library has to hand the pointer to that library itself -- the library's backend reads the window system directly, and an injected pointer is not the window system. With Dear ImGui, between the backend's new frame and your own:

OkInput *input = OkCore::getInput();
if (input != nullptr && input->injectedPointerUsed()) {
  ImGuiIO &io = ImGui::GetIO();
  double px = 0.0;
  double py = 0.0;
  input->injectedPointer(&px, &py);
  io.AddMousePosEvent(static_cast<float>(px), static_cast<float>(py));
  for (int b = 0; b < 3; b++) {
    io.AddMouseButtonEvent(b, input->injectedButton(b));
  }
  double wheel = input->takeInjectedWheel();   // read once: clears it
  if (wheel != 0.0) {
    io.AddMouseWheelEvent(0.0f, static_cast<float>(wheel));
  }
}

takeInjectedWheel clears what it returns, so read it once a frame: a notch delivered twice is one turn of the wheel doing two steps.

And ask injectedPointerUsed() every frame rather than remembering the answer. The injected pointer goes quiet two seconds after the last call, and the instant the physical mouse moves: a hand on the desk beats anything driving from outside. Fed unconditionally once it had been used, it pushes a stale position every frame and the person at the window finds their own mouse does nothing.

Taking the keyboard, and giving it back

An agent measuring something needs the view to hold still, and a person at the same window will move it without meaning to. The launch flag --no-input-style gate (OkCore::setIgnoreUserInput) has always been able to stop that, but only at start-up: changing your mind meant relaunching, which throws away the viewpoint that was being measured.

input is the same gate at runtime. {"enabled": false} takes the keyboard for the agent, {"enabled": true} hands it back, and both report the resulting state.

The part worth understanding is what happens when the agent does not hand it back — because that is the failure this is built around, not an edge case. A block is invisible by nature: the keyboard simply stops answering, which from a chair looks exactly like a hang. So:

• it expires. seconds (default 300) is a deadline, not a hint. An agent that crashes, loses its connection or simply forgets costs the person a wait, never a lockout;
• the window always wins. ctrl + shift + k lifts any block, matched on the device itself rather than on what the application is allowed to see. Nothing running in the background can override that, and the key of the combo is then taken out of the frame until it is let go, so the application never sees it -- a release that hands control back by quitting the app, or by walking the avatar off, is not a release. (Not escape: macOS does not deliver Escape to an application while Control is held, measured on the device with the modifiers arriving and the Escape never doing so.);
• it is on screen. A line in the corner says the input is held, how many seconds are left, and which keys end it.

Both also report device: keys_down, what the keyboard is holding right now, and recent_presses, the last few keys it reported going down with the modifiers held at the time and how long ago. An agent cannot press a key, so when a gesture does not work this is the only way to tell a key that never arrived from a match that is wrong. It earned its place: three rounds of guessing at a combo that would not fire, then one reading -- macOS does not deliver Escape while Control is held.

{"seconds": 0} blocks with no deadline. It exists because the launch flag needs it; an agent asking for it is asking to be the thing the three rules above were written about.

Measuring performance

get_state reports an instantaneous fps. That number is close to useless on its own: it describes one frame, from one viewpoint, and two such readings cannot tell a real change from a lucky moment. Use get_performance, which returns the whole recorded window.

Two things worth knowing when reading the result:

• Compare like with like. The same build reads completely differently close to the ground and from above. State the viewpoint alongside the number, and when comparing two builds, measure both from the same place — ideally in the same session.

• Watch for quantisation. With vsync on, frame times land on multiples of the refresh interval: on a 60 Hz display a median of 16.7, 33.3 or 50 ms means one, two or three refreshes per frame. A median pinned to exactly 33.3 ms says the frame costs somewhere between 16.7 and 33.3 ms, not that it costs 33.3, and a small extra cost will drop it a whole step to 50 ms. Read min alongside the median to see where the real work sits.

  This also explains a frame rate that seems to halve for no reason: near the boundary, a couple of milliseconds either way moves the whole frame between one refresh and two. render.vsync 0 asks for vsync off so the times show the real work instead — but a compositor can enforce it anyway (macOS does), in which case the readings stay quantised and only samples_ms tells the full story.

When a call fails

A tool that cannot do what was asked answers with an ordinary result carrying isError: true and a line of text saying why — an unknown camera name, a key that does not exist, a request that arrived while nothing was rendering. The HTTP status stays 200: the call was delivered and answered, and what failed is the tool, not the transport.

The same applies to a malformed argument. Sending a string where a number belongs is reported as bad arguments: ... rather than being allowed to reach the JSON library and abort the request, so a client that gets a field wrong is told which one instead of receiving an empty response.

An argument the tool does not have is refused outright, naming it and listing the ones the tool does take:

console: no such argument: command. It takes: line

This matters more than it looks. Every schema says additionalProperties: false, and an unknown argument used to be dropped, leaving the tool to run on whatever was left — which for a tool whose arguments are all optional is a call with no arguments at all. That is a different call, it succeeds, and it answers tidily, so a caller with one field misspelled can drive an application for a long time believing it is being obeyed.

Enabling it

From application code, enable the server after OkCore::initialize():

OkCore::enableMcpServer();        // binds 127.0.0.1:8765
OkCore::setIgnoreUserInput(true); // optional: MCP-only control

enableMcpServer(int port = 8765) always exists. If the engine was built without MCP support it logs a warning and does nothing, so apps compile identically with or without the server. The server binds 127.0.0.1 on the given port and logs a line such as MCP server listening on http://127.0.0.1:8765/mcp.

Connecting a client

Once a build with the server enabled is running (it logs MCP server listening on http://127.0.0.1:8765/mcp), point your MCP client at that URL. The transport is streamable HTTP.

With the Claude Code CLI, add it on the command line:

claude mcp add --transport http okinawa http://127.0.0.1:8765/mcp

This registers the server (named okinawa here) at local scope. Reload or reconnect the client so it picks up the tools, then the tools listed above are available. Remove it later with claude mcp remove okinawa. The server only exists while the app runs, so start the app and reconnect each session.

Compile-time toggle

The server is compiled in by default and guarded by OKINAWA_WITH_MCP (xmake option mcp). Exclude it from lean builds with xmake f --mcp=n, or be explicit with --mcp=y. When excluded, no MCP/HTTP code or its header-only dependencies land in the binary.

OkMcpServer methods

OkMcpServer is normally managed by OkCore; you rarely construct it directly. Its public surface:

drainCommands() must be called once per frame from the engine loop thread (the one holding the OpenGL context), after the scene is rendered and before the buffers are swapped. OkCore::loop does this for you when the server is enabled.