Chapter 22 — Input Devices in the Browser

The Cloud Portal Remote Desktop tab (Ch. 21) is more than a keyboard and mouse mirror. It works on phones, tablets, gaming controllers, and keyboards from any country. This chapter is the field guide for every input device the browser RD supports.

What this chapter covers

• Three touch modes — Trackpad, Direct Touch, Scroll.
• The full gesture vocabulary the touch controller understands.
• The on-screen mobile shortcut bar (modifier keys + arrows + F-keys).
• International keyboard layouts (en-US / en-GB / Deutsch QWERTZ / Français AZERTY / Español).
• Presence and spectate — seeing other people on the same device.
• "I do not have a touch screen / gamepad / international keyboard yet" — how to practise without the hardware.

Touch modes

Open the Remote Desktop tab on a phone or tablet and a small pill appears at the bottom of the screen with three icons:

The mode switcher gives a short haptic buzz (navigator.vibrate) when you switch modes — so you know it took even if you cannot see the bottom of the screen.

Trackpad mode and the virtual cursor

When Trackpad is active a crosshair overlay (the AidaVirtualCursor) shows where your next tap will land. The crosshair is a real DOM element — it does not consume server input — so it stays smooth even on a slow tunnel. Sensitivity is 1.5× by default: a one-inch finger sweep moves the cursor about 1.5 inches on the remote screen. Pinch-zoom does not change the sensitivity, so once you have zoomed in for precision the cursor stays predictable.

Gesture vocabulary

The TouchController class in web/static/js/touch_controls.js recognises eight gestures:

The top-edge swipe is handy on phones where the toolbar would otherwise eat 10% of your screen. Swipe down from the top to summon it, swipe up to dismiss.

Pinch is client-side only. Pinching does not zoom anything on the host machine — it zooms your browser-side bitmap. The host has no idea you have done it. This keeps the round-trip latency-free and means the host never sees a flurry of pinch events.

The mobile shortcut bar

Three-finger tap brings up the shortcut bar (built by mobile_keyboard.js). Two rows of keys you cannot type on a touchscreen keyboard alone:

• Row 1 (essentials): Esc, Tab, Ctrl, Alt, Shift, ←, →, ↑, ↓.
• Row 2 (function row): F1–F12, plus a numpad chip for 0–9.

Modifier keys (Ctrl / Alt / Shift / Win) latch when tapped — tap once to arm, then tap a letter to send the chord, then the modifier auto-releases. The latched state is visible (the chip glows). Tap a modifier again to manually release it.

This bar is also the only way to send the Windows key from an iPad — there is no native physical shortcut for it.

International keyboard layouts

web/static/js/hid_keyboard_layouts.js ships five layouts:

Pick a layout from the keyboard chooser in the RD toolbar. The vendor-neutral passthrough layout (vendor: 'passthrough') means physical key positions are translated to the right scan codes for the chosen logical layout, so an AZERTY user on a QWERTY hardware keyboard still types correctly.

Switching layouts only affects what your browser sends — the host's own keyboard layout setting still applies as the final translation. If you have a layout mismatch where one side is set wrong, fix the side you control.

Presence and spectate

If two people are looking at the same device at the same time, both their cursors appear on the host's screen (small coloured circles) and each browser tab shows a watcher pill in the corner with the other person's name + status dot.

• Watcher pill — listed names of everyone currently looking at this device. Click a name to spectate.
• Spectate mode — your view follows another watcher's scroll position and chapter / tab. You can read along but cannot click; spectating ends as soon as you scroll on your own.
• Heartbeat — /api/presence/heartbeat fires every 12 seconds; you drop off the watcher list 25 seconds after the last heartbeat. Phones going to sleep are detected and broadcast a clean leave.

This is enabled for any session with at least Collaborator permission. Read-Only viewers appear in the watcher list but their cursor does not show on the host (privacy default).

Edge cases worth knowing

• iPad Pencil is treated as a single-finger touch with hover support. Long-press still works.
• Android stylus is the same as a finger touch — no hover events on most stock pens.
• Wacom tablet over USB to a tablet computer works because the browser exposes it as a pointer; pressure is not forwarded to the host (it is treated as a left-click).
• Two simultaneous cursors (someone on a Wacom + your finger) — the touch controller defers to the most-recent pointer.

Terms & Setup

Touch mode

Trackpad / Direct Touch / Scroll. The pill at the bottom of the RD tab. Default is Trackpad.

Virtual cursor (AidaVirtualCursor)

The crosshair overlay shown in Trackpad mode. DOM-rendered; not sent to the host.

Shortcut bar

The two-row keypad summoned by three-finger tap. Built by mobile_keyboard.js.

Layout (keyboard)

The logical map from physical key positions to characters. Five ship in-the-box; default en-US.

QWERTZ

The German keyboard layout — Y and Z swapped compared to QWERTY. Selected with the de layout ID.

AZERTY

The French keyboard layout — A/Q and Z/W swapped, digits require Shift. Selected with the fr layout ID.

Spectate

Read-only follow-mode: your viewport scrolls with another watcher's. Cancels the moment you scroll independently.

Watcher pill

Header-corner widget listing who else is currently looking at this device. Each name is a clickable spectate button.

Heartbeat (presence)

The 12-second poll to /api/presence/heartbeat that keeps you on the watcher list. 25-second timeout for stale entries.

"I do not have a touch screen yet" — practise without one

Chrome and Edge ship a touch emulator inside DevTools:

1. Open the Cloud Portal Remote Desktop tab.
2. F12 to open DevTools.
3. Click the Toggle device toolbar icon (or Ctrl+Shift+M).
4. Pick Responsive / iPad / Pixel 7 etc. from the device dropdown.
5. The browser now emits touch events instead of mouse events. The touch mode pill appears.

This emulator does not simulate pinch-zoom — for that you need real fingers or two mice.

"I do not have a gamepad yet" — Gamepad API

If a gamepad ships in a later release, the same testing path applies: any USB or Bluetooth controller appears as a Gamepad in the browser's navigator.getGamepads() array. Plug a controller in, press any button, and the browser surfaces it. Test pages like <https://gamepad-tester.com/> confirm the OS sees it.

"I do not have a foreign-language keyboard yet" — practise without one

Pick the layout from the chooser; your physical keys do not need to match. With the de (QWERTZ) layout selected on a US keyboard:

• Press the physical Y key — the host sees Z.
• Press the physical Z key — the host sees Y.
• Press Right-Alt + Q — the host sees @.

This is exactly how a French user with an AZERTY keyboard physically types a character that lives in a different position than where it is printed on the keycap.

"I do not see other watchers" — checks

If presence is silent when you expect a teammate to be online:

1. The server must be at AidaIDE 0.5.66+ — earlier builds had no presence backend.
2. Your role must be Collaborator or higher to broadcast presence; Read-Only is silent by default.
3. Network: SSE (/api/presence/stream) must not be blocked by an aggressive proxy. Some hotel Wi-Fi captures long-lived connections.
4. Open the browser console and look for presence.js log lines — heartbeat failures print there.

See also

• Chapter 21 — Cloud Portal and Web Access for how the Remote Desktop tab is opened in the first place.
• Chapter 9 — AI Hierarchy Builder for keyboard shortcuts on the desktop builder (separate from RD's shortcut bar).
• Chapter 13 — Settings §SSH for the host-side terminal type (xterm-256color) that affects how arrow keys are reported.
• Chapter 18 — Troubleshooting & FAQ for RD PIN issues and tunnel-down recovery.