The shape of the boundary
A running program lives in user space. The kernel lives elsewhere, protected by hardware: a CPU privilege ring that physically prevents user code from touching device drivers, memory mappings, or the scheduler directly. Every time a program needs something that crosses that line — opening a file, allocating memory, sending a packet — it issues a system call. The CPU traps into kernel mode, the kernel does the work, and control returns. The program never sees the inside; it only sees the result.
This boundary has a clean, well-defined shape. On x86-64 Linux the calling convention is fixed: arguments go into specific registers, the syscall number goes into rax, and the syscall instruction fires the trap. On Windows the shape is similar but the numbers differ, the conventions differ, and the kernel objects on the other side differ considerably. A program compiled for one system issues calls that mean nothing — or mean the wrong thing — on the other.
That mismatch is precisely what a compatibility layer exploits. It does not pretend to be different hardware; it sits at the boundary and rewrites what crosses it. A Windows program asks for a file handle via NtCreateFile. The layer catches that call, translates the semantics — path format, sharing flags, disposition — and issues the equivalent POSIX open() call on the real kernel beneath. The program receives a handle. From its perspective, Windows answered.
The mechanism in brief
FROM THIS ENTRY- user space / kernel spacethe CPU privilege ring that enforces the divide; user code cannot cross it directly
- syscallthe trap instruction that transfers control to the kernel; the only legitimate crossing point
- syscall numberthe integer that identifies which kernel service is being requested; differs between operating systems
- translation layersoftware sitting at the boundary that intercepts calls for one kernel and re-issues them to another
- semantic gapthe difference in meaning between analogous calls on two systems, beyond mere naming
Why here, and not somewhere else
The boundary is the right interception point for two compounding reasons: it is narrow, and it is stable.
Narrow means the surface area is manageable. A modern Linux kernel exposes a few hundred syscalls. Windows NT exposes a comparable set through its native interface layer. Translating that set is a large but finite engineering problem. Compare that to intercepting at the hardware instruction level — which is what translation is not emulation describes as the heavier alternative — and the boundary approach is surgical by comparison.
Stable means the interface does not change arbitrarily. Kernel ABIs are maintained with exceptional care because breaking them breaks every program ever compiled. The Linux kernel has a famous policy of never breaking user space; Windows maintains its own long record of backward compatibility. A translation layer written against these interfaces does not need constant revision just because the kernel version increments.
Below the syscall boundary, the compatibility layer has no role. The kernel it runs on is the real kernel: its scheduler, its drivers, its page tables. The layer neither knows nor needs to know what hardware is present. Above the boundary, the program's own code runs as compiled — the layer is not interpreting it. The boundary is the seam, and the layer lives exactly at the seam.
What the boundary cannot hide
The boundary is clean, but it does not make the two systems identical. The objects on each side differ in semantics even when their purpose is the same. Windows file handles carry sharing semantics negotiated at open time; POSIX file locking is a separate and somewhat advisory mechanism. Threading primitives, socket options, timer resolution, signal delivery — each has an analogue on the other side, but analogues are not equivalents. The translation layer has to understand the semantic difference and bridge it, not merely rename the call.
Some things do not translate at all. A program that reaches below its own runtime to inspect kernel structures — certain anti-cheat systems do exactly this — is looking at objects the compatibility layer cannot convincingly fake, because those objects are genuinely absent. The boundary is the right place to intercept the calls that carry normal program intent; it is not a place that can manufacture kernel internals that do not exist.
Everything visible in a compatibility layer's performance cost, correctness work, and occasional failure traces back to this geometry. The syscall boundary is narrow, stable, and hardware-enforced. It is where the two systems are closest to speaking the same language — and where careful engineering can close the remaining gap.
How the interception compares
FROM THIS ENTRY| Hardware instruction interception: rewrite every instruction the CPU would execute | enormous surface area, very high overhead |
|---|---|
| Syscall boundary interception: intercept only the calls that cross the privilege boundary | finite set, stable interface |
| In-process hooking: intercept at the API library level, above the boundary | fragile, revision-sensitive |
| The boundary sits between the two extremes: narrower than instruction-level, more principled than library hooking |
Cedega is an independent publication about compatibility layers. It is not a software vendor, distributor, or support service.