Week in review: 22 June 2026

A recap of what shipped during the week of 22 June. Performance and parity week: an arrays refactor that brings Pext closer to the packed-array shape used by PHP internally, a rework of scope emission that produces vanilla let bindings when the semantics allow it, and a best-effort tokenizer that gives opensourcepos byte-for-byte token parity against the PHP source.

Packed arrays for the fast path

PHP's Zend engine distinguishes two internal array shapes: packed and hashed. A packed array is a dense sequence of consecutive integer keys starting at zero, and the engine stores it as a contiguous C array with direct indexed access. A hashed array is the general form: any key, any order, backed by a hash table with the usual bookkeeping. Most real-world PHP arrays are packed at rest and the engine reaps a considerable speedup on iteration, index reads, and index writes as a result.

Until this week, the Pext array module used a single general representation for both shapes. That was correct, but it left every dense integer-indexed array paying the hash-table overhead. The refactor introduces a packed representation as the fast path: when an array is created empty or with only sequential integer keys, or when a series of operations preserves that invariant, the storage is a flat JS array with direct index access. Any operation that would violate the invariant (setting a non-sequential key, inserting a string key, deleting a middle element) upgrades the array in place to the general representation.

The measured improvement so far is meaningful (double-digit percentages on iteration-heavy code, a smaller but real gain on random indexed access) and there is still headroom. The pending work is around foreach, where the emitter can lower directly to a JS for loop when the analyzer proves the array stays packed for the duration of the loop, and around the operations that split a packed array (like array_slice with negative offsets) where a specialized packed-in packed-out path can avoid the upgrade.

Static scope emission and vanilla let bindings

The transpiler used to route every variable read and write through the dynamic scoping layer, which handles PHP's compact $$name style variable-variable access, its reference semantics through &, its per-scope destructor timing, and its compact and extract family of introspection helpers. Correct in general, and heavier than needed for the common case where none of those features are used.

The rework introduces static access as the default. When the analyzer can prove that a scope does not use variable variables, does not take any addresses, does not close over its locals into a callable that outlives the scope, and does not call any of the introspection helpers, the emitter now emits plain JS let bindings and direct name references. The result reads like hand-written JavaScript: let x = 1; instead of a Pvar wrapper allocated into a Pscope, indexed by string, and unwrapped on read.

The heuristics are conservative. Any evidence that the scope needs the dynamic path (a call to compact, a global declaration, a by-reference parameter, a closure capture that outlives the frame) reverts the whole scope to the general representation. The fast path applies to the majority of functions in a typical codebase, which is where the compounded speedup comes from. This work stacks on the analyze pass introduced in the earlier grab-bag lowering; the analyzer answers more questions, the emitter picks vanilla shapes more often.

A best-effort tokenizer for opensourcepos

The last piece of the opensourcepos parity story was the PHP tokenizer surface. CodeIgniter 4 uses token_get_all to introspect converted files during class loading, and the pext-tokenizer module has been shipping a best-effort implementation that wraps the TypeScript scanner and synthesizes the tokens PHP would have emitted. This week rounded out the coverage to the point where the token stream produced from a converted opensourcepos file matches, at the token level, what native PHP produces from the original.

Byte-for-byte token parity is the precondition for the class-discovery paths CI4 uses in production, and it is also what makes the performance tests fair. If Pext's token stream differed from PHP's in any way, an argument about performance could always be deflected by "the tokens are different, of course the runs are different". With the token surface locked down, the numbers on the ten end-to-end REST scenarios from last week are directly comparable, and the profiler output is now telling a clean story about where each runtime spends its time.