Language oracle#
The oracle corpus lives in the pymcu-avr repo at tests/oracle/ – it needs the AVR
backend and the Avr8Sharp emulator, not just the compiler, so the 2026-06 repo split
left it behind here until it was moved over. What this document describes (the probe
header grammar, the divergence-citation registry, the running log of what the oracle
has caught) is still the contract; the citations it names (docs/language/...) resolve
in this repository.
Each probe under tests/oracle/probes/ is a small top-level-statement program that is
run two ways – once directly under CPython, once compiled with PyMCU and executed in
the avr8sharp ArduinoUno emulator through a small C# runner on Avr8Sharp.TestKit
(UART0 captured until the program prints END) – and the two outputs are compared
line by line. tests/oracle/test_oracle.py is the pytest runner; it is parametrized
one test per probe file.
Each probe carries headers:
# expect: match– CPython and the emulator must print byte-identical UART output.# expect: refuse <substring>– the compiler must reject the program, and the diagnostic text must contain<substring>.# expect: divergence <doc citation>– the compiler’s output is DELIBERATELY different from CPython’s, and the citation names where the docs say so (docs/language/type-system.md:20:boolfolds to1/0, notTrue/False;docs/language/roadmap.md:64: a triple-quoted string’s leading newline is stripped). The runner applies the documented transform to CPython’s own output and compares that to the emulator, so the probe passes for the documented reason and starts failing again the moment the divergence disappears (DIVERGENCE_TRANSFORMSintest_oracle.py; an unregistered citation is a hard error, not a silent pass).# doc: <file>:<line>– the roadmap or limitations entry the probe exercises. A probe pinned to a filed-but-not-yet-documented gap may instead cite the GitHub issue URL.# tracked: #<N>– optional; marks the probe as a known, filed compiler bug. The runner turns these intoxfail(strict=True): the suite stays green while the issue is open, and the moment someone fixes the underlying bug the probe XPASSes, which pytest reports as a hard failure – a nudge to remove the# tracked:line and let the probe start asserting again.# frontend: default/# frontend: py-parser– optional; restricts the probe to one compiler front end (the C# parser, orPYMCU_PY_PARSER=1’s Python-astone) and skips it under the other. The two front ends are meant to accept the same language subset, so this exists only for a genuine, filed disagreement between them (one refuses a construct the other accepts, sometimes with different runtime behaviour) – a shape no single# expect:can assert honestly for both engines at once. Such a gap is always covered by a pair of probes, one per front end, both citing the same issue, so the disagreement itself stays visible in the suite rather than being silently narrowed to whichever engine happens to run.
Run it from a pymcu-avr checkout (with PyMCU beside it, both venvs installed):
dotnet build tests/oracle/runner/PyMCU.OracleRunner.csproj -c Release -o build/oracle
.venv/bin/python -m pytest tests/oracle -q
PYMCU_PY_PARSER=1 .venv/bin/python -m pytest tests/oracle -q
PYMCU_BIN defaults to the pymcu-avr checkout’s .venv/bin/pymcu (editable installs
resolve pymcuc through the sibling PyMCU’s src/driver/pymcuc and the backend
through pymcu-avr/build/bin/pymcuc-avr); PYMCU_BACKEND_BINARY and
PYMCU_ORACLE_RUNNER override the backend binary and the prebuilt runner DLL. Every
probe is meant to be run, and kept green, under both front ends.
A probe is never edited to make a genuine mismatch go away. A probe IS edited when
the probe itself is wrong – a stale import path, a header that cites a limitation the
program does not actually hit, a # expect: refuse <substring> that does not match the
compiler’s real (correct) wording – and the fix is checked by confirming the corrected
probe now tests what its own header and doc citation claim. Every other mismatch is
evidence about the compiler: it is filed as a GitHub issue, the probe is left exactly as
it stands, and # tracked: #<N> is added so the suite reports it honestly instead of
quietly skipping it.
Result of the last full run (2026-09-15, after the language-surface sweep)#
178 probes (120 before this sweep, +58: probes 121-178, one per bullet of a
Python-language-reference pass over builtins, integer semantics, strings, control flow,
the data model, collections, exceptions, functions, modules and async – skipping only
what an existing probe already covered). Two of those 58 come as # frontend:-scoped
pairs (163/164, 168/169, 170/171 – six probes covering three front-end
disagreements as three pairs) rather than single probes, because the two front ends
give genuinely different answers.
Per front end:
Front end |
match |
divergence (documented) |
refused (documented) |
tracked (known bug) |
frontend-scoped, N/A here |
|---|---|---|---|---|---|
default (C#) |
108 |
11 |
31 |
25 (18 distinct issues) |
3 |
|
108 |
11 |
30 |
26 (19 distinct issues) |
3 |
Both runs are green: every non-tracked probe matches or refuses exactly as documented,
and every tracked probe’s failure is the one named issue, xfail(strict) so a fix shows
up as a hard XPASS rather than a silent pass.
Compiler bugs, by cause#
Cause |
Issue |
Probes |
|---|---|---|
Unannotated integer arithmetic (loop accumulator, straight-line expression, or runtime-bounded slice sum) is sized from its first store and never widened, so it wraps mod 256 |
#364 (existing, open) |
|
A field read through a |
|
|
A class with no explicit |
|
|
|
|
|
|
|
|
List comprehensions beyond one clause: nested |
|
|
Operator dunders on a ZCA class ( |
|
|
An explicit |
|
|
A two-index |
|
|
|
|
|
Indexing a one-character result out of a runtime string returns the character code, not a one-character string |
|
|
Reading an Enum member ( |
|
|
|
|
|
|
#436 (filed by this sweep) |
|
|
#438 (filed by this sweep) |
|
|
#439 (filed by this sweep) |
|
|
#440 (filed by this sweep) |
|
A type reached through a module alias ( |
#449 (filed by this sweep) |
|
Probe |
Feature |
Doc |
Expectation |
Outcome |
First differing line / diagnostic |
|---|---|---|---|---|---|
|
if elif else |
|
match |
match |
|
|
while break continue |
|
match |
match |
|
|
range runtime bound |
|
match |
match |
|
|
range counter width 16 |
|
match |
match |
|
|
range signed descending |
|
match |
match |
|
|
for fixed array |
|
match |
match |
|
|
for list literal |
|
match |
tracked #364 |
CPython= |
|
for named tuple long |
|
match |
match |
|
|
for string list |
|
match |
match |
|
|
for pair list |
|
match |
match |
|
|
enumerate list |
|
match |
match |
|
|
enumerate range runtime |
|
match |
match |
|
|
zip lists |
|
match |
match |
|
|
reversed list |
|
match |
match |
|
|
reversed range |
|
match |
match |
|
|
match literal or wildcard |
|
match |
match |
|
|
match guard |
|
match |
match |
|
|
match sequence |
|
match |
tracked #401 |
CPython= |
|
match capture |
|
match |
match |
|
|
function defaults keywords |
|
match |
match |
|
|
keyword only defaults |
|
match |
match |
|
|
tuple multireturn inline |
|
match |
match |
|
|
top level script |
|
match |
match |
|
|
class fields methods |
|
match |
match |
|
|
property setter |
|
match |
match |
|
|
nested class constants |
|
match |
match |
|
|
enum constants |
|
match |
tracked #400 |
|
|
inheritance super defaults |
|
match |
match |
|
|
with context |
|
match |
tracked #390 |
CPython= |
|
multi with context |
|
match |
tracked #390 |
CPython= |
|
assert true |
|
match |
match |
|
|
global access |
|
match |
match |
|
|
nonlocal inline |
|
match |
match |
|
|
try except else finally |
|
match |
match |
|
|
except tuple |
|
match |
match |
|
|
except as args |
|
match |
match |
|
|
raise method caller |
|
match |
tracked #391 |
|
|
unhandled raise |
|
match |
match |
|
|
integer promotion |
|
match |
match |
|
|
int cast wrap |
|
match |
match |
|
|
signed unsigned compare |
|
divergence |
match |
|
|
floor div mod negative |
|
match |
match |
|
|
true division float |
|
match |
match |
|
|
divide zero caught |
|
match |
match |
|
|
fstring stream formats |
|
match |
match |
|
|
fstring float stream |
|
match |
match |
|
|
print bytearray |
|
match |
match |
|
|
print array slice |
|
match |
match |
|
|
print obj slice |
|
match |
tracked #392 |
|
|
print float rounding |
|
match |
match |
|
|
many arguments |
|
match |
tracked #364 |
CPython= |
|
in not in |
|
divergence |
match |
|
|
is none |
|
divergence |
match |
|
|
divmod builtin |
|
match |
match |
|
|
bitcast builtin |
|
match |
match |
|
|
hex bin str pow |
|
match |
tracked #393 |
CPython= |
|
sum any all |
|
divergence |
match |
|
|
bytes literal |
|
match |
match |
|
|
bytearray mutation |
|
match |
match |
|
|
int from bytes |
|
match |
match |
|
|
raw string |
|
match |
match |
|
|
extended unpacking |
|
match |
match |
|
|
nested list comprehension |
|
match |
tracked #394 |
CPython= |
|
comprehension filter |
|
match |
tracked #394 |
|
|
class instance comprehension |
|
match |
tracked #394 |
|
|
class list parameter instances |
|
match |
match |
|
|
class list parameter numbers |
|
match |
match |
|
|
class bytearray field |
|
match |
match |
|
|
str join constant |
|
match |
match |
|
|
str join runtime buffer |
|
match |
match |
|
|
slice read |
|
match |
match |
|
|
slice assign |
|
match |
match |
|
|
slice iter runtime bounds |
|
match |
tracked #364 |
CPython= |
|
lambda no capture |
|
match |
match |
|
|
dunder arithmetic comparison |
|
match |
tracked #395 |
CPython= |
|
len bool truth name |
|
match |
tracked #396 |
CPython= |
|
bool truth field |
|
match |
match |
fixed by PyMCU#385 ( |
|
two index get set |
|
match |
tracked #397 |
CPython= |
|
class attribute instance read |
|
match |
tracked #391 |
|
|
descriptor get set |
|
match |
tracked #391 |
|
|
extern decorator refuse without symbol |
|
refuse undefined reference |
refused |
|
|
name guard |
|
match |
match |
|
|
triple quoted string |
|
divergence |
match |
|
|
heap list |
|
match |
tracked #398 |
CPython= |
|
closed dict set literals |
|
divergence |
match |
|
|
fixed dict |
|
match |
match |
|
|
fstring value |
|
match |
tracked #399 |
CPython= |
|
generator for |
|
match |
match |
|
|
type inference unannotated |
|
match |
match |
|
|
nested zca field method |
|
match |
match |
|
|
optional default none |
|
match |
match |
|
|
module constants reassign |
|
match |
match |
|
|
import inside try |
|
match |
match |
|
|
match dotted name |
|
match |
match |
|
|
while else |
|
match |
match |
|
|
for else |
|
match |
match |
|
|
break continue unrolled |
|
match |
match |
|
|
range membership |
|
divergence |
match |
|
|
async run |
|
match |
match |
|
|
range value refused |
|
refuse range |
refused |
|
|
runtime slice read refused |
|
refuse slice |
refused |
|
|
inline fstring expr refused |
|
refuse f-string |
refused |
|
|
dict comprehension refused |
|
refuse dict comprehensions |
refused |
|
|
set comprehension refused |
|
refuse set comprehensions |
refused |
|
|
generator expression refused |
|
refuse |
refused |
|
|
yield in method refused |
|
refuse yield |
refused |
|
|
yield expression refused |
|
refuse yield |
refused |
|
|
multiple inheritance refused |
|
refuse inheritance |
refused |
|
|
recursion refused |
|
refuse recursive |
refused |
|
|
builtin abs min max |
|
match |
match |
|
|
builtin round refused |
|
refuse |
refused |
|
|
builtin int float bool cast |
|
divergence |
match |
|
|
builtin chr ord hex bin |
|
match |
match |
|
|
builtin oct refused |
|
refuse |
refused |
|
|
len array and string |
|
match |
match |
|
|
chr across function return refused |
|
match |
tracked #436 |
line 1: CPython=’B’, emulator=’66’ |
|
shift variable amount |
|
match |
match |
|
|
bitwise on signed |
|
match |
match |
|
|
comparison chain |
|
divergence |
match |
|
|
augmented assignment all operators |
|
match |
match |
|
|
annotated width wraps vs cpython bigint |
|
divergence |
match |
|
|
folded overflow refused |
|
refuse |
refused |
|
|
string concatenation literals |
|
match |
tracked #438 |
line 1: CPython=’helloworld’, emulator=’257’ |
|
string equality comparison |
|
match |
tracked #438 |
line 1: CPython=’yes’, emulator=’no’ |
|
fstring alignment specs refused |
|
refuse |
refused |
|
|
str upper lower refused |
|
refuse |
refused |
|
|
str strip startswith find refused |
|
refuse |
refused |
|
|
in on string refused |
|
refuse |
refused |
|
|
string slice refused |
|
refuse |
refused |
|
|
nested loop break continue |
|
match |
match |
|
|
ternary expression |
|
match |
match |
|
|
walrus operator |
|
match |
match |
|
|
pass statement |
|
match |
match |
|
|
nested function without inline refused |
|
refuse |
refused |
|
|
dunder eq le sub mul |
|
match |
tracked #395 |
line 1: CPython=’True’, emulator=’1’ |
|
dunder str refused |
|
refuse |
refused |
|
|
dunder contains refused |
|
refuse |
refused |
|
|
dunder call |
|
match |
match |
|
|
dunder iter next protocol refused |
|
refuse |
refused |
|
|
class attribute write |
|
match |
match |
|
|
isinstance refused |
|
refuse |
refused |
|
|
list for over heap list |
|
match |
tracked #398 |
line 1: CPython=’9’, emulator=’0’ |
|
list index method refused |
|
refuse |
refused |
|
|
tuple indexing |
|
match |
match |
|
|
dict membership in |
|
divergence |
match |
|
|
nested try except |
|
match |
match |
|
|
bare reraise |
|
match |
match |
|
|
user exception class |
|
match |
match |
|
|
exception crosses inline boundary |
|
match |
match |
|
|
args star compile time |
|
match |
match |
|
|
kwargs double star key access |
|
match |
match |
|
|
positional only refused default frontend (frontend: default) |
|
refuse |
refused |
|
|
positional only unenforced py parser (frontend: py-parser) |
|
match |
match |
|
|
default arg from global |
|
match |
match |
|
|
type annotation ignored at runtime |
|
match |
match |
|
|
inline vs plain function |
|
match |
match |
|
|
match tuple pattern refused default frontend (frontend: default) |
|
refuse |
refused |
|
|
match tuple pattern wrongcode py parser (frontend: py-parser) |
|
match |
tracked #439 |
line 1: CPython=’5’, emulator=’0’ |
|
match class pattern default frontend (frontend: default) |
|
match |
match |
|
|
match class pattern refused py parser (frontend: py-parser) |
|
refuse |
refused |
|
|
import as module alias |
|
match |
tracked #449 |
line 1: CPython=’300’, emulator=’44’ |
|
import star |
|
match |
match |
|
|
name main idiom |
|
match |
match |
|
|
chip conditional |
|
match |
match |
|
|
sys implementation refused |
|
refuse |
refused |
|
|
asyncio sleep loop |
|
match |
match |
|
|
asyncio gather await refused |
|
refuse “only await asyncio.sleep(n)” |
refused |
Probe defects fixed in this pass#
Nine probes were wrong, not the compiler; each is now corrected to test what its header and doc citation actually claim:
018_match_sequence.pymatched on a runtime tuple bound to a name first, which trips the separately-documented tuple-as-value limitation rather than exercising the sequence pattern the probe names. Rewritten to match on a list, the documented sequence-pattern subject – which then surfaced a real bug,#401above.022_tuple_multireturn_inline.py/033_nonlocal_inline.pyimportedinlinefrompymcuinstead ofpymcu.types, where it actually lives; the test harness’s CPython shim was also missingpymcu.types.inline, fixed alongside the probes.048_print_array_slice.pycompared auint8[4]-annotated list literal, which the oracle’s CPython shim leaves as a plain list, against PyMCU’sbytearray-style slice repr. Rewritten to build a realbytearrayon both sides, the same pattern already used by047/059/068/071.050_print_float_rounding.pywrappedprint(float)in afor x in [3.25, ...]:loop over a float list literal, a form the docs do not claim is an iterable. Rewritten as four directprint()calls.091_optional_default_none.pywrotefrom typing import Optional, which the compiler refuses (typingis an explicitly-refused module);docs/language/limitations.md:377saysOptional[X]needs no import at all. The documented idiom (try: from typing import Optional / except ImportError: pass) compiles under PyMCU and imports normally under CPython, so the probe now uses that.103_dict_comprehension_refused.py/104_set_comprehension_refused.py/105_generator_expression_refused.pynamed a# expect: refusesubstring in the wrong case (Dict/Set/Generator) against diagnostics that actually saydict/setcomprehensions in lowercase, and a generator expression that is refused at parse time (SyntaxError: Expected ')') rather than with a message naming “Generator” at all.
What the docs claim that the oracle could not exercise#
PIC / RISC-V / ARM backends: the oracle only targets
atmega328pthroughavr8sharp; none of the other backends the roadmap documents are exercised here.HAL and driver modules (
pymcu.hal.*,pymcu.drivers.*): out of scope for a language-feature oracle; they need real or emulated peripherals, not just UART.Mutable globals across separate module files: every probe is one top-level file (
compile_probe()always writes a singlesrc/main.py); a genuine cross-module probe needs multi-file project support the harness does not have yet.MemoryErrorfrom the bounded bump allocator: exercising it needs the arena to actually fill, which needslist[T].append()to actually store elements – currently broken (#398, probes084/153). Revisit once that is fixed.
Regression sweep of 2026-09-21 (corpus moved to pymcu-avr)#
The corpus moved out of this repository with the backend split and sat unrun while six probes went red on main, in both front ends. Bisecting each to its introducing commit:
009_for_string_list.py/010_for_pair_list.py: printed the interned ids256,257, … where the element texts were meant. Introduced by107fb08a(“fold const names and field ternaries in a for-in tuple”), which let the element folder run before the literal’s string branch and reduced each literal to its interned id. Fixed by binding a folded id that resolves back to an interned string as text (and keeping the character code for length-1 texts, the conventionos.listdirunrolling already used).070_str_join_runtime_buffer.py:print(s)afters = "".join([chr(b) for b in buf])printed the buffer once for every string write the program made – “ABCABCABCABC” with no newline and no “END”.a38f4480(“resolve module-scope array stores to the canonical name”, #460) gave the indexed LOAD path aModuleScopeArrayNamefallback whosemain.<suffix>probe ignores function-local bindings, so thesparameter ofuart_write_strread the module’smain.sand the callee never touched its argument. Fixed by theLocalScopeBindsguard the store path’sResolveArrayVaralready had.134_string_concatenation_literals.py: strict XPASS. #438 was fixed the same day (a + bof two string variables folds their texts again); the# tracked:marker was dropped and the reason moved into the probe’s docstring.135stays tracked:==on a bound string name still does not fold.139_in_on_string.py/148_dunder_contains.py: stalerefuseexpectations from before the features shipped –needle in sfolds on a compile-time string name (cbd581d4, roadmap.md:49) andx in <bound instance>dispatches__contains__(334d8bef, roadmap.md:41). Both now expect the documented bool divergence (type-system.md:20:1where CPython printsTrue) and were renamed accordingly.
The corpus now has a gate: just test-oracle in pymcu-avr builds the runner and runs
the suite under both front ends, and it is named among the suites a commit must keep
green in AGENTS.md/CLAUDE.md.
Comparison dunders and the construction hooks, 2026-09-25 (#491)#
Eleven probes, 281 to 291, added with the fix for the five silent divergences of
#491. What they pin:
281,282,283– all six comparison dunders reach their method as a condition (if,while, both operands ofand/or) and in value position. Before the fix the condition path lowered the comparison as a conditional jump over the flattened instance handles, which are never written, so everyif a == b:answered “equal” while the same comparison assigned to a name dispatched correctly.284– a module-level instance is a dunder receiver.print(a + b)at top level answered 0 for any operator dunder, arithmetic included, because the receiver was looked up under the synthesized module body’s scope while the binding is filed under its bare name. This is the half of #395 that closed, which is why146is untracked.285– a class-typed field receiver,self.lhs == self.rhs.286– two instances of a class with no comparison dunder compare by IDENTITY, as CPython does, withb = arecognised as the same object.287,288– an ordering between two such instances, andmax()/min()over instances, are refused rather than answered from the never-written handles.289,290–__new__and__init_subclass__are refused where the method is written.291– a documented divergence rather than a fix:__del__is emitted nowhere, so whatever its body prints is absent from the firmware’s output. The transform registered againstdocs/language/limitations.md:383drops exactly those lines from CPython’s.