I posted a comment directly on the blog so I guess I'll do so here as well (with minor edits):
Hi Laurie, I think yk is really impressive work. However, I think the methodology and reporting is very flawed. IIUC the premise of yk is that you take a interpreter with some modifications and you get a JIT. This isn't doing just that. ykcpython is not deriving a JIT from an existing interpreter, but using the existing CPython 3.15 JIT. I looked at the code and ykcpython is using the already-optimized uop stream from the CPython 3.15 JIT tracer + optimizer, just leaving out the Copy-and-Patch backend. The high-level Python optimizations are already done by the existing CPython JIT pipeline at that point, and we know that yk (for now) cannot do most of these optimizations. This article is not actually comparing JITs but instead JIT backends. Also, the tier 2 interpreter is not an actual CPython interpreter we use or ship, it's purely for differential testing of the JIT. I don't think pure interpreters include a JIT optimizer normally, a JIT bytecode tracer, or a differential testing interpreter for their JIT IR. All of which ykcpython uses. Would ykcpython even be able to effectively JIT CPython without leveraging its pre-existing JIT infrastructure? It's hard to conclude from this article.
This is significant, because last I measured, about >90% of 3.15's own JIT speedups came from the 3.15 JIT optimizer, and only 10% of that was from the JITting to machine code. In fact, without the JIT optimizer, the 3.15 JIT is usually outright slower than the interpreter. There are probably gains from the indirect function call inlining. However, it is hard to say how the entire system will perform.
To expound further, 3.15's JIT optimizer traces bytecodes to uops and optimizes that. ykcpython is taking the stream of uops directly from the JIT optimizer. Most of the high-level optimizations the CPython JIT optimizer does cannot be done by yk as they require knowledge of Python semantics. Unlike what you may have seen in 3.14, the 3.15 JIT optimizer actually does quite a bit of reference counting elimination, unique reference tracking, top-of-stack caching, type check removal via callback insertion, and general high-level Python semantic-aware optimizations. yk (for now) cannot reason about any of these Python semantics, and so this implementation outsourced all the work that yk cannot do to the CPython JIT optimizer + tracer, and then effectively compares the yk optimizer vs the non-optimizing Copy-and-Patch backend, which is not as interesting. Again, the JIT tracer+optimizer is responsible for the vast majority of 3.15's speedups.
On top of that, the CPython JIT tracer is itself, a bytecode tracing engine. Which means that it already traces and linearizes all the high-level Python control-flow + performs stitching/trace-trace transitions which cuts out the bulk of the work of region selection for the yk tracer. I think that's also an important factor as tracing JITs are highly-dependent on the quality of their traces. The JIT trace builder is again a part of 3.15's JIT, not by default with its interpreter.
A suggestion for a more interesting and accurate comparison would be to turn off the specializing interpreter and yk trace+compile that (the most apples-to-apples comparison). I expect in this case, ykcpython (for the time being) will perform significantly worse just by tracing the bytecode interpreter than it currently does, potentially even worse than the specializing interpreter. I say for the time being, because I am confident that yk will improve so I make no pronouncements on future results.
If this article were about comparing JIT backends or producing an interesting JIT infrastructure, I would agree with it. However, I cannot agree with the original yk premise that this is taking a mostly unmodified interpreter and deriving a JIT from that to get good results. This piggybacks off an existing JIT's trace recorder and optimizer. So I think yk's original claim doesn't hold at all in this case. If CPython's JIT infrastructure were completely removed in 3.16, ykcpython would (as the current implementation stands) be ineffective and not be able to handle the specializing interpreter at all.
The article is pretty captivating. There is also an interesting comment from Ken Jin (who is working on JIT for cPython). This supports my opinion that comments are an important part of a blog. :)
kenjin | 2 hours ago
I posted a comment directly on the blog so I guess I'll do so here as well (with minor edits):
Hi Laurie, I think yk is really impressive work. However, I think the methodology and reporting is very flawed. IIUC the premise of yk is that you take a interpreter with some modifications and you get a JIT. This isn't doing just that. ykcpython is not deriving a JIT from an existing interpreter, but using the existing CPython 3.15 JIT. I looked at the code and ykcpython is using the already-optimized uop stream from the CPython 3.15 JIT tracer + optimizer, just leaving out the Copy-and-Patch backend. The high-level Python optimizations are already done by the existing CPython JIT pipeline at that point, and we know that yk (for now) cannot do most of these optimizations. This article is not actually comparing JITs but instead JIT backends. Also, the tier 2 interpreter is not an actual CPython interpreter we use or ship, it's purely for differential testing of the JIT. I don't think pure interpreters include a JIT optimizer normally, a JIT bytecode tracer, or a differential testing interpreter for their JIT IR. All of which ykcpython uses. Would ykcpython even be able to effectively JIT CPython without leveraging its pre-existing JIT infrastructure? It's hard to conclude from this article.
This is significant, because last I measured, about >90% of 3.15's own JIT speedups came from the 3.15 JIT optimizer, and only 10% of that was from the JITting to machine code. In fact, without the JIT optimizer, the 3.15 JIT is usually outright slower than the interpreter. There are probably gains from the indirect function call inlining. However, it is hard to say how the entire system will perform.
To expound further, 3.15's JIT optimizer traces bytecodes to uops and optimizes that. ykcpython is taking the stream of uops directly from the JIT optimizer. Most of the high-level optimizations the CPython JIT optimizer does cannot be done by yk as they require knowledge of Python semantics. Unlike what you may have seen in 3.14, the 3.15 JIT optimizer actually does quite a bit of reference counting elimination, unique reference tracking, top-of-stack caching, type check removal via callback insertion, and general high-level Python semantic-aware optimizations. yk (for now) cannot reason about any of these Python semantics, and so this implementation outsourced all the work that yk cannot do to the CPython JIT optimizer + tracer, and then effectively compares the yk optimizer vs the non-optimizing Copy-and-Patch backend, which is not as interesting. Again, the JIT tracer+optimizer is responsible for the vast majority of 3.15's speedups.
On top of that, the CPython JIT tracer is itself, a bytecode tracing engine. Which means that it already traces and linearizes all the high-level Python control-flow + performs stitching/trace-trace transitions which cuts out the bulk of the work of region selection for the yk tracer. I think that's also an important factor as tracing JITs are highly-dependent on the quality of their traces. The JIT trace builder is again a part of 3.15's JIT, not by default with its interpreter.
A suggestion for a more interesting and accurate comparison would be to turn off the specializing interpreter and yk trace+compile that (the most apples-to-apples comparison). I expect in this case, ykcpython (for the time being) will perform significantly worse just by tracing the bytecode interpreter than it currently does, potentially even worse than the specializing interpreter. I say for the time being, because I am confident that yk will improve so I make no pronouncements on future results.
If this article were about comparing JIT backends or producing an interesting JIT infrastructure, I would agree with it. However, I cannot agree with the original yk premise that this is taking a mostly unmodified interpreter and deriving a JIT from that to get good results. This piggybacks off an existing JIT's trace recorder and optimizer. So I think yk's original claim doesn't hold at all in this case. If CPython's JIT infrastructure were completely removed in 3.16, ykcpython would (as the current implementation stands) be ineffective and not be able to handle the specializing interpreter at all.
vbernat | 2 hours ago
The article is pretty captivating. There is also an interesting comment from Ken Jin (who is working on JIT for cPython). This supports my opinion that comments are an important part of a blog. :)