Writing a basic x86-64 JIT compiler from scratch in stock Python

by cslon 11/9/2017, 8:22 PMwith 50 comments

by Jasper_on 11/10/2017, 12:13 AM

To be clear about the `del block` thing, the only thing that the `del` keyword does in Python is unbind a name from the local scope. It does not cause an object to be destructed, call `__del__`, or anything else. It's morally equivalent to "block = None", except future references to block will raise a NameError instead of giving you None.

When used at the end of a function, when all locals go out of scope, it is actually doing nothing.

by dscoon 11/9/2017, 10:10 PM

Just what I love, a scientific and pedagogic CS article, in well formulated English.

I would appreciate some introductory notes or links to prerequisites though. That would perhaps entice an audience who has no previous experience in writing compilers.

by one_thawton 11/9/2017, 10:45 PM

In my opinion, the most tedious part of writing this sort of thing from scratch is the x86/x86-64 instruction encoding. Grab the Intel/AMD manuals or an opcode database and have fun. Other projects to study: LLVM, dynasm, asmjit, peachpy, luajit jit, Intel XED, etc.

by tamer_con 11/9/2017, 10:04 PM

off topic question: When do you find time do such interesting things as these? I'd really like to dive deeper in this topic. As a CS student, who also got to work as a dev at the same time for a living, I'm even happy to get to sleep 8 hours.

by mnembrinion 11/10/2017, 9:35 AM

A nice trick for debugging code generated at runtime is temporarily adding a CC instruction at the beginning of your assembly function, so you don't have to care where in memory your snippet is going to end up

by definedon 11/10/2017, 5:47 AM

This is a very interesting and well-written article, but I am puzzled by the technique being described as "JIT compilation".

Isn't JIT compilation the process of compiling some sort of intermediate code into machine code at runtime? For example, compiling JVM bytecode into machine code during execution, and using the cached machine code transparently on subsequent calls to that segment of code.

Not to detract from the article, or the interesting techniques and explanation, but I didn't see any compilation other than by gcc, which IMHO makes this AOT rather than JIT compilation.

What am I missing?

by pjmlpon 11/9/2017, 9:36 PM

Love the idea of only using stock Python.

Congratulations on the article.

by petefordeon 11/10/2017, 4:37 AM

I am reminded of a most excellent piece of science fiction, "Coding Machines".

https://www.teamten.com/lawrence/writings/coding-machines/

by Bojjaganion 11/10/2017, 12:34 AM

good performance

by kovrikon 11/9/2017, 9:51 PM

That is awesome! Thanks!

Does anybody know the easiest way to compile to JVM bytecode? I have a Scheme interpreter written in Kotlin, what is the best way to compile it, instead of interpreting? Where do I start?