Currently the only 3rd party tokenization VSCode supports is a massive pile of RegEx. There's a whole discussion about how procedural tokenization could be supported without running extension code in the UI thread. The central argument against delegating this to an external worker is that if the reply doesn't arrive fast enough it might interfere with characters typed later.

1. Any computer that can run VSCode can execute somewhere in the order of a _billion_ instructions per second. To a program, the delay between keystrokes is an eternity. The only way to run out of time here is if either the dev isn't aware that the request is time sensitive, or the framework communicates to the OS that the task isn't urgent and an arbitrary amount of work is scheduled before it.

2. Chromium is the pinnacle of cybersecurity and its primary job is to sandbox untrusted user code. You don't need another thread to do it.

3. This use case fits squarely in the original design objectives of Webassembly.