Just one question. If I'm running a local model, can I do something other than just a context free grammar? Does it makes sense to have something more general, or it would be just too slow?

I guess the only hard constraint is to not have backtracking, right? To not waste previously emitted tokens

Thinking shouldn't be too hard to deal with---just let the model generate freely until it hits a </think> token, then do constrained decoding, right?

Sure, but does llama-cpp support that?

It does and this is how I did it.

But actually getting that grammar right as well as actually making it work with the correct Jinja template to correctly enable thinking mode and parse it out was a lot more work than I expected.