gh-97973: Return all necessary information from the tokenizer#97984

lysnikolaou merged 7 commits into

python:mainpython/cpython:mainfrom

lysnikolaou:tokenizer-return-token-2lysnikolaou/cpython:tokenizer-return-token-2Copy head branch name to clipboard