gh-150207: Raise MemoryError on tokenizer allocation failure instead of crashing

[grantlouisherman]

Original report was around this snippet of code:

MB_512 = 512 * 1024 * 1024
resource.setrlimit(resource.RLIMIT_AS, (MB_512, MB_512)) 
eval("A"* 420000000)

Throwing a system error. Based off the write up made in #150207 I felt like there was a fair case to actually show the user a Mem error instead of a system crash.

... MB_512 = 512 * 1024 * 1024
... resource.setrlimit(resource.RLIMIT_AS, (MB_512, MB_512))
... eval("A"* 420000000)
...
Traceback (most recent call last):
  File "<python-input-0>", line 4, in <module>
    eval("A"* 420000000)
    ~~~~^^^^^^^^^^^^^^^^
MemoryError
>>>

I think this does provide a bit nicer of an experience but I think I need an actual adult to verify this for me. I also dont know the larger ramifications but it felt worth to throw something up

• Issue: Allocation failures generated in _PyTokenizer_FromUTF8 lead to SystemError/abort()s #150207

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[grantlouisherman]

Hey @aisk @stestagg let me know your thoughts

[stestagg]

This seems reasonable to me.

One thing that's important is that any callers of these that try to recover from an allocation error know that there's a py exception to clear. In this case, I don't think there is one, just adding here in case I missed something!

[grantlouisherman]

@pablogsal @lysnikolaou do these changes seem reasonable?

[aisk]

@grantlouisherman Hi, please don't use force push in the future, see the Dev Guide.

[aisk]

I guess there is another missing check here:

cpython/Parser/tokenizer/file_tokenizer.c

Lines 379 to 381 in 540b3d0

	if ((tok->buf = (char *)PyMem_Malloc(BUFSIZ)) == NULL) {
	_PyTokenizer_Free(tok);
	return NULL;