gh-145261: multiprocessing.shared_memory: fix ShareableList corruption for multi-byte strings and null bytes

[zetzschest]

Issue

ShareableList has two issues:

1. It uses len(item) (character count) for string slot allocation instead of len(item.encode('utf-8')) (byte count), causing UnicodeDecodeError with multi-byte UTF-8 characters.
2. It uses rstrip(b'\x00') to recover bytes values, which strips legitimate trailing null bytes.

Reproducer

from multiprocessing.shared_memory import ShareableList

# String corruption
sl = ShareableList(['0\U00010000\U00010000'])
sl[0]  # UnicodeDecodeError

# Bytes corruption
sl = ShareableList([b'\x00'])
sl[0]  # b'' instead of b'\x00'

Fix

Use len(item.encode('utf-8')) for string slot allocation. For bytes, store the actual byte length in the format metadata so retrieval reads exactly the right number of bytes without needing rstrip(b'\x00').

This fix attempts to resolve both the new UTF-8 corruption issue and the long-standing trailing null bytes issue reported in #106939.

Test updates

Two assertions in test_shared_memory_ShareableList_basics needed adjustments since they were based on the previous behavior:

• Format string assertions updated to reflect actual byte lengths stored for bytes values.
• Removed format comparison between original and copy, as copies may allocate differently with byte-accurate lengths.

• Issue: ShareableList corrupts multi-byte UTF-8 strings and bytes with trailing nulls #145261

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[jakelodwick]

I was working on a consolidation PR (#145488) for the same two bugs and arrived at the same fix approach independently. Closed it in favor of this one.

The title should also reference gh-106939, since the null-stripping bug is the older and more-tracked issue.

Tests

The fix changes observable behavior but the PR doesn't add test coverage for either bug. Here are two test methods covering both, including cross-process verification via name=:

def test_shared_memory_ShareableList_trailing_nulls(self):
    # gh-106939: ShareableList should preserve trailing null bytes
    # in bytes and str values.
    sl = shared_memory.ShareableList([
        b'\x03\x02\x01\x00\x00\x00',
        '?\x00',
        b'\x00\x00\x00',
        b'',
        b'no nulls',
    ])
    self.addCleanup(sl.shm.unlink)
    self.addCleanup(sl.shm.close)

    self.assertEqual(sl[0], b'\x03\x02\x01\x00\x00\x00')
    self.assertEqual(sl[1], '?\x00')
    self.assertEqual(sl[2], b'\x00\x00\x00')
    self.assertEqual(sl[3], b'')
    self.assertEqual(sl[4], b'no nulls')

    sl2 = shared_memory.ShareableList(name=sl.shm.name)
    self.addCleanup(sl2.shm.close)
    self.assertEqual(sl2[0], b'\x03\x02\x01\x00\x00\x00')
    self.assertEqual(sl2[1], '?\x00')
    self.assertEqual(sl2[2], b'\x00\x00\x00')
    self.assertEqual(sl2[3], b'')
    self.assertEqual(sl2[4], b'no nulls')

def test_shared_memory_ShareableList_multibyte_utf8(self):
    # gh-145261: ShareableList should correctly handle multi-byte
    # UTF-8 strings without corruption or spillage.
    sl = shared_memory.ShareableList([
        'ascii',        # 1-byte per char (5 bytes)
        'café',         # 2-byte char: é (5 bytes)
        '中文测试',     # 3-byte per char (12 bytes)
        '𐀀𐀁',         # 4-byte per char (8 bytes)
    ])
    self.addCleanup(sl.shm.unlink)
    self.addCleanup(sl.shm.close)

    self.assertEqual(sl[0], 'ascii')
    self.assertEqual(sl[1], 'café')
    self.assertEqual(sl[2], '中文测试')
    self.assertEqual(sl[3], '𐀀𐀁')

    # Verify cross-process access via name-based attachment.
    sl2 = shared_memory.ShareableList(name=sl.shm.name)
    self.addCleanup(sl2.shm.close)
    self.assertEqual(sl2[0], 'ascii')
    self.assertEqual(sl2[1], 'café')
    self.assertEqual(sl2[2], '中文测试')
    self.assertEqual(sl2[3], '𐀀𐀁')

These go in _test_multiprocessing.py after test_shared_memory_ShareableList_pickling_dead_object.

Cross-version compatibility

The shared memory layout (offsets, block sizes) is unchanged by this fix. Old writer → new reader returns padded data (same as current release, no regression). New writer → old reader gets exact data, then the old rstrip(b'\x00') is harmless unless the data ends in nulls — which is the existing bug. Neither direction crashes. Might be worth a note in the PR description.