ecc: fix crash in wc_ecc_import_point_der_ex on invalid format byte

[MarkAtwood]

Summary

wc_ecc_import_point_der_ex crashes with a double-free (SIGABRT) when given a full-length EC point blob with an invalid first byte (0x01, 0x05, 0xFF, etc.).

For example, passing a 65-byte buffer (the correct size for an uncompressed P-256 point) but with first byte 0x01 instead of 0x04 causes a SIGABRT.

Root Cause

The format byte validation (checking for 0x02, 0x03, or 0x04) was performed after mp_clear, mp_init_multi, and SAVE_VECTOR_REGISTERS had already executed. When an invalid byte was detected, err was set to ASN_PARSE_E, but execution continued into code paths that partially initialized state (reading data into mp_ints, etc.). On the error cleanup path, already-freed or uninitialized memory was freed again.

The sequence:

1. mp_clear + mp_init_multi + SAVE_VECTOR_REGISTERS execute
2. Format byte check sets err = ASN_PARSE_E
3. Code continues past the error (many paths check if (err == MP_OKAY) but some don't)
4. Cleanup frees resources that were partially initialized or already freed

Reproducer

// P-256: full uncompressed length but invalid prefix byte
unsigned char bad_point[65];
memset(bad_point, 0x42, sizeof(bad_point));
bad_point[0] = 0x01;  // invalid format byte

ecc_point *pt = wc_ecc_new_point();
// SIGABRT / double-free before fix
int ret = wc_ecc_import_point_der_ex(bad_point, 65, curve_idx, pt, 0);

Also crashes with prefix bytes 0x05, 0xFF, or any byte other than 0x02/0x03/0x04 when the total length matches a valid uncompressed point size.

Security Impact

This is a denial-of-service vector. Any code path that passes untrusted EC point data to wc_ecc_import_point_der_ex (or indirectly via EC_POINT_oct2point in the OpenSSL compat layer) can be crashed by an attacker supplying a malformed public key with an invalid format byte. In TLS, the peer's ECDHE public key is parsed through this path.

Fix

Moved the format byte validation to before any memory operations (mp_clear, mp_init_multi, SAVE_VECTOR_REGISTERS). Invalid format bytes now return ASN_PARSE_E immediately, before any state is touched. Removed the now-redundant later check.

Test Plan

• Verify valid uncompressed point import still works (prefix 0x04)
• Verify valid compressed point import still works (prefix 0x02/0x03)
• Verify prefix bytes 0x00, 0x01, 0x05, 0xFF return an error without crashing
• Run existing wolfSSL test suite
• Valgrind/ASAN run to confirm no memory errors

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Fixes a double-free crash in wc_ecc_import_point_der_ex when parsing EC point blobs with an invalid format byte by validating the prefix earlier, before any multi-precision memory operations occur.

Changes:

• Validate in[0] point format byte upfront and return ASN_PARSE_E immediately on invalid values.
• Remove the later redundant point type check.
• Add a compressed-point length check when compressed points are enabled.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.