Improve OpenSSL 4.0 compatibility, part 1

[rhenium]

OpenSSL's master branch is going to be OpenSSL 4.0. These functions return const pointers:

• PKCS7_get_signed_attribute()
• X509_ATTRIBUTE_get0_object()
• X509_ATTRIBUTE_get0_type()

[rhenium]

It seems there have been other changes in OpenSSL master between 2026-02-03 and today. I will check these later.

[rhenium]

I will not rush merging this until 4.0 alpha is out, as I expect there will be additional breakages, and I want to avoid cluttering the git history with incremental fixes.

[rhenium]

4.0.0-alpha1 was tagged this week. Test failures with OpenSSL::PKCS12 seem to be an issue in OpenSSL.

[junaruga]

Thank you for working on this issue.

When trying to compile the current latest Ruby master branch commit ruby/ruby@77b5ab1 with the current latest OpenSSL 4.1 dev master commit openssl/openssl@e1eb881, I hit an compile error related to this ticket. Maybe thie PR's patches fix it.

$./configure \
  --prefix=/home/jaruga/.local/ruby-4.1.0dev-debug-77b5ab1f6f-openssl-4.1.0-dev-e1eb88118a \
  --with-openssl-dir=/home/jaruga/.local/openssl-4.1.0-dev-fips-debug-e1eb88118a \
  ...

$ make
...
ossl_asn1.c: In function ‘obj_to_asn1bstr’:
ossl_asn1.c:242:9: error: invalid use of incomplete typedef ‘ASN1_BIT_STRING’ {aka ‘struct asn1_string_st’}
  242 |     bstr->flags &= ~(ASN1_STRING_FLAG_BITS_LEFT|0x07); /* clear */
      |         ^~
ossl_asn1.c:242:22: error: ‘ASN1_STRING_FLAG_BITS_LEFT’ undeclared (first use in this function)
  242 |     bstr->flags &= ~(ASN1_STRING_FLAG_BITS_LEFT|0x07); /* clear */
      |                      ^~~~~~~~~~~~~~~~~~~~~~~~~~
ossl_asn1.c:242:22: note: each undeclared identifier is reported only once for each function it appears in
ossl_asn1.c:243:9: error: invalid use of incomplete typedef ‘ASN1_BIT_STRING’ {aka ‘struct asn1_string_st’}
  243 |     bstr->flags |= ASN1_STRING_FLAG_BITS_LEFT | unused_bits;
      |         ^~
ossl_asn1.c: In function ‘decode_bstr’:
ossl_asn1.c:377:15: error: invalid use of incomplete typedef ‘ASN1_BIT_STRING’ {aka ‘struct asn1_string_st’}
  377 |     len = bstr->length;
      |               ^~
ossl_asn1.c:379:12: error: invalid use of incomplete typedef ‘ASN1_BIT_STRING’ {aka ‘struct asn1_string_st’}
  379 |     if(bstr->flags & ASN1_STRING_FLAG_BITS_LEFT)
      |            ^~
ossl_asn1.c:379:22: error: ‘ASN1_STRING_FLAG_BITS_LEFT’ undeclared (first use in this function)
  379 |     if(bstr->flags & ASN1_STRING_FLAG_BITS_LEFT)
      |                      ^~~~~~~~~~~~~~~~~~~~~~~~~~
ossl_asn1.c:380:28: error: invalid use of incomplete typedef ‘ASN1_BIT_STRING’ {aka ‘struct asn1_string_st’}
  380 |         *unused_bits = bstr->flags & 0x07;
      |                            ^~
In file included from ../.././include/ruby/internal/value_type.h:30,
                 from ../.././include/ruby/internal/fl_type.h:38,
                 from ../.././include/ruby/internal/core/rstring.h:30,
                 from ../.././include/ruby/internal/arithmetic/char.h:29,
                 from ../.././include/ruby/internal/arithmetic.h:24,
                 from ../.././include/ruby/ruby.h:28,
                 from ../.././include/ruby.h:38,
                 from ossl.h:16,
                 from ossl_asn1.c:10:
ossl_asn1.c:381:40: error: invalid use of incomplete typedef ‘ASN1_BIT_STRING’ {aka ‘struct asn1_string_st’}
  381 |     ret = rb_str_new((const char *)bstr->data, len);
      |                                        ^~
../.././include/ruby/internal/constant_p.h:33:55: note: in definition of macro ‘RBIMPL_CONSTANT_P’
   33 | # define RBIMPL_CONSTANT_P(expr) __builtin_constant_p(expr)
      |                                                       ^~~~
ossl_asn1.c:381:11: note: in expansion of macro ‘rb_str_new’
  381 |     ret = rb_str_new((const char *)bstr->data, len);
      |           ^~~~~~~~~~
In file included from ../.././include/ruby/intern.h:57,
                 from ../.././include/ruby/ruby.h:194:
ossl_asn1.c:381:40: error: invalid use of incomplete typedef ‘ASN1_BIT_STRING’ {aka ‘struct asn1_string_st’}
  381 |     ret = rb_str_new((const char *)bstr->data, len);
      |                                        ^~
../.././include/ruby/internal/intern/string.h:1503:21: note: in definition of macro ‘rb_str_new’
 1503 |       rb_str_new) ((str), (len)))
      |                     ^~~
At top level:
cc1: note: unrecognized command-line option ‘-Wno-self-assign’ may have been intended to silence earlier diagnostics
cc1: note: unrecognized command-line option ‘-Wno-parentheses-equality’ may have been intended to silence earlier diagnostics
cc1: note: unrecognized command-line option ‘-Wno-constant-logical-operand’ may have been intended to silence earlier diagnostics
make[2]: *** [Makefile:330: ossl_asn1.o] Error 1
make[2]: Leaving directory '/home/jaruga/var/git/ruby/ruby/ext/openssl'
make[1]: *** [exts.mk:297: ext/openssl/all] Error 2
make[1]: Leaving directory '/home/jaruga/var/git/ruby/ruby'
make: *** [uncommon.mk:346: build-ext] Error 2

[junaruga]

I see the test failure on CI openssl maser branch case below. It is the same with the above failure.

https://github.com/ruby/openssl/actions/runs/23138225126/job/67207433887#step:8:52