feat: pass spark.comet.datafusion.* configs through to DataFusion session

[andygrove]

Summary

• Spark configs with the prefix spark.comet.datafusion.* are now passed through to DataFusion's SessionConfig
• The spark.comet. prefix is stripped so that e.g. spark.comet.datafusion.sql_parser.parse_float_as_decimal=true becomes datafusion.sql_parser.parse_float_as_decimal=true in DataFusion
• This allows users to tune DataFusion behavior from Spark via --conf without code changes

🤖 Generated with Claude Code

[comphead]

Thanks @andygrove I'm happy we put back sending those user defined DF configs to the native execution

[mbutrovich]

Are there any configs that we can think of that would absolutely break Comet correctness? Either adding to a disallow list to filter out, or warn users about?

[andygrove]

Are there any configs that we can think of that would absolutely break Comet correctness? Either adding to a disallow list to filter out, or warn users about?

Good point. We could put this functionality behind a new config flag with appropriate documentation. Something like spark.comet.exec.respectDataFusionConfigs defaulting to false?

[mbutrovich]

Are there any configs that we can think of that would absolutely break Comet correctness? Either adding to a disallow list to filter out, or warn users about?

Good point. We could put this functionality behind a new config flag with appropriate documentation. Something like spark.comet.exec.respectDataFusionConfigs defaulting to false?

At a minimum I think that's a good start. It's a huge config space, so I think guarding it behind another config that says something to the effect of "DataFusion configs are exposed for development purposes and possible workload-specific optimizations. Changing DataFusion configs could affect Comet correctness and/or stability."

[andygrove]

Thanks for the review @mbutrovich. I added the config.

[mbutrovich]

LGTM, thanks @andygrove!