That might not help much because the folders would be added in a sequence and a change to an earlier folder would invalidate all following layers. Maybe keep this as-is and make it a limitation of the non-buildx case. (I think performance with buildx is better in general, so anyone concerned about performance should make sure they have it installed.)
Originally posted by @chrmarti in #13 (comment)