Sounds like a class loader problem.
The one registering LibraryExtension uses the class from one class loader,
while the one trying to get the extension uses the class from a different class loader.
As the classes are from different class loaders, they are not considered the same class and so the extension cannot be found.
I guess you use withPluginClassPath. Iirc, this can lead to such problems in more sophisticated situations like yours.
You should consider to instead publish the plugin to some file-based local repository (but not mavenLocal() which is broken by design), and then consume your plugin from that repository in your tests. This should hopefully resolve the class loader issue.