I am able to set them in gradle.properties but I have not been able to set them from within a plugin. I have tried project.setProperty and project.ext.setProperty but neither works.
You simply cannot.
There is no supported way to set a project property from a plugin. project.setProperty also just sets an extra property and those would only be found when actually using project.getProperty or project.ext.getProperty.
But the property is read using providerFactory.gradleProperty which only finds true Gradle Project properties and only from the root project gradle.properties or from commandline / environment.
What you could do to enforce this as rule when your plugin is applied is verifying that it is set to the expected value and otherwise error out, requiring the plugin consumer build to properly configure the value as expected.