generatedBy task always run, even when SourceSet is not used, how to make it stop?

I’m adding generated headers to a NativeComponent which works. However I want to only generated some of the headers for a certain buildType/productFlavor but the task is always run even if the SourceSet isn’t used.
Small example:

model {
  buildTypes {
    debug
    release
  }
  components {
    main(NativeExecutableSpec) { m ->
      sources {
        headers(CppSourceSet) {
          generatedBy tasks.generateHeaders
        }
        cpp {
          lib sources.headers
        }
      }
      binaries.all {
        sources {
          if (buildType == buildTypes.debug) {
            buildHeaders(CppSourceSet) {
              generatedBy project.tasks.generateDebugHeader
            }
            m.sources.cpp.lib sources.buildHeaders
          }
        }
      }
    }
  }
}

As you can see I generate two sets of headers, common ones always included and one set only for the buildType debug.
But when I build mainReleaseExecutable output is:

:generateDebugHeader
:generateHeader
:compileMainReleaseExecutableMainCpp
:compileMainReleaseExecutableMainHeaders UP-TO-DATE
:linkMainReleaseExecutable
:mainReleaseExecutable

Only main headers is compiled that is good, but why is generateDebugHeader task executed and is there any way to stop it?