The plugin is already working much as you describe
No, it’s not. The fact that it’s using Set<String> and Set<Task> shows the problem.
Set<String>
Set<Task>
Please read here for related topic