Your method cannot be called unsupervised!

[DomenicoET]

The method is not completely unsupervised. Only feature representation learning is unsupervised, instead you use labels for knn fitting. So the paper title is misleading. Also the abstract is misleading. For example, "We propose a novel system for unsupervised skeleton-based action recognition" or "In contrast, our system is fully
unsupervised, does not require action labels at any stage".
Have you tried to replace knn with an unsupervised clustering approach?