You're correct about comparing the directions. Using cosine similarity would be interesting to evaluate the similarity between the directions created by different protocols.

I think the use of random direction is not a fair comparison because all gradients naturally compute the direction. I do think there is value in analyzing a more "robust pool of algorithms" as you put it. That's something I always integrate in my pipelines.

You are spot-on about how these notions are not very clearly defined. That's something I stress in my content. Also why I try to keep my articles jargon free. It allows for a wider area of discussion and acknowledgment of flaws/improvements. I hope you'll continue to share your insights on my articles. . They benefit everyone involved.