Microsoft, Timnit Gebru, and Google AI
A lot of interesting events happening at once
To help me understand you fill out this survey (anonymous)
Last week was very interesting for Machine Learning. There were a lot of events with potentially long-lasting consequences for Machine Learning. This article will go over some of them so that you’re more informed about some important aspects of Machine Learning and the discussions surrounding them. These events might seem disjointed, but they provide different sides to a very important discussion in machine learning.
Microsoft Takes a Huge Leap Forward in NLP
GLUE (General Language Understanding Evaluation) and SuperGLUE are benchmarks for Natural Language Processing. According to their website, GLUE consists of:
- A benchmark of nine sentence- or sentence-pair language understanding tasks built on established existing datasets and selected to cover a diverse range of dataset sizes, text genres, and degrees of difficulty,
- A diagnostic dataset designed to evaluate and analyze model performance with respect to a wide range of linguistic phenomena found in natural language, and
- A public leaderboard for tracking performance on the benchmark and a dashboard for visualizing the performance of models on the diagnostic set.
SuperGLUE is a harder version of GLUE with more tests etc.
Microsoft made some very impressive strides with their newest publication, “Efficiently and effectively scaling up language model pretraining for best language representation model on GLUE and SuperGLUE”. Allegedly. Why allegedly?
If you look at the publication, they don’t share much. The publication is often vague, using language like, “We also leverage the training dataset and the data processing pipeline optimized for developing previous T-NLR releases, including DeBERTa and UniLM, as well as the implementation optimizations from other Microsoft pretraining research efforts, such as TUPE.”
This is very different from standard protocol. Typically researchers share their methods, processing, and other steps. This ensures that others can replicate, critique, and advance from their work.
However, MS is a private company. Their goal is to make profits. Going by the way they have presented their findings, it is likely they will also eventually sell the model as a paid API/service. Thus it would make sense for them to obscure many details in order to keep their competitive advantage. MS makes up for this by topping a public leaderboard with many heavy-hitters.
“Notably, T-NLRv5 first achieved human parity on MNLI and RTE on the GLUE benchmark, the last two GLUE tasks which human parity had not yet met. In addition, T-NLRv5 is more efficient than recent pretraining models, achieving comparable effectiveness with 50% fewer parameters and pretraining computing costs.”
Timnit Gebru and Big Tech
Timnit Gebru is an Ethiopian American computer scientist who works on algorithmic bias and data mining. She is well known for her push for increasing diversity in AI. She was ousted from Google AI last year, over a conflict about a certain paper. Gebru said that the paper, “didn’t meet our bar for publication”. Since she was a well-known figure, this caused a huge backlash. You can see a huge online letter of protest here.
Last week, she established her own institution. “The Distributed AI Research Institute is a space for independent, community-rooted AI research free from Big Tech’s pervasive influence.”
It is being funded by donations. They were able to raise money for now. However, it will be interesting to see how the institute handles the finances required for ML research going forward. Relying on donations will put them in many of the same pressures (and cause the same issues) that traditional academia faces. This article goes into more depth about the issues caused by the layout in peer review and finding grants.
Going for commercialized research will add many outside factors that might detract from the purpose of the institution. And getting corporate funding will definitely expose their research to influence and meddling.
Such issues are complex with a lot of moving parts. The institution is new, so it doesn’t have to be now, but these are some of the issues when they will have to deal with. And seeing how they handle these issues will be reverberating consequences for Machine Learning Research.
How do these tie together? What should you do about it?
So how do they tie together? MSs move to not open out their work is a pretty clear indication of how they plan to proceed. It’s also part of a trend these days where Big Tech companies use publications for hype rather than academic verification. They share the results without sharing the process which would allow others to find potential issues with the work. This is one of the things that Gebru will seek to combat.
How these influence the Machine Learning Research space is to be determined. There are many things to consider. Regulating the Big Tech method/publications might improve replicability and work for quality control. But it will also disincentivize them from sharing their knowledge (which has been responsible for some of the coolest discoveries in the last decade). For now, we will have to wait and watch.
What can you do about it? For now, educate yourself. It is not enough to just learn about the mechanics of ML, but also how the field operates. Learning about the incentive structures, usages, and biases in the field will help you understand the nuances of situations. This will allow you to see the situations and make a difference in the areas you consider important. Make no mistake, this is something that will affect everybody. And it’s best to be prepared.
If you liked this article, check out my other content. I post regularly on Medium, YouTube, Twitter, and Substack (all linked below). I focus on Artificial Intelligence, Machine Learning, Technology, and Software Development. If you’re preparing for coding interviews check out: Coding Interviews Made Simple, my free weekly newsletter.
For one-time support of my work following are my Venmo and Paypal. Any amount is appreciated and helps a lot:
Venmo: https://account.venmo.com/u/FNU-Devansh
Paypal: paypal.me/ISeeThings
Reach out to me
If that article got you interested in reaching out to me, then this section is for you. You can reach out to me on any of the platforms, or check out any of my other content. If you’d like to discuss tutoring, text me on LinkedIn, IG, or Twitter. If you’d like to support my work, use my free Robinhood referral link. We both get a free stock, and there is no risk to you. So not using it is just losing free money.
Check out my other articles on Medium. : https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y
My Instagram: https://rb.gy/gmvuy9
My Twitter: https://twitter.com/Machine01776819
If you’re preparing for coding/technical interviews: https://codinginterviewsmadesimple.substack.com/
Get a free stock on Robinhood: https://join.robinhood.com/fnud75
