Interesting Content in AI, Software, Business, and Tech- 9/20/2023

Content to help you keep up with Machine Learning, Deep Learning, Data Science, Software Engineering, Finance, Business, and more

7 min readSep 20, 2023

A lot of people reach out to me for reading recommendations. I figured I’d start sharing whatever AI Papers/Publications, interesting books, videos, etc I came across each week. Some will be technical, others not really. I will add whatever content I found really informative (and I remembered throughout the week). These won’t always be the most recent publications- just the ones I’m paying attention to this week. Without further ado, here are interesting readings/viewings for 9/20/2023. If you missed last week’s readings, you can find it here.

Important announcement: We started an AI Made Simple Subreddit. Come join us over here- https://www.reddit.com/r/AIMadeSimple/

Community Spotlight- Manish Gupta

Came across a fantastic YouTube channel by Manish Gupta recently. Manish makes detailed videos on topics in AI Research and Deep Learning. His videos are on the longer side, but they balance mathematical depth and intuition flawlessly. Would highly recommend giving his YouTube channel a spin. I found his channel through his video on RetNets, and it’s definitely worth checking out if you want to learn more.

If you’re doing interesting work and would like to be featured in the spotlight section, just drop your introduction in the comments/by reaching out to me. There are no rules- you could talk about a paper you’ve written, an interesting project you’ve worked on, some personal challenge you’re working on, ask me to promote your company/product, or anything else you consider important. The goal is to get to know you better, and possibly connect you with interesting people in our chocolate milk cult. No costs/obligations are attached.

Join 150K+ tech leaders and get insights on the most important ideas in AI straight to your inbox through my free newsletter- AI Made Simple

Highly Recommended

These are pieces that I feel are particularly well done. If you don’t have much time, make sure you atleast catch these works.

Vector Search with OpenAI Embeddings: Lucene Is All You Need

Given all the hype around Vector Databases, this is an interesting counterpoint.

We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ranking test collection. The main goal of our work is to challenge the prevailing narrative that a dedicated vector store is necessary to take advantage of recent advances in deep neural networks as applied to search. Quite the contrary, we show that hierarchical navigable small-world network (HNSW) indexes in Lucene are adequate to provide vector search capabilities in a standard bi-encoder architecture. This suggests that, from a simple cost-benefit analysis, there does not appear to be a compelling reason to introduce a dedicated vector store into a modern “AI stack” for search, since such applications have already received substantial investments in existing, widely deployed infrastructure.

Shoutout to Sebastian Raschka, PhD for this finding.

MANAGING AI/ML: How to build and deploy AI/ML systems

Dr. Chris Walton is a Senior Applied Science Manager at Amazon. He came on AI Made Simple to share his insights about AI Management. Highly recommend it if you want to setup AI Operations in your own organization.

In today’s article, Chris will be sharing his wisdom about building and deploying ML systems. Chris covers various crucial points: including the difference between traditional software and Machine Learning, the difference in managing science/oriented AI teams vs ML Engineering teams, how to effectively manage expectations, and the challenges that you would implementing AI based systems. I’m sure you will find this as insightful as I did.

Birds Aren’t Real? How a Conspiracy Takes Flight | Peter McIndoe | TED

Deep look on conspiracy theories, and how societal mocking of conspiracy theorists further drives extremism.

Peter McIndoe isn’t a fan of birds. In fact, he has a theory about them that might shock you. Listen along to this eye-opening talk as it takes a turn and makes a larger point about conspiracies, truth and belonging in divisive times.

Watch it

Hyperloop in 2023: Where Are They Now?

Hyperloops have been a “transportation of the future” for a while now. Adam looks into various hyperloop hyped companies of the last decade to see whether they lived up to any of the hype.

Watch it here.

BM25 : The Most Important Text Metric in Data Science

RitvikMath is one of my favorite channels to understand mathematical intuitions behind various formulae in data science, AI, and deep learning. His video on BM25 is a masterpiece.

Watch it

AI Papers/Writeups

LongNet: Scaling Transformers to 1,000,000,000 Tokens

Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between any two tokens in a sequence; 2) it can be served as a distributed trainer for extremely long sequences; 3) its dilated attention is a drop-in replacement for standard attention, which can be seamlessly integrated with the existing Transformer-based optimization. Experiments results demonstrate that LongNet yields strong performance on both long-sequence modeling and general language tasks. Our work opens up new possibilities for modeling very long sequences, e.g., treating a whole corpus or even the entire Internet as a sequence.

Read here

Language Modeling Is Compression

It has long been established that predictive models can be transformed into lossless compressors and vice versa. Incidentally, in recent years, the machine learning community has focused on training increasingly large and powerful self-supervised (language) models. Since these large language models exhibit impressive predictive capabilities, they are well-positioned to be strong compressors. In this work, we advocate for viewing the prediction problem through the lens of compression and evaluate the compression capabilities of large (foundation) models. We show that large language models are powerful general-purpose predictors and that the compression viewpoint provides novel insights into scaling laws, tokenization, and in-context learning. For example, Chinchilla 70B, while trained primarily on text, compresses ImageNet patches to 43.4% and LibriSpeech samples to 16.4% of their raw size, beating domain-specific compressors like PNG (58.5%) or FLAC (30.3%), respectively. Finally, we show that the prediction-compression equivalence allows us to use any compressor (like gzip) to build a conditional generative model.

Read here

Scaling the Instagram Explore recommendations system

AI plays an important role in what people see on Meta’s platforms. Every day, hundreds of millions of people visit Explore on Instagram to discover something new, making it one of the largest recommendation surfaces on Instagram.

To build a large-scale system capable of recommending the most relevant content to people in real time out of billions of available options, we’ve leveraged machine learning (ML) to introduce task specific domain-specific language (DSL) and a multi-stage approach to ranking.

As the system has continued to evolve, we’ve expanded our multi-stage ranking approach with several well-defined stages, each focusing on different objectives and algorithms.

How Our Deadliest Parasite Turned To The Dark Side

Around 10,000 years ago, somewhere in Africa, a microscopic parasite made a huge leap. With a little help from a mosquito, it left its animal host — probably a gorilla — and found its way to a new host: us.

Watch here

String of Beads Puzzle From The Manga Q.E.D.

This is from a Manga called Q.E.D. I thank Sparky from the Philippines for the suggestion! A string of beads divides circles of equal size into blue and orange areas. What is the orange area minus the blue area equal to?

Watch here

Why Spotify Keeps Losing Money

Analysis of Spotify’s profitability problem and the company’s recent efforts in turning the situation around. Despite being the world’s largest music streaming platform, Spotify has never made a single dime in profits so far. The company has never had a profitable year in its history, and over the years its losses has added up to over 4 billion euros. For a company that has not only disrupted the entire music industry but also completely revolutionised the way we consume music, one would reasonably expect Spotify to be drowning in profits. However, the reality is that Spotify has never been able to turn a profit, making it the only software company that generates over 10 billion euros in revenue each year to consistently lose money.

Watch here.

If you liked this article and wish to share it, please refer to the following guidelines.

If you find AI Made Simple useful and would like to support my writing- please consider getting a premium subscription to my sister publication Tech Made Simple below. Supporting gives you access to a lot more content and enables me to continue writing. You can use the button below for a special discount for readers of AI Made Simple, which will give you a premium subscription at 50% off forever. This will cost you 400 INR (5 USD) monthly or 4000 INR (50 USD) per year.

Support AI Made Simple