Interesting Content in AI, Software, Business, and Tech- 11/22/2023 [Updates]
Content to help you keep up with Machine Learning, Deep Learning, Data Science, Software Engineering, Finance, Business, and more
Hey, it’s Devansh 👋👋
In issues of Updates, I will share interesting content I came across. While the focus will be on AI and Tech, the ideas might range from business, philosophy, ethics, and much more. The goal is to share interesting content with y’all so that you can get a peek behind the scenes into my research process.
If you’d like to support my writing, consider becoming a premium subscriber to my sister publication Tech Made Simple to support my crippling chocolate milk addiction. Use the button below for a lifetime 50% discount (5 USD/month, or 50 USD/year).
A lot of people reach out to me for reading recommendations. I figured I’d start sharing whatever AI Papers/Publications, interesting books, videos, etc I came across each week. Some will be technical, others not really. I will add whatever content I found really informative (and I remembered throughout the week). These won’t always be the most recent publications- just the ones I’m paying attention to this week. Without further ado, here are interesting readings/viewings for 11/22/2023. If you missed last week’s readings, you can find it here.
Reminder- We started an AI Made Simple Subreddit. Come join us over here- https://www.reddit.com/r/AIMadeSimple/
Community Spotlight- The Discord Group
For those of you not on Tech Made Simple, I wanted to drop a quick reminder- we’re planning meetups, networking events, and other cool IRL events. If you’d like to stay on top of things, join the discord for our cult here. Also, I am moving from Denver to NYC for the next few months. If you’re around the area, holla at your boy. Any introductions you can make would also be super appreciated.
If you're doing interesting work and would like to be featured in the spotlight section, just drop your introduction in the comments/by reaching out to me. There are no rules- you could talk about a paper you've written, an interesting project you've worked on, some personal challenge you're working on, ask me to promote your company/product, or anything else you consider important. The goal is to get to know you better, and possibly connect you with interesting people in our chocolate milk cult. No costs/obligations are attached.
Highly Recommended
These are pieces that I feel are particularly well done. If you don't have much time, make sure you at least catch these works.
Pizza Wars: The Exorcism of Papa John’s
The most important content I've shared so far. YouTuber Modern MBA (highly recommend subbing to his channel) did a phenomenal deep dive into Papa John's Pizza, covering both the strengths and weaknesses of strong founder-led companies. Very relevant given how many Tech Companies seem to be following a similar path of worshipping strong personalities (Musk, Altman, ...). Also, the way media organizations went out of their way to misrepresent the Papa John's CEO as a racist should be illegal. It's scary how easily they got away with their slander campaign and manufactured outrage, and how most people never tried to look beyond the headlines before starting a witch hunt.
A Data-Driven Look at the Rise of AI
Our main man, Andrew Gillies, shared a great writeup looking at the data around various points in AI. Some great insights (even though I'm not a huge fan of tracking Github stars and Commits to a project).
In the presentation, Viswanath argues that fast-moving incumbent technology companies are best positioned in the coming AI wave. But notes that in past technology waves new startups were founded just as technology emerged and far later once a new technology was well established. There isn’t a wrong time to build a disruptive technology company — if it’s a novel use of new technology. Uber emerged early in the life of the smartphone app stores but TikTok climbed to dominance many years later.
He shows how foundation models are open and closed in a wide variety of permutations. LlaMa 1 shared its training data but didn’t offer a commercial license; whereas, LlaMa 2 didn’t share its training data but opened up a partial commercial license.
GitHub stars on AI projects are down, indicating that the AI hype cycle may be waning. But AI commits to projects are going strong, showing that people are still hard at work coding. Meanwhile, ChatGPT usage can rise and fall when newer versions come out, drawing users back to the chatbot to try out fresh features.
FOUND: Foot Optimization with Uncertain Normals for Surface Deformation Using Synthetic Data
Given how many of you there are, I'm sure we have a few feet enjoyers over here. Do you look at AI Research, and feel disappointed by the lack of feet? Well, I gotchu. The following paper contains 3-D Geometery, Reconstruction, and Gen AI. And 50,000 photorealistic pictures of feet. Go have fun, my dear people of culture <3. (Real talk though- super worth reading the paper).
Surface reconstruction from multi-view images is a challenging task, with solutions often requiring a large number of sampled images with high overlap. We seek to develop a method for few-view reconstruction, for the case of the human foot. To solve this task, we must extract rich geometric cues from RGB images, before carefully fusing them into a final 3D object. Our FOUND approach tackles this, with 4 main contributions: (i) SynFoot, a synthetic dataset of 50,000 photorealistic foot images, paired with ground truth surface normals and keypoints; (ii) an uncertainty-aware surface normal predictor trained on our synthetic dataset; (iii) an optimization scheme for fitting a generative foot model to a series of images; and (iv) a benchmark dataset of calibrated images and high resolution ground truth geometry. We show that our normal predictor outperforms all off-the-shelf equivalents significantly on real images, and our optimization scheme outperforms state-of-the-art photogrammetry pipelines, especially for a few-view setting. We release our synthetic dataset and baseline 3D scans to the research community.
Lost in the Middle: How Language Models Use Long Contexts
Given how many people make a big deal about Context Length, this paper is worth a read.
While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
Artificial Intuition, not Artificial Intelligence
Barak Epstein shared a gem. A few articles back, I covered how LLMs don't exhibit thinking and understanding as we would define it. This article proposes an interesting alternative, that AI's operations are closer to intuition and not Intelligence. In terms of our psychology, AI is closer to the fast System 1 than the slower System 2. A good frame can radically improve performance, and this analogy is worth thinking about.
An LLM guesses-next-token by assembling good candidates from a hyperdimensional field of associations, their model. They don’t seem to do abstract reasoning. Reasoning has a deep, recursive shape and LLMs are computationally shallow. Token prediction executes a fixed number of computational steps, and returns. No deep recursion.
So, not reasoning, then. Perhaps more like intuition?
Why the world is mad | Kafka's The Trial
A fantastic look of how we can be our own hell if we tie into the absurdity of the world. Kafka was quite a thinker wasn't he?
Franz Kafka's The Trial is one of the great philosophical novels of the 20th century. It follows Joseph K, an upper-middle-class citizen of a bizarre society where anyone can be put on trial at any moment with no explanation. They will not be told what they are accused of or how to defend themselves. In short, K lives in a world that has gone mad. And The Trial shows how K deals with his arrest, his trial, and his eventual fate.
Albert Camus vs. Jean-Paul Sartre
Speaking of the absurd, hell, and great thinkers: here is a great look into Camus and Sartre, how their lives influenced their philosophies, and why they fell apart. the most ironic part of the whole video: the aristocratic Sartre spent his life championing communism from his Parisian Ivory Tower while denouncing Camus. Nevertheless, both are great philosophers with sharp insight and this video is great for contextualizing their thought.
How Your Brain Organizes Information
The video covers some fantastic neuroscience research into the structure of our brain and how it handles information (and how it leads to generalization). Super duper recommended- copying designs from neuroscience will lead to a breakthrough in AI.
AI Content
Since the discovery of the quasicrystal, approximately 100 stable quasicrystals are identified. To date, the existence of quasicrystals is verified using transmission electron microscopy; however, this technique requires significantly more elaboration than rapid and automatic powder X-ray diffraction. Therefore, to facilitate the search for novel quasicrystals, developing a rapid technique for phase-identification from powder diffraction patterns is desirable. This paper reports the identification of a new Al–Si–Ru quasicrystal using deep learning technologies from multiphase powder patterns, from which it is difficult to discriminate the presence of quasicrystalline phases even for well-trained human experts. Deep neural networks trained with artificially generated multiphase powder patterns determine the presence of quasicrystals with an accuracy >92% from actual powder patterns. Specifically, 440 powder patterns are screened using the trained classifier, from which the Al–Si–Ru quasicrystal is identified. This study demonstrates an excellent potential of deep learning to identify an unknown phase of a targeted structure from powder patterns even when existing in a multiphase sample.
Identifying keystone species in microbial communities using deep learning
Previous studies suggested that microbial communities can harbour keystone species whose removal can cause a dramatic shift in microbiome structure and functioning. Yet, an efficient method to systematically identify keystone species in microbial communities is still lacking. Here we propose a data-driven keystone species identification (DKI) framework based on deep learning to resolve this challenge. Our key idea is to implicitly learn the assembly rules of microbial communities from a particular habitat by training a deep-learning model using microbiome samples collected from this habitat. The well-trained deep-learning model enables us to quantify the community-specific keystoneness of each species in any microbiome sample from this habitat by conducting a thought experiment on species removal. We systematically validated this DKI framework using synthetic data and applied DKI to analyse real data. We found that those taxa with high median keystoneness across different communities display strong community specificity. The presented DKI framework demonstrates the power of machine learning in tackling a fundamental problem in community ecology, paving the way for the data-driven management of complex microbial communities.
Removing RLHF Protections in GPT-4 via Fine-Tuning
As large language models (LLMs) have increased in their capabilities, so does their potential for dual use. To reduce harmful outputs, produces and vendors of LLMs have used reinforcement learning with human feedback (RLHF). In tandem, LLM vendors have been increasingly enabling fine-tuning of their most powerful models. However, concurrent work has shown that fine-tuning can remove RLHF protections. We may expect that the most powerful models currently available (GPT-4) are less susceptible to fine-tuning attacks.
In this work, we show the contrary: fine-tuning allows attackers to remove RLHF protections with as few as 340 examples and a 95% success rate. These training examples can be automatically generated with weaker models. We further show that removing RLHF protections does not decrease usefulness on non-censored outputs, providing evidence that our fine-tuning strategy does not decrease usefulness despite using weaker models to generate training data. Our results show the need for further research on protections on LLMs.
Software Content
How Shopify’s engineering improved database writes by 50% with ULID
When and how to run group-targeted A/B tests
Run group-targeted experiments when:
You want to measure the impact of a change on an entire group of users.
The change in how an individual user uses your product significantly impacts the behavior of other users.
If you liked this article and wish to share it, please refer to the following guidelines.
Reach out to me
Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.
Small Snippets about Tech, AI and Machine Learning over here
AI Newsletter- https://artificialintelligencemadesimple.substack.com/
My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/
Check out my other articles on Medium. : https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y
My Instagram: https://rb.gy/gmvuy9
My Twitter: https://twitter.com/Machine01776819