Interesting Content in AI, Software, Business, and Tech- 5/31/2023
A possible new type of content on this newsletter
Hey all,
A lot of people have reached out to me to ask for paper recommendations/other content that I find interesting. So last week, I thought I’d try a new kind of format. A lot of you are subscribed to both Tech Made Simple and AI Made Simple, so I didn’t want to be in your inbox all day. This is why I wrote a LinkedIn article with interesting papers, articles, videos, etc I came across. That got some positive feedback, so figured I’d see if it was content you wanted as part of the newsletter. I will leave a poll at the end of this. Let me know if you’d want this to be a regular feature in this newsletter, or if you’d prefer that I leave this content for LinkedIn/Medium. Make sure you vote, because it will impact the future content in this newsletter.
Copying the article as is for you to judge. If you like the idea but want me to change something, or have any other feedback for me, I’d love to hear it. You know how to reach out.
A lot of people reach out to me for reading recommendations. I figured I'd start sharing whatever AI Papers/Publications, interesting books, videos, etc I came across each week. Some will be technical, others not really. I will add whatever content I found really informative (and I remembered through the week). These won't always be the most recent publications- just the ones I'm paying attention to this week. Without further ado, here are interesting readings/viewings for 5/31/2023. If you missed last week's readings, you can find it here.
AI Papers/Writeups
1) The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python
Link- https://arxiv.org/abs/2305.15507
Abstract- Large Language Models (LLMs) have successfully been applied to code generation tasks, raising the question of how well these models understand programming. Typical programming languages have invariances and equivariances in their semantics that human programmers intuitively understand and exploit, such as the (near) invariance to the renaming of identifiers. We show that LLMs not only fail to properly generate correct Python code when default function names are swapped, but some of them even become more confident in their incorrect predictions as the model size increases, an instance of the recently discovered phenomenon of Inverse Scaling, which runs contrary to the commonly observed trend of increasing prediction quality with increasing model size. Our findings indicate that, despite their astonishing typical-case performance, LLMs still lack a deep, abstract understanding of the content they manipulate, making them unsuitable for tasks that statistically deviate from their training data, and that mere scaling is not enough to achieve such capability.
Authors- Antonio Valerio Miceli Barone, Fazl Barez, Ioannis Konstas, Shay Cohen.
2) Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Paper: https://arxiv.org/abs/2305.10601
Abstract: Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, Tree of Thoughts (ToT), which generalizes over the popular Chain of Thought approach to prompting language models, and enables exploration over coherent units of text (thoughts) that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices. Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved 4% of tasks, our method achieved a success rate of 74%.
Authors- Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan
My recommendation when tackling this paper would be to watch Yannic Kilcher's excellent breakdown over here-
3) Is Avoiding Extinction from AI Really an Urgent Priority?
Link- https://aisnakeoil.substack.com/p/is-avoiding-extinction-from-ai-really
Authors- Seth Lazar, Jeremy Howard, and
.Great writeup on the dangers of focusing on existential risks of AI and forgetting the other dangers. Love this particular passage-
"Indeed, focusing on this particular threat might exacerbate the more likely risks. The history of technology to date suggests that the greatest risks come not from technology itself, but from the people who control the technology using it to accumulate power and wealth. The AI industry leaders who have signed this statement are precisely the people best positioned to do just that. And in calling for regulations to address the risks of future rogue AI systems, they have proposed interventions that would further cement their power. We should be wary of Prometheans who want to both profit from bringing the people fire, and be trusted as the firefighters."
4) The AI Healthcare Report: 5/26/23
Link- https://www.linkedin.com/pulse/ai-healthcare-report-52623-dylan-reid-moskowitz-/
Good resource if you'd like to track some of the advancements in the space of legislation for AI in healthcare.
Author- Dylan Reid(Moskowitz)
5) A rant about why technologists are bad at predictions
https://twitter.com/KevinAFischer/status/1662853371118641154
Hinton infamously predicted that AI would replace radiologists (not dissimilar to how Musk promised Full Self-Driving by 2020). And neither prediction worked out. So why do prominent technologists get predictions so wrong?
Kevin Fischer has a great thread on why the Radiologists never got replaced, and why so many predictions on AI have been off. One of my favorite quotes is- "thinkers have a pattern where they are so divorced from implementation details that applications seem trivial, when in reality, the small details are exactly where value accrues."
6) What they Don't tell you about A.I. Jobs Disruption?
Link- https://aisupremacy.substack.com/p/what-they-dont-tell-you-about-ai
Much has been made about the hype of AI replacing people's jobs. While most of it is hype, there are some jobs - think typists, telemarketers, etc, which have a higher likelihood of being replaced. Unfortunately, much of the work being done here is female-dominated, making them vulnerable to replacement. This has the potential to deepen gender inequality
Author- Michael Spencer
7) LIMA: Less Is More for Alignment
Link- https://arxiv.org/abs/2305.11206
Abstract- Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.
Found this paper on Davis Blalock's amazing newsletter over here (or click below).
Authors- Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy
Other interesting Reads
1) Unlocking the Potential of Hybrid Work
Abi's newsletter delivers hit after hit for software engineering and tech. His most recent writeup on hybrid work is another certified classic if you're into leadership
Link- https://newsletter.abinoda.com/p/hybrid-work-productivity
Author-
2) Native language shapes brain wiring
Scientists at the Max Planck Institute for Human Cognitive and Brain Sciences in Leipzig have found evidence that the language we speak shapes the connectivity in our brains that may underlie the way we think. With the help of magnetic resonance tomography, they looked deep into the brains of native German and Arabic speakers and discovered differences in the wiring of the language regions in the brain.
Link- https://www.mpg.de/20008844/our-native-language-shapes-the-brain-wiring
Authors- Max Plank Institute Berlin
Cool Vids-
Low-rank Adaption of Large Language Models: Explaining the Key Concepts Behind LoRA- Chris Alexiuk
Don't Underestimate The German Economy | Economics Explained-
Why Julian Nagelsmann is the perfect Real Madrid manager- Tifo Football
I'll catch y'all with more of these next week. In the meanwhile, if you'd like to find me, here are my social links-
This was how all my posts will look. Do you think these will be useful?
Peace <3
Devansh