Interesting Content in AI, Software, Business, and Tech- 01/10/2024 [Updates]

Content to help you keep up with Machine Learning, Deep Learning, Data Science, Software Engineering, Finance, Business, and more

Jan 11, 2024

Hey, it’s Devansh 👋👋

In issues of Updates, I will share interesting content I came across. While the focus will be on AI and Tech, the ideas might range from business, philosophy, ethics, and much more. The goal is to share interesting content with y’all so that you can get a peek behind the scenes into my research process.

I put a lot of effort into creating work that is informative, useful, and independent from undue influence. If you’d like to support my writing, consider becoming a premium subscriber to my sister publication Tech Made Simple to support my crippling chocolate milk addiction. Use the button below for a lifetime 50% discount (5 USD/month, or 50 USD/year).

Help me buy chocolate milk

A lot of people reach out to me for reading recommendations. I figured I’d start sharing whatever AI Papers/Publications, interesting books, videos, etc I came across each week. Some will be technical, others not really. I will add whatever content I found really informative (and I remembered throughout the week). These won’t always be the most recent publications- just the ones I’m paying attention to this week. Without further ado, here are interesting readings/viewings for 01/10/2024. If you missed last week’s readings, you can find it here.

Reminder- We started an AI Made Simple Subreddit. Come join us over here- https://www.reddit.com/r/AIMadeSimple/. If you’d like to stay on top of community events and updates, join the discord for our cult here: https://discord.com/invite/EgrVtXSjYf.

Community Spotlight-Abhishek Gupta

Abhishek Gupta is part of the amazing education non-profit, Navgurukul, a group helping girls from underprivileged communities in India get a software education and place them in companies. Abhishek is leading a new venture, where they take on contracts to provide additional opportunities for Navgurukul students/graduates. If you're looking for cost-effective and socially conscious software services (or you just want to help out), consider reaching out to them.

If you're doing interesting work and would like to be featured in the spotlight section, just drop your introduction in the comments/by reaching out to me. There are no rules- you could talk about a paper you've written, an interesting project you've worked on, some personal challenge you're working on, ask me to promote your company/product, or anything else you consider important. The goal is to get to know you better, and possibly connect you with interesting people in our chocolate milk cult. No costs/obligations are attached.

PS- AI Made Simple is about to hit its 1st anniversary. We will be hosting an AMA (ask me anything) to celebrate this. You can ask your questions in the comments, message me on any of the social media links, reply to this email, or drop any questions you have in this anonymous Google Form- https://docs.google.com/forms/d/1RkIJ6CIO1w7K77tt0krCGMjxrVp1tAqFy5_O2FSv80E/.

Highly Recommended

These are pieces that I feel are particularly well done. If you don't have much time, make sure you at least catch these works.

AI Alignment Research Is More Science Fiction Than Science

I'll say this upfront- I disagree with a chunk of this piece. I think the concerns about AI actively engaging in deceptive alignment are misguided. Deceptive alignment is very real, but it is caused more by our inadequate systems of measurement and a generally poor capacity in humans to not know what we don't know.

That being said, this is an excellent article by Jurgen Gravestein. His article hits on one of the biggest problems (for alignment ) in modern society- we no longer have a strong overarching cultural narrative on what is right and wrong (certainly not when it comes to defining more or less correct). This allows us to be free and pursue our own goals, but this can cause all kinds of wrinkles in AI Alignment. Despite my disagreement with parts of this piece, I'd suggest reading it because it's good food for thought.

The findings show potential, however the experiment had many assumptions baked into it, which the researchers openly acknowledge. It assumes, for example, that we know exactly what values we want the AI to pursue in the first place!...Liberal society is built on the right to pursuit of happiness, but much of the muddiness of life comes from the fact that my idea of freedom can be very different from yours. Not only will humans want to use an AI system that is aligned with their personal values, they will want to use it to pursue their own goals. So, how do we make sure they don’t bite each other? And who gets to decide what values these AI systems have?

THE FUTURE WE SIMULATE IS THE ONE WE CREATE

Barak Epstein shared this masterpiece with me. This is a pretty interesting argument for the need to invest into High Performance Computing (HPC). To me, HPC is at the point now that ML was a few decades back, and what computing was like in its infancy. Investing in the right infrastructure would be a game-changer for society. HPC seems like an asymmetric bet to me- lot of upside, limited downside. Would love to know what you think.

Another reason, we think, why HPC isn’t a profitable business is because it deals in hard to quantify benefits and is not driving a worldwide advertising, search engine, and cloud computing machine that is at the heart of the Internet and therefore the habits of users and potential product buyers. And thus, HPC is a bit like machine learning back in the 1980s, when all of the groundwork was laid for success in the 2010s and beyond. Back then, AI researchers had the right algorithms for convolutional neural networks, but they did not have a lot of labeled or unlabeled data on which to train networks and they certainly did not have enough parallel computing power to build the networks in a reasonable timeframe. Similarly, the best HPC systems in the world can only simulate a rude approximation of anything for a reasonably long term, or a high fidelity approximation for a very short term, measured in picoseconds to seconds depending on what it is. We just don’t have enough compute to really simulate at the necessary scale.

An Overview of Text Summarization Methods from Simple to Complex

Our boy Logan Thorneloe has been on demon time with his recent ML oriented pieces. He plans to do more ML Engineering related pieces, so make sure you keep an eye on his newsletter.

I've been researching text summarization methods for a project I'm working on. Since the advent of Large Language Models (LLMs), LLMs have become a go-to solution, even making their way into the Google app with Bard. It's no wonder, given our era of information overload, that the ability to create concise, natural, and accurate summaries is super useful.

While LLMs are gaining popularity, my research has revealed that they are not always the optimal solution for summarizing information. Despite their prowess, there are numerous other efficient methods available.

I go over many of these methods below.

Easily Train a Specialized LLM: PEFT, LoRA, QLoRA, LLaMA-Adapter, and More

If you're looking for LLM and Deep Learning-related resources, Cameron R. Wolfe, Ph.D. is one of the best in the game. I've shared his work a few times, but I seriously can't recommend his newsletter enough.

Within this overview, we will learn about a popular solution to the issues outlined above—parameter-efficient finetuning. Instead of training the full model end-to-end, parameter-efficient finetuning leaves pretrained model weights fixed and only adapts a small number of task-specific parameters during finetuning. Such an approach drastically reduces memory overhead, simplifies the storage/deployment process, and allows us to finetune LLMs with more accessible hardware. Although the overview will include a many techniques (e.g., prefix tuning and adapter layers), our focus will be upon Low-Rank Adaptation (LoRA) [1], a simple and widely-used approach for efficiently finetuning LLMs.

Why Everything Is A Scam (Except For Scams)

Darin Soat makes top tier finance and business content. Y'all should definitely check his work out (we've shared the YouTube channel HowMoneyWorks a few times).

The reported Erin Griffith with the New York Times wrote about the hundreds of fake companies with fake business plans and fake users being created to attract venture capital investment. Firms like Sequoia, Andreesen Horowitz, and Accel had so much money that they couldn’t invest it into new ideas fast enough. This started to encourage hopeful founders to fudge their numbers a bit to get life changing investments. There were companies that used the money and did built out a real business, but there were other companies that had to keep on lying as more and more investors piled in. If you were a company founder trying to be completely honest about your business you would have found it much harder to get an audience with these investors to pitch your idea because you were competing with other founders who weren’t afraid to stretch the truth, or flat out lie, to tell the investors what they wanted to hear.

c3.ai, The Laughing Stock of A.I.

I caught a lot of flak for my skepticism around C3 AI and its viability when it's hype was at its peak. Looks like my concerns were well founded.

In this video we look at the enterprise artificial intelligence company c3.ai. Following its IPO in late 2020 it became one of the most hyped up software companies. This hype has increased with the general investor euphoria around all things A.I. related. But is c3.ai really the A.I. leader that it portrays itself to be?

Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Sebastian Raschka, PhD is an absolutely elite AI Researcher. Highly recommend everything he does, including this publication.

The correct use of model evaluation, model selection, and algorithm selection techniques is vital in academic machine learning research as well as in many industrial settings. This article reviews different techniques that can be used for each of these three subtasks and discusses the main advantages and disadvantages of each technique with references to theoretical and empirical studies. Further, recommendations are given to encourage best yet feasible practices in research and applications of machine learning. Common methods such as the holdout method for model evaluation and selection are covered, which are not recommended when working with small datasets. Different flavors of the bootstrap technique are introduced for estimating the uncertainty of performance estimates, as an alternative to confidence intervals via normal approximation if bootstrapping is computationally feasible. Common cross-validation techniques such as leave-one-out cross-validation and k-fold cross-validation are reviewed, the bias-variance trade-off for choosing k is discussed, and practical tips for the optimal choice of k are given based on empirical evidence. Different statistical tests for algorithm comparisons are presented, and strategies for dealing with multiple comparisons such as omnibus tests and multiple-comparison corrections are discussed. Finally, alternative methods for algorithm selection, such as the combined F-test 5x2 cross-validation and nested cross-validation, are recommended for comparing machine learning algorithms when datasets are small.

Inference Race To The Bottom - Make It Up On Volume?

A good look at the business side of LLMs by Dylan Patel and the rest of the SemiAnalysis crew.

For those keeping score, there will be a total of 11 firms in just a handful of months from now. It’s clear that pre-training of a GPT-3.5 caliber model has become completely commoditized. OpenAI is still the king of the hill with GPT-4, but that lead has been compressed significantly. While we believe that most of the long-term value will be captured by the highest-end models, it’s also clear that the next tier down in model quality and cost will enable a multi-billion-dollar niche in the marketplace, especially when fine-tuned.

But who can actually make money off these models if they are everywhere?

Data Waste

Very important read about the impact of data-driven infrastructures.

In this contribution, we highlight a phenomenon we call “data waste,” or the carbon emissions, natural resource extraction, production of waste, and other harmful environmental impacts directly or indirectly attributable to data-driven infrastructures. These include platform-based business models, the programming and use of AI systems, and blockchain-based technologies.We describe data infrastructures as infrastructures that reflect an ideology of permissionless innovation and tend toward monopolization. We explain how these models' platform and network-based structure and the permissionless incentives that characterize them shape their impact on the climate. We discuss how law and legal institutions have facilitated these developments and guide the way towards a systemic understanding of the factors contributing to data waste. We end with a call to frame the issue of data waste as a sociotechnical controversy, and issue that raises important questions about the manifestation of power within data-driven infrastructures and the need to imagine alternatives.

AI Content

GAIA: a benchmark for General AI Assistants

We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark.

Don’t Build AI Products The Way Everyone Else Is Doing It

Lost in the Middle: How Language Models Use Long Contexts

While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.

News 🆕 and Trends on TechBio

Artificial Intelligence Made Simple

Discussion about this post