"I want you to also consider what has been a growing problem with AI Research in the last few years- the reproducibility crisis. Because of bad incentives, the increasing computational costs of running ML Research, and the increasing volume of AI papers- the field has been flooded by papers and claims that can’t be reproduced and verified. It’s hard to fact-check Open AI’s claims when matching their training would cost you millions. LLMs make this problem much worse. Not only is training and testing on them very expensive, they are also very fragile, and we don’t really know all that we don’t know about them. Not the winning combination for confident and comprehensive research.". That is why my goal as a scientist and researcher in AI will be to have an internal impact, improve and deploy products, help create new architectures and systems, and make a lot of money in the process, all while doing good in the world. However, I am not interested in the "publish or perish" mentality, and I actually see the OpenAI stance (that as AI improves, their secretiveness will increase) as good and reasonable. Am I a "bad scientist" for thinking like this? I just cannot see how you can genuinely do good research and good science while managing all these externalities (without burning out) of publishing or perishing and while considering potential risks of IP theft and avoiding privacy scandals (nightmares). When you look at the most successful researchers and scientists in history, they have few publications, not that many.
We should really regulate the amount of material that gets published and channel funds to fewer, larger entities. In return, these entities would hire more researchers and scientists to do guided work. For sustainable science and research to truly be sustainable and advance at a reasonable pace, the time of 'just for curiosity research' has passed. We need everyone working towards ideas that work and make a great impact together. That is why I also fundamentally disagree with the Nobel Prize and similar prizes, because they result in all the people involved, except for one or a few persons, getting all the credit.
That's an interesting view point. While I do agree with some of your sentiments, I would urge you to look at the Open Source Movement as a counter-point to "We should really regulate the amount of material that gets published and channel funds to fewer, larger entities. In return, these entities would hire more researchers and scientists to do guided work." OSS has been the key driver of growth an movement both within AI and Tech since it enables everyone to contribute in more diverse ways. This is not something that centralized organizations can replicate.
Two reads that I would love to get your thought on:
I do think we need to get better with designing incentives for publications. Focus has to be on quality, not quantity. You're very correct, "publish or perish" is a huge problem. Will likely do a post on improving this broken replication crisis.
Reading back through this again and have some thoughts:
Can you elaborate more on this? "just don’t give it the capability to do that to begin with". I'm guessing this is a simplified way of saying: If you don't want it to be able to expose a certain piece of data, don't train it on that piece of data but just making sure I understand.
I find alignment super interesting because I don't think anyone (at the commercial scale at least) has done it well yet. You have examples like this where privacy is clearly an issue. There are also examples like Bard where it's doesn't seem to leak private info, but it also hinders the core functionality because it constantly tells me it doesn't have access to things of mine it should have access to. As you kind of touched on toward the end, it makes me wonder how alignment-as-a-service will play out. Will it be helpful? Will it work? Will it scale?
The memorization definition makes me realize just how much we don't understand about evaluating LLMs. I feel like there should be a better method of quantifying extracting training data, but I can't think of one myself.
The DeepMind research on extracting training data from ChatGPT is a real eye-opener. It challenges our assumptions about AI security and the effectiveness of alignment in preventing data leaks. This discovery is a reminder that in AI development, sometimes the simplest solution—limiting certain capabilities from the start—is more effective than trying to fine-tune our way out of potential problems. The nuances in AI behavior, especially in large language models, underscore the importance of a cautious and critical approach in AI research and application.
"I want you to also consider what has been a growing problem with AI Research in the last few years- the reproducibility crisis. Because of bad incentives, the increasing computational costs of running ML Research, and the increasing volume of AI papers- the field has been flooded by papers and claims that can’t be reproduced and verified. It’s hard to fact-check Open AI’s claims when matching their training would cost you millions. LLMs make this problem much worse. Not only is training and testing on them very expensive, they are also very fragile, and we don’t really know all that we don’t know about them. Not the winning combination for confident and comprehensive research.". That is why my goal as a scientist and researcher in AI will be to have an internal impact, improve and deploy products, help create new architectures and systems, and make a lot of money in the process, all while doing good in the world. However, I am not interested in the "publish or perish" mentality, and I actually see the OpenAI stance (that as AI improves, their secretiveness will increase) as good and reasonable. Am I a "bad scientist" for thinking like this? I just cannot see how you can genuinely do good research and good science while managing all these externalities (without burning out) of publishing or perishing and while considering potential risks of IP theft and avoiding privacy scandals (nightmares). When you look at the most successful researchers and scientists in history, they have few publications, not that many.
We should really regulate the amount of material that gets published and channel funds to fewer, larger entities. In return, these entities would hire more researchers and scientists to do guided work. For sustainable science and research to truly be sustainable and advance at a reasonable pace, the time of 'just for curiosity research' has passed. We need everyone working towards ideas that work and make a great impact together. That is why I also fundamentally disagree with the Nobel Prize and similar prizes, because they result in all the people involved, except for one or a few persons, getting all the credit.
That's an interesting view point. While I do agree with some of your sentiments, I would urge you to look at the Open Source Movement as a counter-point to "We should really regulate the amount of material that gets published and channel funds to fewer, larger entities. In return, these entities would hire more researchers and scientists to do guided work." OSS has been the key driver of growth an movement both within AI and Tech since it enables everyone to contribute in more diverse ways. This is not something that centralized organizations can replicate.
Two reads that I would love to get your thought on:
https://artificialintelligencemadesimple.substack.com/p/unpacking-the-financial-incentives
https://codinginterviewsmadesimple.substack.com/p/what-googles-leaked-letter-tells
I do think we need to get better with designing incentives for publications. Focus has to be on quality, not quantity. You're very correct, "publish or perish" is a huge problem. Will likely do a post on improving this broken replication crisis.
Reading back through this again and have some thoughts:
Can you elaborate more on this? "just don’t give it the capability to do that to begin with". I'm guessing this is a simplified way of saying: If you don't want it to be able to expose a certain piece of data, don't train it on that piece of data but just making sure I understand.
I find alignment super interesting because I don't think anyone (at the commercial scale at least) has done it well yet. You have examples like this where privacy is clearly an issue. There are also examples like Bard where it's doesn't seem to leak private info, but it also hinders the core functionality because it constantly tells me it doesn't have access to things of mine it should have access to. As you kind of touched on toward the end, it makes me wonder how alignment-as-a-service will play out. Will it be helpful? Will it work? Will it scale?
The memorization definition makes me realize just how much we don't understand about evaluating LLMs. I feel like there should be a better method of quantifying extracting training data, but I can't think of one myself.
The DeepMind research on extracting training data from ChatGPT is a real eye-opener. It challenges our assumptions about AI security and the effectiveness of alignment in preventing data leaks. This discovery is a reminder that in AI development, sometimes the simplest solution—limiting certain capabilities from the start—is more effective than trying to fine-tune our way out of potential problems. The nuances in AI behavior, especially in large language models, underscore the importance of a cautious and critical approach in AI research and application.
ChatGPT reply if I have ever seen one
Lolz. I get comments like this from time to time. I wonder why this happens.
Yep
Doing lots of trial and error to break through a piece of software. Why does this sound familiar!
That’s traditional hacking.
The problem with this piece of software is we do not know how it works.
Patch it too much and it might lose some of its previous magical abilities.