Transcript: The AI-powered Era of Scientific Discovery Is Here

The transcript of AI & I with Bradley Love is below for paying subscribers.

Timestamps

1. Introduction: 00:01:00
2. The motivations behind building a LLM that can predict the future: 00:01:58
3. How studying the brain can solve the AI revolution’s energy problem: 00:11:14
4. Dr. Love and his team have developed a new way to prompt AI: 00:13:32
5. Dan’s take on how AI is changing science: 00:18:27
6. Why clean scientific explanations are a thing of the past: 00:22:54
7. How our understanding of explanations will evolve: 00:29:49
8. Why Dr. Love thinks the way we do scientific research is flawed: 00:37:31
9. Why humans are drawn to simple explanations: 00:40:42
10. How Dr. Love would rebuild the field of science: 00:45:03

Transcript

Dan Shipper (00:01:00)

Bradley, welcome to the show. 

Bradley Love (00:01:02)

Yeah, thanks, Dan. Thanks for having me here.

Dan Shipper (00:01:03)

Yeah, I'm super excited to have you. So, for people who don't know, you are the professor of cognitive and decision sciences in experimental psychology at University College London, and you are also one of the main builders of a large language model that is focused on helping people do better neuroscience research, called BrainGPT. And I'm super psyched to have you on. This is going to be a slightly nontraditional episode because I think we're going to go deep into science and using AI for science, and sort of how science might change as a result of AI. But, yeah, super psyched to have you.

Bradley Love (00:01:42)

No, I'm excited too. And like you hint at, I think there are larger ramifications that go beyond neuroscience, which we developed this model for, that should affect all your listeners.

Dan Shipper (00:01:53)

So, let's get started with BrainGPT. Tell us about what it is.

Bradley Love (00:01:57)

Sure. I mean, first maybe, I’ll give some of the motivation. So, like probably a lot of your listeners, I was never really a big tool development person, but I just saw how science was going and it's exponentially increasing literature. And we just can't keep up. It's just not really a human-readable literature.

And so just kind of getting a grip on it, it seems like we need tools to do that. And people are making all kinds of tools, particularly using large language models, but we kind of had a different take on it. So, there's a lot of great work that we call backward-looking, which is not to be pejorative. It's work involving summarization of the scientific literature, kind of writing instant reviews or almost meta-analyses in cases, and that's great and that's valuable. But we want to focus more on what I think is really important in science, namely prediction, what we call forward-looking, and can we actually predict the outcome of studies before they happen?

And so we just really have this project and I'm happy to dive in as far as you like, but we wanted to see can large language models—both just off-the-shelf models, but also models that we fine-tune on 20 years of the neuroscience scientific literature—can they actually predict the results of neuroscience studies or experiments better than human experts like professors of neuroscience? There's a lot to say, but in short, they can. They're a lot better at it.

Dan Shipper (00:03:28)

That's fascinating. Wait. So just to go back to the original motivation, it sounds like the first thing was, okay, there's way too much information. There's way too much science being done for anyone to keep up. But it sounds like the thing that you're building, BrainGPT, isn't necessarily about summarizing the research. It's about predicting what future research might hold. So, what's the connection between not being able to keep up with the literature and future research predictions?

Bradley Love (00:03:52)

Sure. That's a great question. Because I think to predict the future, you have to have some understanding of the past, so not necessarily a nice, clean text that summarizes, but a model that could draw on thousands of findings through different literatures at different levels, because neuroscience is multi-level. It has everything from psychology and behavior all the way down to really low-level cellular, molecular findings, things involving DNA and so forth. And so no one could really draw on all that information, but it might be that biology is really messy. It's not how computer scientists create these abstraction layers. We have the hardware, the software, and so forth. 

So to make a prediction, you might very well need to draw on all that information. And why do you want to make a prediction? Because I mean, well, first, there's so many uses in science for this. So you may want to run a more informative study. So, if BrainGPT predicts that your study is going to work out 99.9 percent certain as you expect, there's really no information gain in that. There's no reason to run the study. On the other hand if you, the system says, oh no, this is unlikely to give the pattern results you expect, but you have an intuition that the literature has gone off course and there's a systematic bias, that the same bias is affecting what BrainGPT is trained on, then, in some sense, you're trailblazing, you're making a really impactful discovery. So really everything: getting into replication is a huge issue in science. Most findings just don't actually replicate. And so I think we could, in the very near future, use systems like this to kind of get a handle on what's true, what we could count on, what next step we should take in scientific discovery.

Dan Shipper (00:05:40)

When you're training it, how are you taking into account the fact that p-hacking occurs and maybe the literature is somewhat damaged. Are they training it on literature? How do you filter that out, I guess?

Bradley Love (00:05:53)

Yeah. I mean, right now, not so much, but that's something that's definitely of interest. So the way I see it, I don't think scientists see it this way. I mean, I'm a scientist, but most people think of papers as individual contributions or discoveries, but I think of each contribution as—even if it's not p-hacked—is really flawed, noisy, incomplete. You have this tapestry of thousands of papers, and so hopefully, if you just aggregate— It's almost like in machine learning, if you do an ensemble, it's an ensemble of everybody, hopefully has the signal in the correct direction. I mean, that's why I raised the possibility before that there could be systematic biases or issues, but I think a lot of the problems just really come down to underpowered studies—statistically. And that opens the door to p-hacking or just a careless mistake. And so hopefully there's not that many systematic flaws. And if we could just not start reasoning about individual papers, but about thousands of papers at once, I think we'll get the signal.

Dan Shipper (00:07:00)

That's really interesting. And you said it predicts better than neuroscience experts which studies are going to work and which won't. What are the boundaries around those kinds of predictions? What is it good at predicting?

Bradley Love (00:07:13)

Yeah, yeah. So the way we tested it, kind of taking our cue from what people do in computer science and machine learning where they make benchmarks—like, the ImageNet benchmark was really critical to developing computer vision models. So we made our own brain bench model. And what we did is we looked at the Journal of Neuroscience, which is kind of a standard, well-respected journal in neuroscience. And the reason we chose it is because it gets to your question: it really covers all of neuroscience. There's five subsections to it, and it goes from everything from cognitive behavior, pretty high-level stuff, to cellular molecular systems to developmental psychiatry-type stuff. And so our benchmark would have these five subscores. And what we did is we took recent publications from this journal that are unlikely to have leaked into the training sets of models, and we trained some models from scratch where we know what they're trained on, but what we did is we just subtly altered these abstracts. So a scientific abstract is—to back up for readers that haven't read a scientific paper—there tends to be a structure, where there's a bit of background at first a couple sentences. Then, the method—what kind of experiment was run, and then the result. And so we just altered the result in very subtle ways, keeping the linguistic flow. So if it was like, blah, blah, blah, interior hippocampus, we change it to blah, blah, blah, posterior hippocampus. Or something was this brain activity increases, we had it decrease. And so there was a multiple choice, basically with two options, and we tested, basically, neuroscientists and a whole slew of large language models, including some that we fine-tuned on neuroscience literature and just compared their accuracy.

Dan Shipper (00:09:07)

That's really interesting. And I guess, for you, what do you think is the current sort of frontier for neuroscience? What are the interesting problems right now? And how does BrainGPT fit into that?

Bradley Love (00:09:23)

Yeah. I mean, there's so many things. I mean, in some ways neuroscience has been around for a while, but in other ways, it's still a really young discipline, say, compared to physics. And some people could even say it's almost pre- sort of, there's no standard model or anything like that. And in physics—

Dan Shipper (00:09:42)

Pre-paradigm.

Bradley Love (00:09:44)

Yeah, exactly. That's what I was grasping for. Thank you. So a lot of what I do in research—I do empirical research modeling. But I feel like a lot of time, I'm just trying to figure out what the question is or how to frame things, but there's so many questions and some of them are relevant. It goes from really high-level stuff to really low-level stuff. We have synaptic change and that's the basis of learning, but even that's up for grabs and how that works, even at the very low-level, how is a memory encoded in the brain? Is it in sort of those synapses, the weights, as we do in neural networks? Is it something more internal to the cell? How does error propagate from learning? So that'd be relevant to deep learning. Does the brain do gradient descent? Does it do something else? And then high-level stuff like when we understand space is that the basis for higher-level mental concepts like freedom, justice, or just chair, or is it just a general learning mechanism? So there's all these issues that go from very low-level to high-level. It's such a huge field. I can't really say there's one issue, and I kind of gravitate towards the ones that might have a bleed over or transfer into AI machine learning personally.

Dan Shipper (00:11:01)

And what are the ones that you think are the most bleeding over into AI and machine learning?

Bradley Love (00:11:06)

Yeah, well, one that I don't work on that seems like an obvious candidate is just transferring power consumption ideas. So modern GPUs are amazing and they power transformers, which power the AI revolution and other technologies. But, of course, these data centers are going up every day and it's stressing the grid, there's carbon impacts and so forth. Whereas our brains are doing a lot of computation, but I guess you just have to eat a sandwich or something and it's a lot less power consumption. So that's sort of a neuromorphic computing application.

Dan Shipper (00:11:48)

I'm just excited for a world where Microsoft data centers, they just like ordering a big load of Subway.

Bradley Love (00:11:56)

Yeah, exactly. It’s like a reverse matrix or something—just give it a salad, it doesn't use us as batteries. But yeah, that's really funny. Yeah, but possible. I mean, there's so many things. I mean, I'm really interested in the higher-level stuff. So I still think where people are better is like, somehow we have some tricks for how we represent the world and tasks and promote generalization. And I mean, in some sense, why everyone's so excited about large language models is that they're base or foundational in some sense that you could apply them to other tasks, not just the test they're trained on. Whereas previous generations of machine learning models, even the great, convolutional models that somewhat cracked object recognition or AlphaGo doing its games and so forth. Those are specialized models and I think people are still the kings of that flexibility. So there's probably some secret sauce to gain still from humans and how we represent situations and generalize and link things up.

Dan Shipper (00:13:03)

That's really interesting. Wait, can we see a demo of it? Do you have a way to use BrainGPT that we can look at?