Generative AI is getting exceptionally good at automating tasks and processing information. And that’s making life easier for professionals across the board, from business managers to university professors.
But for researchers like Erzhuo Shao, a PhD student at the Kellogg Center for Science of Science and Innovation (CSSI), most of the current AI models are not integrated with scientific resources in a way that would help them conduct their work at the highest level.
This shortcoming motivated Shao to collaborate with Kellogg CSSI research assistant professor Yifan Qian and Dashun Wang, the Kellogg Chair of Technology and a professor of management and organizations, to develop a prototype AI model—called SciSciGPT—specifically to support the work of researchers in their field.
After testing their AI model, they found that it performed a wide range of research tasks—from statistical analysis to data visualization—much more efficiently than human researchers did while also producing higher-quality work. Yet perhaps more impressively, the AI model demonstrated a capacity to answer, in a matter of minutes, critical, multifaceted questions that even senior researchers would need hours, if not days, to resolve.
With such a powerful AI model at their disposal, “people should use the tool very thoughtfully,” Qian says. Researchers can use the technology to significantly boost the quality and speed of their work but should be careful not to sacrifice the human element in science research and discovery.
“We consider SciSciGPT as an augmentation, not a replacement for human researchers,” adds Shao. “We see it as a way to save researchers time and unleash their creativity.”
A virtual research team
SciSciGPT is a multilayered AI system.
Its backbone is a large language model (LLM) that allows SciSciGPT to function as a general-purpose chatbot that can answer questions, summarize text, write code, and reason on tasks like idea generation. Built upon this foundation is a collection of five AI agents, each of which has a distinct role and assignments that align with a core part of a typical research workflow:
- The Research Manager orchestrates the workflow, translating user questions into tasks that it can then delegate to the other agents;
- The Literature Specialist searches for and organizes relevant information;
- The Database Specialist handles data processing, extraction, and transformation;
- The Analytics Specialist does statistical analysis and modeling as well as data visualization;
- The Evaluation Specialist vets the quality and accuracy of the other agents’ work.
SciSciGPT is also loaded with a massive database (SciSciNet) on the science of science, which is a field of study that centers on fundamental questions about how science research and innovation occur and progress.
The combination of these three layers—an LLM, the AI agents, and the database—is what enables SciSciGPT to perform tasks and reason in a way that is particularly helpful for science-of-science researchers.
“SciSciGPT turns an LLM into a research system,” Shao says. “It’s a functioning layer of knowledge, data, and methodology layered upon an existing large language model.”
Answering a canonical question
The team, which also includes Northwestern colleagues Yifan Wang, Zhenyu Pan, and Han Liu, demonstrated SciSciGPT’s abilities in a series of case studies. Each case study featured a different type of work that science researchers frequently undertake.
For example, in one of the case studies, the team asked SciSciGPT a seemingly simple question: What does scientific collaboration look like among Ivy League universities?
For a human researcher to explore this question through conventional methods, they would need to identify all of the publications of researchers at Ivy League universities and filter for those that involve collaborations across schools, write scripts to query and draw information from the data, and use network-analysis tools to capture the relationships—among many other tasks. This all could take many hours to complete.
On the other hand, when the team presented SciSciGPT with this prompt, the Research Manager automatically decomposed it into various tasks for the Literature, Data, and Analytics Specialists to handle, with the Evaluation Specialist assessing the agents’ work at every step.
SciSciGPT successfully completed the whole process within a few minutes, including creating a final visualization of the collaboration patterns across Ivy League universities. And unlike a typical LLM, or even most humans, the AI model clearly detailed each of its steps and all of the mistakes it made along the way.
“SciSciGPT will meticulously document everything—all the errors it makes in the process—as it tries to correct itself,” Qian explains. “It is actually, from this perspective, much more transparent than when you collaborate with a human.”
The team followed up the case studies with a pilot study comparing the ability of SciSciGPT with that of three human researchers, who had different levels of expertise, on the same set of common research tasks.
On average, SciSciGPT’s output was better by every measure—overall effectiveness, technical soundness, depth of analysis, visualization quality, and documentation clarity—according to a panel of postdoctoral researchers. The quality of the AI model’s work throughout the process also was considered “stronger,” and it completed most of the tasks about ten times faster than humans.
Shifting the research paradigm
Though the team acknowledges the need for broader evaluation, the early results are promising—and the potential applications vast.
Researchers could leverage SciSciGPT to automate time-consuming low-level tasks so that they could conduct their work more effectively and have more time for high-level tasks. SciSciGPT also helps lower technical barriers to research, paving the way for nonexperts such as policymakers or even the general public to engage with crucial data.
These effects have the potential to “reshape how researchers choose what questions to ask, and how they collaborate,” Qian says. “When we combine data with the new AI tool, we can really start to understand and study many new questions that prior scholars wouldn’t have had the luxury to do even 10 years ago.”
Of note, SciSciGPT is both open source and interchangeable on multiple fronts.
Though SciSciGPT currently relies on Anthropic’s LLM, Claude, users can modify it to use other LLMs, like ChatGPT, instead. They can also swap out the science-of-science database with data from other disciplines.
“The beautiful part of SciSciGPT,” Qian says, “is that people can adapt this kind of environment to their own domain. So, it’s not only science-of-science researchers but people in other areas who can go to this practical tool and start using it right now.”
A collaborator, not a competitor
Despite these advantages, not everyone is entirely comfortable with the idea of depending on AI tools for their work.
After three expert researchers trialed SciSciGPT, they expressed a general concern about being able to trust the AI model, even though they all agreed it was valuable for research tasks. As one expert said, “I feel uncomfortable trusting something not generated by myself. As a researcher, I’m responsible for all mistakes.”
That’s a natural response to a rapidly evolving technology, where there are new AI applications being released basically every day, Shao notes. “So we plan for SciSciGPT to be a research assistant and collaborator to boost researchers’ productivity, not a replacement,” he says.
“AI agents promise a golden era of research, but they also bring real and significant risks,” adds Wang, who also serves as director of CSSI and the Northwestern Innovation Institute, and as codirector of the Ryan Institute on Complexity. “Domain-grounded AI collaborators like SciSciGPT, for example, have the potential to dramatically accelerate discovery while preserving key human elements. Getting that balance right is one of the defining challenges for science in the years ahead.”