Large language models (LLMs) are reshaping how people seek and produce technical knowledge. To evaluate the nature of knowledge where human expertise remains essential, we find a natural testbed in the largest technical Question and Answer community, StackOverflow. Each question asked in the community carries tags that mark the knowledge domains it draws upon. We term the unique combinations of these knowledge domains (i.e., unique sets of tags) a “specialization” that characterizes the skill set needed to answer that question. Following ChatGPT’s public release in November 2022, total question volume on StackOverflow fell sharply—a widely noticed trend. Yet beneath this decline, we discovered a notable trend: the proportion of novel questions that require stitching together knowledge domains in unprecedented ways rose significantly. More interestingly, we find that this surge in novelty emerges mainly from new recombinations of existing domains rather than new domains introduced by LLMs. This shift has also resulted in the reorganization of the community’s knowledge structure. In particular, modeling the knowledge in the community as a network of co-occurring domains, we find that popular knowledge domains, i.e., historically well-discussed domains, have weakened significantly, as activity migrates toward niche domains. Consequently, the community network becomes more fragmented with a weaker core. Our findings suggest that LLMs substitute for standardized queries, while novel problems that require creative integration of knowledge domains still need human expertise. Such selective substitution of questions has profound implications for the sustainability of online knowledge communities and for the reliability of training data that future AI systems will depend on.
