Bots Behaving Badly: Implications of Bot Traffic on Research and Knowledge in the AI Economy

Most academic and nonprofit institutions have mission- and policy-driven mandates to make research data publicly available at no cost to users. However, the viability of maintaining “open access” public collections in the rapidly developing AI ecosystem is uncertain. Since 2022, rising bot traffic associated with large language models and machine learning has cost research collection hosts and maintainers increasing amounts of time, labor, and money. Academic institutions, data hubs, and open collections like Wikipedia all report struggles to minimize downtime and compute costs due to bot confrontations. While defense against bots is an urgent, visceral concern right now, collections managers in these institutions have a particular existential concern: bot traffic signifies that entrepreneurial technologists are absorbing generations of curatorial work into systems that may ultimately undermine those collections. This concern is exacerbated by the difficulty in gauging collection “use” (and therefore, value) by traditional website analytics and metrics. What if these challenges, taken together, could also signify opportunities—including mechanisms to encourage better behavior from bots? Our presenters will share their findings regarding current approaches to bot traffic and usage analytics taken by collection curators. We will share several new approaches that explicitly work to shift both technical and social incentives in order to address these issues.