AI Bot Defenders: Protecting Duke's Digital Infrastructure from Unwanted Web Scraping

Description

Ever wonder how ChatGPT or Claude.AI gather massive amounts of data they need to function? They deploy automated bots that crawl websites, extracting content at scale, sometimes without permission. This project puts you on the front lines of understanding & controlling this data collection process. Working with Duke's web logs & cutting-edge tools from the Argus Lab, you'll analyze how AI bots behave in the wild, identify patterns in their scraping activities & develop defensive strategies to protect institutional resources. You'll work at the foundational data layer of the Paladin AI Tech Stack, tackling one of the most pressing challenges in the AI era: who gets to control the data that powers AI?

This is your chance to shape how organizations defend their digital assets & gain hands-on experience with large-scale data analysis and cybersecurity principles while contributing to tools that could influence how Duke and other institutions manage their web presence. Whether you're interested in AI ethics, data privacy, cybersecurity or system design, this project offers practical skills & real-world impact.