My niche little site, http://golfcourse.wiki seems to be very popular with AI bots. They basically become most of my traffic. Most of them follow robots.txt, and that's nice and all, but they are costing me non-trivial amounts of money.
I don't want to block most search engines. I don't want to block legitimate institutions like archive.org. Is there a whitelist that I could crib instead of pretty much having to update my robots file every damn day?
Comments URL: https://news.ycombinator.com/item?id=42861047
Points: 15
# Comments: 8
Accedi per aggiungere un commento
Altri post in questo gruppo
Article URL: https://tolstoyan.substack.com/p/youth
Comments URL: https://news.y
Hi HN!
My latest side project is knowledge graph that maps the French culinary network using data extracted from restaurant reviews from LeFooding.com. The project uses LLMs to extract structure
Please lead with either SEEKING WORK or SEEKING FREELANCER, your location, and whether remote work is a possibility.
Please only post if you are personally looking to hire a freelancer or work a
Please state the location and include REMOTE for remote work, REMOTE (US) or similar if the country is restricted, and ONSITE when remote work is not an option.
Please only post if you pe
Article URL: https://arxiv.org/abs/2502.20547
Comments URL: https://news.ycombinator.c
