My niche little site, http://golfcourse.wiki seems to be very popular with AI bots. They basically become most of my traffic. Most of them follow robots.txt, and that's nice and all, but they are costing me non-trivial amounts of money.
I don't want to block most search engines. I don't want to block legitimate institutions like archive.org. Is there a whitelist that I could crib instead of pretty much having to update my robots file every damn day?
Comments URL: https://news.ycombinator.com/item?id=42861047
Points: 15
# Comments: 8
Inicia sesión para agregar comentarios
Otros mensajes en este grupo.
Article URL: https://www.construction-physics.com/p/why-its-so-hard-to-build-a-jet-engine
Comments
Article URL: https://github.com/cmackenzie1/torii-rs
Comments URL: https://news
Some background: I work on Langfuse and we've been collaborating with LiteLLM.
(LiteLLM is a Python library and proxy/gateway that handles cost management, virtual keys, caching, and rate-limiti
