It lives at yoursite.com/robots.txt and is standardized as RFC 9309. Well-behaved crawlers read it before fetching pages. It's a request, not a lock: it doesn't hide pages from people.
Many sites block AI assistants by accident, through a security plugin, a theme default or a copied rule. One line can stop ChatGPT or Perplexity from reading every page.