robots.txt is a file that tells compliant crawlers which parts of a site they may or may not crawl.
In plain English
This is one part of how search systems find, interpret, compare or present information. You do not need to become an SEO technician to use the idea. The useful question is what job the term describes: discovery, understanding, ranking, presentation or measurement. Once you know that, the jargon becomes much less intimidating.
A real example
You might block a crawler from a technical area that should not be repeatedly fetched.
The point is the relationship: a real person has a need and the search system is trying to connect that need with useful information. The term describes one part of that connection.
Why it matters
It helps manage crawler access and crawl load.
For a small business, the point is not to chase every search feature. It is to make important pages easy to find, easy to understand and technically available. Knowing the term helps you separate a real problem from an impressive-sounding recommendation you may not need.
When do you actually need to care?
You need to care when an important page is not being found, when search traffic is arriving for the wrong reasons, when a redesign changes URLs or structure, or when someone recommends “SEO work” you cannot connect to a real problem. On a healthy small site, you do not need to obsess over every search-system detail. You need enough understanding to know what matters now and what can wait.
Useful questions to ask
- Is this about being discovered, understood, displayed or measured?
- Does it affect a page that is important to customers or revenue?
- What evidence shows there is a problem worth fixing?
What should you do with this?
Ask three things: what practical problem is this supposed to solve, how would you recognise that problem on your site, and what evidence would show that the work helped? Those questions turn most search jargon into a sensible business decision.
What this does not mean
It is not a reliable way to keep private information secret, and blocking crawling does not always guarantee a URL can never appear in search.
The short version
Use the right tool for the job: crawl control is different from noindex and access control.
Related: noindex and crawling.
Further reading: Google on robots.txt.