Is this URL blocked by your robots.txt?
Paste the file and a URL. The answer comes with the rule that produced it, so you can see why.
Paste a robots.txt and a URL to test it against.
How does a search engine decide which rule applies?
One group applies: the one whose User-agent matches most specifically, with a named crawler beating the wildcard. Within that group the matching rule with the longest path wins, whether it is Allow or Disallow, and Allow wins a tie. This matters because the naive reading, that any matching Disallow blocks the URL, is wrong and will tell you a page is blocked when it is not.
Does Googlebot follow the wildcard rules too?
No, and this is the mistake that costs people the most. If there is a User-agent: Googlebot group, Googlebot uses only that group and ignores the User-agent: * group entirely. Rules you thought applied to everyone do not apply to the crawler that has its own section.
What do the wildcards mean?
An asterisk matches any run of characters, so Disallow: /*.pdf$ blocks every PDF. A dollar sign anchors the end of the path, which is why that rule stops matching as soon as a query string is appended.
Does blocking a page in robots.txt remove it from Google?
No, and this surprises people. Blocking stops the page being crawled, not indexed. A blocked URL that other pages link to can still appear in results, without a description, because Google knows it exists but cannot read it. To remove a page use a noindex tag, which requires the page to be crawlable.
Other tools
This tool checks one page at a time. Visorank does the same work across every page on your site continuously, and shows you which queries people actually search rather than estimates.
What Visorank does