He explains how and why in the article, and gives arguments. You do none. The pr... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

Aissen on June 7, 2012 | parent | context | favorite | on: Testing 3 million hyperlinks, lessons learned

He explains how and why in the article, and gives arguments. You do none.

The problem with whitelist-only robots.txt is that they favor monopolies and startups are the ones getting the "fuck you". But maybe you don't care about that.

tomjen3 on June 7, 2012 | [–]

As a webmaster, why would I want bots to go to my site that doesn't bring any (or much) trafic?

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact