One fair baseline is whether or not the custom User Agent you're using to scrape has request timing that's on the order of what a fast human visitor might do. If the site can't handle that, it's certainly not the UA's fault.