EDIT: On second thought you basically answered my question in the last paragraph of your reply, I'm keeping it up in case you want to confirm my understanding, but the way I read it, the 15 % speed up comes from the fact that indeed CPU processing (the regexes) was comparable to the file reading in speed and breaking up the bits to individual core caches is what gives way for the speed up, memory is fast enough for this to be possible.
I don't understand the I/O part of the whole deal. AFAIK you can't read multiple files at once off the drive, so are the files read sequentially to memory and then searched in parallel in memory? Does memory allow you to read multiple places at once, even with multiple threads? Surely the files are too large for the processor caches to be of any effect. So if you'd entertain my curiosity a bit more, is what's happening the fact that multiple files are cached in memory and memory is so fast that loading bits of the files to the processor cache takes less time than regexing those bits, moving the balance of this process to the CPU-bound side? Is the actual parallelism in the fact that multiple cores can search their individual caches at the same time and loading those caches from the RAM is fast enough to not become a bottleneck? Sorry for possibly amateurish question, I've never dug deep enough when it comes to parallelism to understand this, but I spent a good amount of time thinking about parallelising I/O stuff and came to the conclusion that I/O must be magnitudes slower and thus always is a bottleneck and any sort of I/O-bound problem (which file search surely is) must be non-parallelizable and instead can only be sped up by keeping indexes the way some OS's do.
One of the more important jobs a system's OS does is manage I/O devices. Modern kernels don't wait for a read request and then go and read from the disk drive. They both read ahead, anticipating future read requests from the pattern of requests already made, and cache as much read data as they can to avoid touching the disk when a file is revisited. (I was a kernel architect on IBM's AIX.)
While coding, your overall system is really just idling and it wouldn't be unusual for many of your projects header files and source files to be cached in memory because of your last compile. This effect is more pronounced on machines with a lot of memory of course. The cost savings from careful disk management is very important overall, but will it speed up the ag application? I'm surprised that ag only gets a 15% speed up with threads, but naturally it will depend on many factors.
On a modern linux server the kernel caches files aggressively.
As an example, we have a server that can saturate 40 cores of CPU during load if fed data quickly enough. On the first load after cold boot, it takes about 2 minutes to run (fed by SSDs). Second boot, about 30 seconds.
If we run a sequential data load off SSD, it's more like 8 minutes on the cold run. So, even off non-RAM storage, parallel reads can help a lot.
Can accessing data through vfs cache be parallel? On systems with lots of free RAM (that system can use for caching) ag is faster after the first run on same directory tree. Of course speed up is mainly due to cache in itself.
I don't understand the I/O part of the whole deal. AFAIK you can't read multiple files at once off the drive, so are the files read sequentially to memory and then searched in parallel in memory? Does memory allow you to read multiple places at once, even with multiple threads? Surely the files are too large for the processor caches to be of any effect. So if you'd entertain my curiosity a bit more, is what's happening the fact that multiple files are cached in memory and memory is so fast that loading bits of the files to the processor cache takes less time than regexing those bits, moving the balance of this process to the CPU-bound side? Is the actual parallelism in the fact that multiple cores can search their individual caches at the same time and loading those caches from the RAM is fast enough to not become a bottleneck? Sorry for possibly amateurish question, I've never dug deep enough when it comes to parallelism to understand this, but I spent a good amount of time thinking about parallelising I/O stuff and came to the conclusion that I/O must be magnitudes slower and thus always is a bottleneck and any sort of I/O-bound problem (which file search surely is) must be non-parallelizable and instead can only be sped up by keeping indexes the way some OS's do.