I suspect this kind of janitor work is why I gravitate toward python. Even in a nice, controlled environment like kaggle, it's remarkable how much string/text manipulation you have to do just to get a basic benchmark or cram stuff into a random forest - and that's where the data has already been put into a text file (in my case, a csv) for you. My guess is that it would take 10 times more data munging to get that file produced in the first place. Python has good libraries and is very good for all that data munging.
Trying to outsource this stuff is appealing, I suppose, but that has turned out in my life to surprisingly difficult as well. I once worked with a supply planning division of a manufacturing organization to try to reduce inventory by anticipating when orders would come in, so I got access to the sales database. It appeared that there was a large spike in orders later in the quarter, and that we were carrying inventory unnecessarily. Actually, as we discovered during a pilot, sales people who got new orders from the same customer later in the quarter would just delete and re-enter an entire order rather than doing small updates, which they found to be a hassle (this was quite a while ago, when these systems weren't as easy to use).
These attempts to section off and outsource the "low value" work often just make things worse. It's just too unpredictable.
Trying to outsource this stuff is appealing, I suppose, but that has turned out in my life to surprisingly difficult as well. I once worked with a supply planning division of a manufacturing organization to try to reduce inventory by anticipating when orders would come in, so I got access to the sales database. It appeared that there was a large spike in orders later in the quarter, and that we were carrying inventory unnecessarily. Actually, as we discovered during a pilot, sales people who got new orders from the same customer later in the quarter would just delete and re-enter an entire order rather than doing small updates, which they found to be a hassle (this was quite a while ago, when these systems weren't as easy to use).
These attempts to section off and outsource the "low value" work often just make things worse. It's just too unpredictable.