I don't remember and wasn't able to find it either, but I know which study you're talking about. There was a professor who gave his intro CS students an exam where they were supposed to interpret code. Of course, most of them had no idea what they were reading. The statement "X = X + 1" is (as a mathematical equation) garbage, even though it's legitimate in a conventional CS context (whereby '=' represents assignment). So almost all of the students interpreted the code incorrectly.
What separated the (future) successes from the failures was not how accurate their interpretations were, but whether they were consistent. The students who came up with consistent interpretations of these (a priori) meaningless symbols and could rigorously apply them were able to grok programming-- through trial and error, they picked up the actual meanings over time. The ones who weren't consistent, who just didn't "get" that these symbols had rigorous and inflexible meanings, were the ones who failed.
I actually think this style of rigor in thinking can be learned, even for average people, but it takes time. You can't go from zero to fluency in one semester. The problem (of American mediocrity in mathematical thinking, at least over the bottom 90 percentiles) is similar to what comes up in affirmative action debates: we're addressing it 18 years too late. In the US, many students never encounter mathematical proof or computer programming, except among the elite (e.g. USAMO, IMO). These aren't impossibly hard. Bulgarian and Japanese secondary students tackle proofs. Unfortunately, in the U.S., proofs are extremely rare in the secondary curriculum and computer programming is, if offered, an elective.
The study you are thinking of is "The Camel Has Two Humps" by Dehnadi and Bornat, which was never officially published (and having read it, has several control issues that are cause for concern with respect to internal validity). However, most people continue to quote this study without realizing that the authors have retracted their original claims. In a larger, later follow-up replication study (Mental models, Consistency and Programming Aptitude by Bornat, Dehnadi, Simon), the authors state:
"We now report that after six experiments, involving more than 500 students at six institutions in three countries, the predictive effect of our test has failed to live up to that early promise. We discuss the strength of the e ffects that have been observed and the reasons for some apparent failures
of prediction."
What separated the (future) successes from the failures was not how accurate their interpretations were, but whether they were consistent. The students who came up with consistent interpretations of these (a priori) meaningless symbols and could rigorously apply them were able to grok programming-- through trial and error, they picked up the actual meanings over time. The ones who weren't consistent, who just didn't "get" that these symbols had rigorous and inflexible meanings, were the ones who failed.
I actually think this style of rigor in thinking can be learned, even for average people, but it takes time. You can't go from zero to fluency in one semester. The problem (of American mediocrity in mathematical thinking, at least over the bottom 90 percentiles) is similar to what comes up in affirmative action debates: we're addressing it 18 years too late. In the US, many students never encounter mathematical proof or computer programming, except among the elite (e.g. USAMO, IMO). These aren't impossibly hard. Bulgarian and Japanese secondary students tackle proofs. Unfortunately, in the U.S., proofs are extremely rare in the secondary curriculum and computer programming is, if offered, an elective.