Book-Reader-Assist/library/The Psychology of Computer Programming/Chapter-09-Intelligence-or-Problem-Solving-Ability/Chapter-09-source-text.md

The Psychology of Computer Programming — Chapter-09: Intelligence, or Problem-Solving Ability

• Source: /library/The Psychology of Computer Programming/source-file.pdf
• PDF pages: 213–233
• Pages without text: none
• Note: Includes the Silver Anniversary comments that immediately follow the original chapter. Text comes from the scan's OCR layer and may contain recognition errors.

---

INTELLIGENCE, OR PROBLEM-SOLVING ABILITY orking with programmers, as we have seen, is working with people that are above average in intelligence. Although no formal study has been reported on the subject, we could make a fair guess that the average programmer’s IQ is well above the average even of college graduates, and that the more successful programmers, by and large, have an even higher average IQ. Not, of course, that intelligence is all there is to the matter—we have long since disposed of that fallacy. But since above-average intelligence is something most programmers possess, we are sure to understand programming better if we look at it in the context of how programming work is affected by in- telligence, whatever that may be. 161

162 Programming as an Individual Activity PSYCHOLOGICAL SET For certain types of error location activities, psychological set proves a major impediment. Numerous experiments have confirmed that the eye has a tendency to see what it expects to see. For instance, when looking rapidly over lists of words, a subject may encounter a “word” which is actually not a word at all, such as ‘“‘dack.” The first influence of set can be seen in the fact that the subject sees this nonword as a word, for he finds it among words and thus has a predisposition, or set, to see it as a word. Secondly, if the subject has been told that the words in the list have to do with “animals or birds,” he is quite likely to read ‘‘dack’’ as “duck.” If, on the other hand, he has been told the words have to do with ‘means of transportation,’ he will much more often read it as “deck” or “dock.” Anyone who has ever tried proofreading is aware of this type of set phenomenon, and anyone who has ever tried to locate a mispunched word in a computer program is more than just aware—he is scarred. In such tasks as proofreading, it is hard to measure the difficulty in over- coming the effects of set, for texts that are sufficiently perfect to serve as standards and sufficiently difficult to simulate actual conditions are difficult to obtain. Not so in computer programs, for the computer pro- vides an automatic testing ground for the efficacy of proofreading. The testing that a computer can provide for this type of error in its own programs has quite a range of subtlety and power. On the simple end of the range lie the tests for unrecognizable syntax, misspelled keywords, and ill-formed constants. The type of cross-checking provided by symbol tables, cross-reference lists, and flow analyses go one step deeper, but are still essentially static. Dynamic checking for uninitialized variables, flow-tracing, and subroutine call-tracing contribute to cleaning up such typographical errors as may have sifted through the other levels. Never- theless, no automatic system can be guaranteed to locate all such errors, and we may expect certain improvements by attention to psychological facts when the programs are written or languages are designed. Related to the concept of ‘‘set’’ is the concept of “‘distance.”’ Not all misreadings are equally likely, regardless of the set of the reader. Thus, “daxk”’ is less likely to be mistaken for “duck” than was ‘“‘dack,”’ because the reader will have to make two letter transformations instead of one. In information theory, two “messages’”—which may be taken to be strings of bits—have a “distance” that can be obtained by counting the nurnber of bit positions in which they differ. The importance of this measure is that it indicates the number of bits that would have to be changed to transform the one message into the other—as might happen if noise were present in the transmission.

Intelligence, or Problem-Solving Ability 163 For the symbols of a programming language, just as for the words in a natural language, such a simple measure of ‘“‘distance” can only be taken as a first approximation to the likelihood that one symbol will not be mistaken for another. For instance, psychological tests have shown a tendency for initial and final letters to be more significant in making dis- tinctions. Thus, “gucr’’ would be much less likely to be seen as “duck’’ than would “daxk,’”’ even though they each differ in exactly two letters from “duck.” Moreover, each pair of symbols cannot be assumed to be the same distance apart as each other pair. Although the exact relation- ship must depend upon the typescript used, letter pairs such as “x” and “k” would seem to be more readily confounded than such pairs as “x” and “o.” One of the first lessons the novice programmer learns is to make careful distinction between his handwritten ‘‘zero”’ and ‘‘oh,” if someone else is keying his program. Most programmers, unfortunately, never ad- vance much beyond this point in developing habits that will facilitate the conquest of set as a programming hazard. For example, no matter how carefully one writes the zero in the symbol ‘“‘STOP,”’ it will be mistaken for an “oh” all along the line. The psychological distance between “STOP” and “STOP” is so slight—because of the similarity of the zero and oh, the middle position of the single differing letter, and the set within the symbol which strongly biases us toward the English word—that the programmer who habitually makes such choices is headed for certain disaster. No doubt, the rather extensive success of automatic methods of de- tecting such errors has lulled many programmers into carelessness when choosing symbols. Nonetheless, there will always be some situations in which the compiler or interpreter cannot make a sensible conclusion that there is an error. In one case, a programmer had used both the symbols “SYSTSTS” and “SYSSTSTS” in the same code and only discovered that one had been substituted for the other after hundreds of hours of errone- ous simulations had been run, a book had been published containing these results, and several systems had been misdesigned on the basis of their errors. All this could have been avoided if he had adopted the prac- tice of keeping a minimum distance of two (dissimilar) characters between his symbols, and perhaps ensuring that at least one of these differences was at the beginning or end. Mnemonic symbols are particularly susceptible of inducing a torpor in the program reader. Mnemonic symbols expose us to misreading for sev- eral reasons:

1. They tend to make programs seem “‘sensible” by their satisfaction of our general set toward sense over nonsense.
2. They play upon our tendency to believe in the name, rather than the denotation of the name. Who would believe that the symbol “FIVE”

164 Programming as an Individual Activity denoted a value of 4? But it did, in one case where the programmer had to modify his code and didn’t have time to change all references to “FIVE.” He did, however, have time to rerun the program—after having taken much time to locate the source of difficulty. 3. They tend to give something less than an optimal “distance’’ pattern. English words, for example, are not random collections of letters. Some patterns such as consonant-vowel-consonant, or consonant- vowel-vowel-consonant, tend to occur more frequently. Even worse, there are homographs such as “LEAD” and “LEAD,” which might pop up in the same program from two different origins. 4. Optimal distance is further reduced because of regularities in the way we abbreviate, leading to such ambiguities as “PEND,” for a record that is held in pending status, and “PEND,” short for “end of part P.” We cannot abandon the subject of set errors without a comment on comments. The whole idea of a comment is to prepare the mind of the reader for a proper interpretation of the instruction or statement to which it is appended. If the code to which the comment refers is correct, the comment could be useful in this way; but if it happens to be incorrect, the set which the comment lends will only make it less likely that the error will be detected. This effect of comments on interpretation of erroneous code can be measured quite nicely in an experiment in which several versions of the same code are produced, one with correct comments, one with one or two incorrect comments, and one with perhaps no comments at all (Okimoto, 1970). For certain types of code, at least, correct interpretation of what the program does can be obtained more reliably and faster without any comments at all. Correct comments, if well constructed, reduce errors when compared with cases in which incorrect or misleading comments are used. Nevertheless, many experienced programmers make a habit of covering all comments when scrutinizing a program listing for errors, thus reducing set which, though helpful to understanding a correct pro- gram, only complicates the already impossible job of debugging. SOME DIMENSIONS OF PROBLEM SOLVING In psychology, ‘‘set” is usually considered part of the study of “per- ception” rather than part of “intelligence.” Yet it should be clear from the preceding section that set phenomena can influence behavior which we would surely label ‘‘problem solving.” Of course, even before the question of problem solving comes the question of problem avoiding. As we Saw, numerous techniques exist whereby a programmer can avoid

Intelligence, or Problem-Solving Ability 165 the problems of set altogether in certain situations. Considered in the abstract, a programmer who avoids a problem altogether is more ‘“‘intel- ligent” than one who brings it upon himself, whether or not he ultimately “solves” it. However, abstract ideas about intelligence rarely fail into accord with our beliefs about concrete situations. Lacking any objective measure, we often judge how difficult a program is by how hard a programmer works on it. Using this sort of measure, we can easily fall into believing that the worst programmers are the best—because they work so hard at it. A case in point was a programmer who worked 14 hours a day, seven days a week, for eight weeks to get a small program running in a new installa- tion. For his efforts, his company gave him an award for exceptional service. Shortly thereafter, another programmer (for the first had been promoted to a management position as an additional reward) was given the job of making some additions to this program. He found that the pro- gram was such a confusing mess that it was easier to rewrite it than to try and modify it. The rewriting and debugging took exactly one week, working normal hours. Even considering that writing a program for the second time is far easier than writing it the first, the difference is significant. Moreover, the new-program ran eight times faster than the old, took half the storage, and was half as many lines of coding. Clearly, the first programmer had been rewarded for making a mountain out of a molehill. The discovery of this misapplication of management largesse then led to a severe drop in morale in this programming group. Problem-avoiding behavior, then, is intelligent behavior at its highest, although not very intelligent if one is trying to attract the eye of a poorly trained manager. It will always be difficult to appreciate how much trouble we are not having, just as it will always be difficult to appreciate a really good job for problem solving. Once the problem solution has been shown, it is easy to forget the puzzlement that existed before it was solved. For one thing, one of the most common reasons for problem difficulty is the overlooking of some factor. Once we have discovered or been told that this factor is significant, working out the solution is trivial. If we present the problem to someone else, we will usually present him with that factor, which immediately solves nine-tenths of the problem for him. He cannot imagine why we had such trouble, and soon we begin to wonder ourselves. Overlooking a factor in a problem is just one special case of assump- tions leading us astray. We assume that a certain factor is not important— probably without even thinking about it in any conscious manner. We are led similarly astray by assuming that a certain factor is important, when it has no significance for the problem at hand. People who spend much time debugging other people’s programs soon learn not to listen to ex-

166 Programming as an Individual Activity planations before tackling the problem, for these explanations tend to put the listener on the same false track of assumptions that prevented the speaker from finding the bug for himself. Psychological set, of course, is another form of making assumptions. Although the assumptions may be buried more deeply in this case, they have the same effect on problem solving. Could we not say, then, that the first rule of problem solving is “don’t make assumptions’? We could say that, but we would be precisely wrong. If we are to be successful at solving problems, we must make assumptions. If we really faced each problem as entirely new, it would be impossible to improve our problem-solving performance. The set we have, for example, which enables us to read a misprint as if it were correct, is a most valuable asset—in most situations. Only when we are proofreading, something few of us spend much time doing, does this particular set cause trouble. In- telligent behavior, then, does not consist in eschewing assumptions, but in being sufficiently flexible to manipulate assumptions as the occasion demands. In other words, being intelligent is not having some magic formula which one can apply to every problem. It is, rather, having a number of “formulas” and not being so much in love with one that it can- not be dropped for another. Before we leave the topic of adaptability in problem solving for closer examination of some of the “‘formulas’”’ which are selectively applied by the successful problem solver, we must lay to rest one more fallacy about intelligence. Intelligence, however it is ultimately defined, is, at best, a statistical concept. We cannot ever hope to measure intelligence by performance on one particular problem, for there are as many non- intelligent reasons for getting the ‘‘right’’ solution as there are intelligent reasons for getting the ‘wrong’ solution. Indeed, the explanations for success given by some programmers bring to mind the story of the village idiot who won the monthly lottery. When asked to explain how he picked the winning number, he said, ‘“‘Well, my lucky number is seven, and this was the seventh lottery this year, so | multiplied seven times seven and got the winning number—63.”’ And, when someone tried to tell him that seven times seven was forty- nine, he merely answered with disdain, “Oh, you’re just jealous’’—which, of course, was true. FACETS OF PROGRAMMING INTELLIGENCE Adaptability, then, is required for all sorts of intelligent behavior. Be- havior which, though mental, only requires carrying out a set of fixed rules is not properly considered intelligent. It might better be carried out

Intelligence, or Problem-Solving Ability 167 by machines than by people. Not that carrying out a set of fixed rules cannot be an important part of intelligent behavior. On the contrary, a programmer who cannot add two numbers together without extraordinary difficulty is heavily handicapped in the race for better problem solving, unless, of course, he turns his handicap into an asset by developing shortcuts that bypass the arithmetic he cannot do. To a great extent, problem-solving technique is idiosyncratic, if only because certain people can do certain things better than others. Each person, if he is intelligent, tends to look for methods of solution that depend on his best qualities and avoid his weakest. As a specific ex- ample of such a quality, consider the facet of memory. There is no doubt that memory is one of the most important aspects of intelligence for a programmer—if he can but harness it. Memory helps a programmer in many ways, not the least of which is by enabling him to “work” on problems when he does not have all his papers in front of him. Consider this anecdote related by a programmer about how he solved a problem while lying in bed: The problem was given to me by a programmer | encountered yesterday morning at the computing center. He said he had the problem since last April (it is now January). The problem was not very serious, but it had puzzled. him on and off since then, and everything he tried failed to work. It was a PL/I program, and the trouble was in the format of the output. He had established an ON ENDPAGE unit, but it only worked at the end of the first page and when it was raised by SIGNAL. After the first page, the listing just went on from page to page without producing the headings he wanted. | checked the PAGESIZE he was using, and his job control cards—to see if he had some strange carriage control situation. | checked the position of the ON-unit, but obviously it was executing once. Were there any switches in it? Nothing. The only unusual thing | found was that he had used a PUT SKIP to print the heading, not a PUT PAGE as | ordinarily do. | pointed out to him that this explained why the heading he had printed failed to go on the top of the page. But, after testing a number of other hypotheses, | knew | wasn’t going to find the major trouble. | showed him how he could get the right output, using a test of LINENO to raise ENDPAGE. That satisfied him, but | knew | couldn’t put my mind at rest until | understood what was happening. | put the problem out of my mind, but when | went to bed last night, my mind seemed clear, so | decided to work on it as | lay there. | reconstructed the whole situation mentally—the coding and the output. When my eyes are closed and | am in a quiet place, | can call up the picture of any program | am currently working on—even one like this, which | had seen only once. | scanned the output in my mind and tried to imagine what kind of program would produce this output. After reviewing my hypotheses from the morning and rejecting each on carefully considered grounds, | decided to look for something new. The strange element, | felt in scanning over the program, was the PUT SKIP in the ON-unit.’ This stood out in my image the more frequently | scanned it, for | never did that. Never? In that case, perhaps it was causing the dif- ference. But why would starting a page in a different way cause the end of

168 Programming as an Individual Activity the page to be missed? Well, how is the end of a page detected? By being so many lines from the top. But how is the top determined? | realized | didn’t know that answer precisely, so | speculated on possible alternatives. By this time, | knew | was on the right path, and | simply considered each alternative in turn, imagining the action each would produce in this program. Finally, when | worked through the action under the hypothesis that only PAGE (or possibly LINE, which did not apply in this case) could start a new page, | realized the problem. PUT SKIP in the ON-unit did not start a new page—the line number simply kept increasing and no new end of page was ever reached because no new page had ever been started. Satisfied that | had solved the problem, | went immediately to sleep. This morning, | made this test case to demonstrate my conjecture, and you can see that it is precisely as | Say. Without the aid of a fine memory, this programmer might never have solved this problem—and learned something new—because he might never have seen this program again. On the other hand, if he did not have this kind of memory, he probably never would have attempted this ap- proach to the problem. He might, for example, utilize his cleverness at creating critical test cases by solving the problem using the computer. For his inadequate memory, he would substitute an actual copy of the problem deck—which is nothing to be ashamed of. Indeed, by attacking the problem on the machine, this programmer might have the solution to the problem before going home, leaving his sleep untroubled by bugs. Which method is superior? We really cannot give an answer in isolation. If, for example, machine access is poor, the second programmer will be at a disadvantage; but if, on the other hand, everyone is working overtime and barely has time for sleep, let alone quiet reflection, the first programmer will never get to show his brilliance. Naturally, it would be best to have both abilities to an equal extent and to apply each as is appropriate to the problem and the overall situation; short of that, we must make the best of what we have. Just as different working conditions favor the application of different forms of intelligent behavior, so do different programming phases give different programmers a chance to shine. For example, when we are making the overall design of a program, what we most need is the ability to create new programming ideas and to screen them on the basis of broad principles. Examples of such screening ideas are symmetry of Structure and generality of function—one leads to simple coding of diffi- cult problems and the other leads to the solution of difficult problems with simple coding. Still, these critical abilities are useless if there is a paucity of ideas to which to apply them. Nothing cannot be criticized. Thus, a programmer lacking in either ability—creativity or selectivity—will be handicapped in attempting to design programs. When coding, however, different abilities come to the fore. Instead of

Intelligence, or Problem-Solving Ability 169 the broad, sweeping mind, the mind which is clever at small things now excels. Then, when testing, the programmer must switch to yet another group of gifts—particularly the eye for wholeness, or gestalt. Consider the following tale. | was eating breakfast and reading an article by Stephen Spender called “The Making of a Poem.” On page 120, | reached the end of one section and set the book down to put some more sugar on my cereal. When | picked the book up again to start reading a section called ‘‘Memory,” | immediately had a feeling that there was something wrong in the first sentence which started: “lf the art of concentrating in a particular way... .” I felt, more or less simultaneously, that the trouble was in the word ‘‘particular”’ and that it involved a misprint. The misprint, however, was rather confusing for | sensed that it was a letter inversion but | also sensed—a little bit more weakly, though, | have a definite impression of that—that there was a letter missing. | examined the word ‘‘particular’’—which, by the way, | often mistype s “particluar’—first for the inversion and, failing to find that, for the omitted letter. | spent a rather long time looking for the error—I could measure that because | ate five or six spoonfuls of cereal in the process, the amount | ordinarily eat between sips of water. But | could not find anything wren, and when | reached for the water glass, | was rather confused. The water glass was empty, so | set down the book and went to the sink to fill it. Upon returning, | drank some water, picked up the book, and started to read. | finished the paragraph without further difficulty, but when | com- menced reading the next, | immediately saw that the line: “All poets have this highly devoiped sensitive apparatus... .” contained the misprint of the word ‘devolped,’’ which stood in the same position in that sentence as ‘particular’ had stood in the first sentence of the preceding paragraph. | had the right impression, but | had “focussed” wrongly. Although this error was in printing, its discovery and location followed very closely the process by which many programming errors are found. First, there is only the gestalt, a general feeling that something is out of place without any particular localization. Then follows the ability to shake loose from an unyielding situation—the ability to change one’s point of view, even by employing external devices such as going for a glass of water. Then, however, one must go from the general to the particular— “focussing,’’ as it was called here. Although one does not find errors efficiently by a detailed search of each line, or word, or character, the ability to get down to details is essential in the end. Thus, for debugging, an almost complementary set of mental powers is needed. No wonder good debuggers are so rare! Even more of a rarity is the good documenter. Documentation difficul- ties come from many sources. In the first place, if the program is not well written, there is not much that documentation can do to resuscitate it. Since programmers usually document their own productions, the good documenter has to be a good programmer to begin with—and then he

170 Programming as an Individual Activity must add the capacity to express himself verbally and graphically. Finally, he must have the patience to work out those last few ambiguities in his documentation—for the last 5 percent of the work makes the document 100 percent better. APTITUDE TESTS If intelligence is so important for programming success, what can be done to select those people who have what it takes to do the job? The possibility of administering tests to select programmers had long be- witched programming managers, but over the years nobody has ever been able to demonstrate that any of the various “programmer’s aptitude” tests was worth the money it cost for printing. We should be remiss, how- ever, if we did not attempt to explore the reasons for this universal failure. In the first place, of course, programming is a diverse activity, and any test that gives a single ‘‘grade” is hardly adequate to measure the aptitudes required. Some tests have been designed to give multiple scores, but even if these had individually valid measures, they would be difficult to apply sensibly, and the average personnel manager simply would not bother. Moreover, just because of the multidimensionality, someone who is sorely deficient in one area may turn out to be outstand- ing in another—provided that the opportunity to use his strong points exists. On the other hand, sometimes a person who scores high on every possible scale performs poorly because he is missing some “minor” ability, such as the ability to get along with the people he has to work with. These are theoretical conjectures, however, because nobody has put together a test with any measurable validity. The closest thing we have to a validation of a programmer’s aptitude test are studies which show that people who scored high on the test were better students in the ensuing programmers’ training. When it got down to the nitty-gritty of actual programming, however, there was no correlation—or even a slight negative correlation—between test scores and rated performance. This sorry picture is not unique to programming. Intelligence tests gen- erally—such as the famous IQ tests—are able to predict success in school-type situations—and in nothing else. In fact, as one wit put it, intelligence tests measure the ability to take tests. We have reason to believe that this appraisal is not far from the truth. For example, in IQ tests, speed is very much a factor, as it is in school situations generally. But, in the office, the difference between one hour and one hour and ten minutes is only ten minutes—not the difference between an A and a C. Slow and steady may not win the race, but programming is not a race.

Intelligence, or Problem-Solving Ability 171 Another typical distortion of intelligence tests is in the emphasis they place on short-term—rather than long-term—memory. They could hardly be expected to do otherwise, given the constraints under which they must be administered. In an IQ test, one is asked to memorize nonsense words or arbitrary lists. But in “real life,” it is selective memory which pays— the ability to forget the unimportant and retain the important over long periods. Not so in school, however, and so we have the IQ test and the school grades going hand in hand—off in a different direction from any- thing we are interested in. Finally, and this leads right into our next topic, 1Q scores, and pro- grammer’s aptitude scores, are demonstrably correlated with certain forms of training. In big cities, an eager parent can take his child to a special tutor who guarantees to raise his 1|Q by so many points for so many dollars. The same techniques are used by certain programming schools to help their graduates get jobs with those companies that rely heavily on aptitude testing. So, no matter how much we would like to have a magic wand which would point out the best programmer prospects, we are just going to have to learn to do without. APTITUDE TESTS FOR PROGRAMMING All of this leads up to specific tests that have been used in a grand attempt to measure aptitude for programming. Probably the foremost among these is the so-called PAT, or Programmer’s Aptitude Test. Actu- ally, this is not a single test, but an unknown number of variants on an original test made up one afternoon by a group of programmers in IBM’s New York Scientific Computing Center sometime before 1956, and ad- ministered there to all job applicants and interested visitors. One of the reasons for the profusion of variants is that the test has been so widely used and so unprotected that nobody knows how many people have taken it, or, in particular, which people have taken it. Not that the variations give much protection, since there is consider- able transfer of learning from one variant to another. Over a series of classes involving IBM and other programmers, | asked people to report the number of times they had taken a variant of the PAT and what their scores had been. Since this was a ‘‘safe’’ situation—with nobody’s job at stake—there is reason to believe that the replies are not too inaccurate. Out of 27 people who had taken the PAT more than once, 23 received A’s the last time they took the test—they were all employed programmers working for companies which regarded A’s as an important hiring crite- rion. Of these, 12 had A’s the first time they took it, 7 had B’s, and 4 had C’s. Nobody had done worse the last time than the first.

172 Programming as an Individual Activity A typical case was a girl working for IBM as a programmer who had graduated from college with a Math major and applied to RCA for a programming job. They had given her the PAT, and she had scored a “low” B. She didn’t get the job, so she interviewed with IBM, who ad- ministered a slightly different version of the test, on which she scored a clear A. She was asked whether she had ever taken the PAT before, and she said “no,” whereupon she was hired. This previous experience with the PAT may be one reason researchers have been unable to correlate PAT performance with job performance. A few correlations have been reported (see Reinstedt, 1964), but we must be careful as to what we regard as “significant” in these cases. For example, the best correlation found in these studies was 0.7 between the PAT and supervisor’s ranking. What does 0.7 correlation mean? For a single study, a 0.7 correlation between two variables means that (0.7)? — 0.49, or 49 percent of the variation in one variable can be ac- counted for by the other—although it doesn’t say which is which. This still leaves more than half of the variation to be explained, even if the correlation coefficient were an ‘“‘explanation.’’ One of the reasons for the popularity of the correlation coefficient is the way it seems to overstate the case, since the number used is always larger than its square. The second problem with such correlations is that the score—which is definite enough—is correlated with the supervisor’s ranking, which is an uncertain measure of programmer performance, to say the least. There really is no instrument today for measuring programmer performance— or reducing it to a single number, in any case. Consequently, there is nothing really reliable with which to correlate the PAT. It may even be that the PAT is a marvelous predictor of programmer performance, but the Supervisors are not themselves sufficiently trained to know it. We must also be wary of one other thing in using such correlations. Nobody knows how many times people have tried to correlate the PAT with job performance. Those who did try and did not obtain ‘‘significant” results more often than not would not publish their trial. We knpw of similar trials because correlations have been found between test scores and school grades (Biamonte, 1964, Gotterer, 1964). But, you see, if given sufficient chances to correlate, we will eventually get some correlation coefficient above any level we desire, if the data are random. Thus, we could say that a correlation of 0.56 will arise, by chance, no more than one time in a hundred, but if we do not know how many trials have been made and not reported, we have no way of evaluating the significance of such a statement. We do know, however, that the PAT is used in hundreds of places. Furthermore, even if the trials have only been made a few times, a correlation of 0.56 occurring one time in a hundred by chance assumes

intelligence, or Problem-Solving Ability 173 that the true correlation is zero. If there is a small positive correlation, say of 0.1, then a spurious measure of 0.56 will occur more frequently than one time in a hundred. But a correlation of 0.1 means that 1 percent of the variation is accounted for by the correlation, and this is hardly the type of information we can use to make personnel decisions. Assuming that the correlations were reliable, we have, on the basis of the recorded literature, no instrument any better than the PAT. Admittedly, it does predict scores in programming classes fairly well, but that is not what we are buying when we hire a programmer. As Mayer (1968) puts it so well, Very probably if you would use all of the tests to select an individual, you can [sic.] obtain a person who has a high probability of successfully com- pleting your training program. Whether this individual is going to like pro- gramming or will possess the motivation that will allow him to take the successful training onto the job site is a question that is not yet answered. Even this condemnation implies that the tests may be all right as far as they go, but that other factors may be more important on the job than pure “aptitude.” Although we can not quarrel with the conclusion, we think a possibility may have been overlooked—namely, that the tests are just not good ones for programmer aptitude. Given the history of the PAT, one wonders why so many hundreds of firms use it slavishly, year after year. A little examination into the structure of the test itself might give us some hints as to what is wrong with it, now that we know that much is wrong with it. The original PAT had three sections: relationship rules for geometric figures (much like standard IQ series), arithmetic reasoning, and number series. The first two have been retained in almost all of the variants of the PAT, although the number series has often been replaced by letter series. Just to get an idea of what might be wrong, consider the letter or number series problems, a typical one of which might be vy Ce eed where the examinee is asked to supply the next number in the series. It is certainly plausible that a modicum of intelligence is required to answer “10”, but is this really the ability we most need in a programmer? Let me give an example of what a good programmer did with such a series. A FORTRAN program had been written with the statement DO 15 |= 10000, 30000, 10000 The program went into a strange loop which finally ended but produced output that nobody could decipher. When it was brought to this program- mer, he thought for a while and then asked himself under what circum- stances this particular DO would not produce the series 10000 20000 30000...

174 Programming as an Individual Activity Suddenly he realized that if the numbers were being kept in a 15-bit register, the next number in the series would be 7232, and the series would look something like this: 10000 20000 30000 7232 17232 27232... which explained precisely what was wrong with the program. In other words, the PAT tests for the ability to see the conventional pat- tern in a series of numbers or letters, but the programmer, especially when debugging, has to look precisely for the unconventional pattern— the deviations. Possibly a much better type of question for programmer aptitude would be something like this: “Given the series 44rénd tell what might normally be expected as the next number, and then de- scribe at least three other choices which might be correct, giving reasons for each choice.” Another section of the PAT is arithmetic reasoning, but | have never had anyone who has been able to explain to me why programmers have to be good at arithmetic. Perhaps before 1956 arithmetic was more im- portant, when most programming was done much closer to machine language than it is today. If you are working in a relatively crude language, it is useful to be able to add two or three hexadecimal numbers so that you can find an address in a dump, but how many programmers do that today? Don’t we have computers just so we don’t have to do arithmetic? | myself have never been able to add 7 and 5 rapidly, but | don’t think that is the thing that is holding me back as a programmer. In fact, knowing that | am not very good at arithmetic, | am more suspicious of my pro- grams, and of my check calculations—which forces me to do more testing of them. It would be rather easy to make an argument that poor arithmetic ability is an asset to a programmer. The third part of the PAT leaves me entirely befuddled. Even in 1956, geometric relationships never seemed to have much to do with program- ming aptitude, but perhaps | missed the point. The one thing that pro- gramming doesn’t seem to be in today’s world is geometric. I’ve never met a programmer who was asked to tell whether two programs were the same if one was rotated 90 degrees. (There was once an article about palindromic programs, which read the same forward and backward, but its contribution to the profession was minor.) Perhaps it is time that some new thinking go into these aptitude tests, if people are going to persist in using them. And persist they will, since even if the promise of success is not great, the rewards are. So let me Suggest a few things that might make more sense for identifying poten- tially successful programmers:

1. Give the examinee thirty or forty papers with random printing on

intelligence, or Problem-Solving Ability 175 them and ask him to place them in ten boxes. One week later, ask him to find the paper with the word ‘‘;COMMODITY” on it. Time his retrieval performance—giving high scores for fast work. Give the series-exception type question suggested above. 3. Tell him you are thinking of a problem which he must solve. Don’t give him any more information unless he specifically asks for it, but answer all his questions literally. Score him on how well he figures out what the problem is. | don’t really have much hope for such tests—either that they will work or that they will be applied if they do work—but they certainly seem more promising than what we have been using so far. Yet all is not quite so bleak. When we are selecting among experi- enced programmers, the situation is potentially different, although few authors or employees seem to realize it. For example, out of 282 organiza- tions using the PAT in one survey, 138 of them still use it for selecting experienced programmers. Why? Because they feel they have nothing else. Lacking anything better, they try what is available in a vain search for the elusive magic test. Such companies are sitting ducks for anyone who comes along with a fancy package of promises—and with lots of sitting ducks, can the hunters be far behind? Not very far—even closer than | thought. After writing the above package, I ran across the following ad in a trade magazine: Nm “TEST THE SKILLS OF YOUR PROGRAMMERS A Program called ¥—————sConsists of procedures, software, instructions and problems designed to evaluate programmer skills. .-----___ measures the ability to write correct, compact, and efficient programs, and can be used to test experienced programmers or to evaluate the programming aptitude of non-programmers. The tests used are language independent, and are weighted in accordance with the difficulty of the problems and the time used to solve them. The ____________ program will run on a ***** System. It is priced at $5,000... .” An evaluation of this ad is left as an exercise for the reader, but aside from its claim to test experienced or inexperienced programmers alike, it does offer one clue as to the proper direction for testing when it speaks of the ‘ability to write correct, compact, and efficient programs.” What does this phrase suggest? As one author comments, if we were hiring an oboe player for an or- chestra, we would be able to give an audition, which would determine in a matter of a few minutes what his qualifications were, at least insofar as rejecting an obvious misfit. After a longer performance, we would have a moderately good measure of the oboist’s abilities. He then goes on to lament, “There is no equivalent of an audition” for programming. But

176 Programming as an Individual Activity why isn’t there? Certainly there is not for a person without experience, but neither is there much for a child who is applying for a traineeship in oboe playing. For a programmer who claims to have experience, why not just have him sit down and write a small program or read and interpret a set of specifications, sketching out his approach to implementing them? Why not, indeed? Perhaps it is because we so rarely read programs that this action never occurs to us, or perhaps it is because when we intend to hire an “experienced” programmer, we deem ourselves lucky if even a single applicant shows up. Testing a single applicant is like testing your wife—if she fails, what are you going to do about it? SUMMARY Everybody involved in hiring programmers has an opinion about what qualities are essential to a good programmer—even though we know that programming is no one single activity. Mayer, as cited previously, believes that his ‘‘X-factor’ is related to aptitude. He speaks wisely, with the caution of one heavily involved in psychological testing. People more involved in programming are more confident of their impressionistic view. Not all of them, however, are as explicit about their X-factors as E. W. Dijkstra, who said: | am engaged in teaching, at graduate level, in producing one variety of “mathematical engineer.’”’ The most powerful test that | know of for an ap- plicant to be one of my students is that he have an absolute mastery of his native tongue: you just need to listen to him. Since Dijkstra is ‘“‘engaged in teaching,” we cannot be too swayed by his remarks, for he may indeed have isolated a factor in academic suc- cess, aS have several other workers. “Verbal ability” is as good a measure as we have of academic success in programming, so Dijkstra’s intuition is probably a good one. But for actual. programming performance, on commercial programs rather than ‘‘toy’’ programs, we lack any aptitude measure at all, except perhaps for general intelligence. For myself, | be- lieve that intelligence has less to do with the matter than personality, work habits, and training. These things, unlike intelligence, can be changed by experience later in life, which turns the problem from one of selecting programmers to creating them. In other words, good program- mers are made, not born; therefore we should turn our attention to the manufacturing, or training process.

intelligence, or Problem-Solving Ability 177 QUESTIONS For Managers

1. Do you use any aptitude tests now in choosing programmers, or does your personnel department use them? If so, what do you know about their validity? Do you make any effort to validate them by evaluating programmers after a period of time on the job? What methods do you use for this evaluation? How convinced are you of their effectiveness?
2. What single important aptitude do you find most often lacking in your programmers? What kind of test do you think would discover that such an aptitude was missing?
3. Would you spend $5000 for the package in the ad? Or $100 per test? If not, why not? If so, have you been reading this book? For Programmers
4. Were you tested for aptitude when you applied for your present job? If so, did you find the test relevant? Do you know if your organization still uses these tests? If they do, and if you feel they are not relevant, what are you doing about it?
5. Do you feel it is possible to test for at least certain crucial aspects of programming aptitude? Make a list of what you consider to be crucial aspects of programming aptitude, and a list of suggestions as to how they could be tested.
6. Has this chapter indicated to you any areas of problem-solving ability or habit in which you may be deficient? If so, what plan do you have for doing something about it? BIBLIOGRAPHY Mayer, David B., and A. W. Stalnaker, Use of Psychological Tests in the Selection of Computer Personnel, SHARE/GUIDE Presentation, October, 1968. Reinstedt, R. N., et a/., Computer Personnel Research Group Programmer Per- formance Prediction Study, The RAND Corporation (RM-4033-PR), Santa Monica, California, March 1964. Biamonte, A. J., Predicting Success in Programmer Training, Proceedings of the Second Annual Computer Personnel Research Conference, Association for Computing Machinery, New York, 1964. Gotterer, M., and A. W. Stalnaker, Predicting Programmer Performance Among Non-preselected Trainee Groups, Proceedings of the Second Annual Computer Personnel Research Conference, Association for Computing Machinery, New York, 1964.

178 Programming as an Individual Activity Okimoto, G. H., The Effectiveness of Comments: A Pilot Study, /BM SDD Technical Report TR 01.1347, July 27, 1970. Luria, A. R., The Mind of a Mnemonist, New York, Avon Books, 1968. A fascinating example of how an individual can use his peculiar abilities or disabilities to achieve fantastic performance—in this case on tasks involving memory. Hunt, J. M., Intelligence and Experience, New York, Ronald, 1961. Hunt reviews the process by which the intellect develops, with an end to seeing what can be done to influence intelligence. Unfortunately, it seems that by the time one is old enough to become a programmer, most of the opportunity for modification has been lost. Perhaps we should start younger. Now that more and more programming fathers and mothers are taking terminals home with them, can it be long before we produce a programming Mozart? Wertheimer, M., Productive Thinking, revised ed. New York, Harper, 1945. Wertheimer is the main founder of the Gestalt school of psychology, and in this book gave his prescription—based upon his experiences and experiments with young children—for teaching people how to think more productively. It seems unlikely that many of the ideas can be applied directly to teaching programmers to program better, but the book certainly is rich in suggestions as to which things to try and which to avoid. Polya, George, How to Solve It, Princeton, Princeton University Press, 1946. Polya, one of the great mathematicians of our time, gives a popular version of his work on techniques of solving mathematical problems and puzzles. Again, the work is highly suggestive for programming, but nobody has put it to a direct test, though many programmers have read it. Ghiselin, Brewster, The Creative Process, Berkeley, California, University of California Press, 1952. People have been studying the “creative process” for a long time—since Plato and Aristotle, at least—and we are a long way from understanding it. We don’t know, even, if programming really requires creativity, or creativity on the level of a Mozart or an Einstein, or even if there are different levels of cerativity, or just differ- ent levels of creations. But for those who suspect that there may be some element of creativity in programming, this collection of original works and excerpts will certainly add fuel to their flame. Even for those who don’t believe in creativity in programming, who could miss a chance to read what Einstein, Mozart, van Gogh, Wordsworth, Coleridge, Yeats, Nietzsche, Jung, and Spencer—to name but a sample—have to say about the subject. Besides, this is the source of the Stephen Spender misprint (in the paper edition). Sackman, Harold, Man-Computer Problem Solving, Princeton, Auerbach, 1970. Sackman’s studies are not concerned solely with programming, but with the more general category of problem-solving behavior. His book contains many useful insights into the effects and lack of effects that working with a computer has on problem-solving behavior, but in a certain way he misses the point on programming as problem solving. Sackman himself is not a programmer, but a psychologist, and the one thing he never seems to have done in his studies was to read the programs produced. Like most psychologists, he apparently assumes, that all completed solutions to the same problem are equivalent, and that only failures to complete a program and get the correct results need be considered carefully. But all programs that give the same output are not alike, as we

Intelligence, or Problem-Solving Ability 179 know, and until psychologists recognize this, most of programming/problem- solving behavior will remain a fog. Weinberg, G. M., Experiments in Problem Solving (Doctoral Thesis), Ann Arbor, University Microfilms, 1965. Recommended for the psychologist who does not understand the commentary in the previous citation. This study shows how sufficient resolution in the ob- servation of the problem-solving task and the ability to characterize the task in terms of the individual subject’s strong and weak characteristics lead to striking insights into problem solving. Such insights will never be available if the level of resolution remains too high, or if individuals are averaged together to get ‘‘statistically significant results.”

«Silver Anniversary Notes Comments on Chapter 9: Intelligence, or Problem-Solving Ability This chapter has an interesting now-it-can-be-told history. Over the years, people have asked why | devote so much space to programmer aptitude testing, only to conclude that it is all nonsense. The truth is, the original manuscript dis- creetly ignored this subject, but two reviewers of the draft happened to be in the business of selling such tests. Naturally, they insisted that the book address “the most crucial subject in computer programming.” To satisfy my original publisher, | wrote the parts of this chapter addressing aptitude testing. Today, programmer aptitude testing is a dead issue. If this chapter had any part in killing this ridiculous practice, then | take pride in having done my job. Yes, intelligence is a factor in programming success—people with an IQ less than 50 are probably not going to create operating systems (although they might design new languages—the Devil made me say that). There are many factors besides intelligence that contribute to programming success (or failure). I’ve Known some rather dull people who consistently turn out fine, useful code and some rather brilliant people who never turn out any useful code ai all. In short, I’m sticking with my twenty-five-year-old position: “. . . intelli- gence has less to do with the matter than personality, work habits, and training. These things, unlike intelligence, can be changed by experience later in life, which turns the problem from one of selecting programmers to creating them” (p. 176).

Silver Anniversary Notes <i By the words “creating them,” | don’t mean forcing them to follow some “best” style of thinking. My own doctoral thesis showed quite clearly that problem-solv- ing style was unique to the individual.1_ Thus, anything that forces a programmer to think in another person’s style reduces that programmer’s ability to solve prob- lems. The greatest challenge, then, is not creative thinking, but creative commu- nicating: representing our thoughts in a way that other persons—each with a unique style—can understand. 1 Gerald M. Weinberg, Experiments in Problem Solving (Ann Arbor, Mich.: University Microfilms, 1965).