Book-Reader-Assist/library/The Psychology of Computer Programming/Chapter-03-How-Can-We-Study-Programming/Chapter-03-source-text.md

The Psychology of Computer Programming — Chapter-03: How Can We Study Programming?

• Source: /library/The Psychology of Computer Programming/source-file.pdf
• PDF pages: 57–76
• Pages without text: none
• Note: Includes the Silver Anniversary comments that immediately follow the original chapter. Text comes from the scan's OCR layer and may contain recognition errors.

---

HOW CAN =A WE STUDY PROGRAMMING? n the first two chapters we have tried to set the stage for a study of pro- gramming as human behavior. We showed that although programming is a form—a complex form—of human behavior, few people have studied programming from this point of view. But perhaps there is a reason why programming has not been so viewed? Perhaps programming is too complex a behavior to be studied and must remain largely a mysterious process. Human knowledge is by necessity incomplete. We cannot know in advance what we might be able to know and what might be essentially unknowable. But of one thing we can be sure: if we do not try to find things out, we shall never succeed. To those readers who despair of success in this venture, | can only suggest that they reread the Einstein quote at the beginning of this part of the book: “The important thing is not to stop questioning.” 27

28 Programming as Human Performance How, then, shall we begin our questioning? Since programming is human behavior, it would seem wise to look to the sciences of human behavior for two kinds of information: results that we can apply directly to programming, and methods that we can use to get information not available directly. Of course, these methods may not be appropriate for the study of programming, and in such cases we shall have to invent new methods, something that all social scientists are doing at all times. But let us begin by seeing what types of methods are generally available to us. INTROSPECTION Although some modern students of human behavior tend to discredit it as nonscientific, introspection has always been the first foundation stone of their science. We may quote, for example, Sigmund Freud:* Psychoanalysis is learnt first of all on oneself, through the study of one’s own personality. This is not exactly what is meant by introspection, but it may be so described for want of a better word. There is a whole series of very common and well-known mental phenomena which can be taken as material for self-analysis when one has acquired some knowledge of the method. Or we may quote William James,** the founder of American psychology: It is in short obvious that our knowledge of our mental states infinitely exceeds our knowledge of their concomitant cerebral conditions. Without introspective analysis of the mental elements of speech, the doctrine of Aphasia, for instance, which is the most brilliant jewel in Physiology, would have been utterly im- possible. Now admittedly these are quotes from the early days of psychoanalysis and psychology, but that is precisely why their precedents are so im- portant for us when we are embarking on the study of a new area of human behavior. Not that we intend to stop at introspection—if Freud and James had done that, their names would not be remembered today. But at the beginning of a new field or subfield, there is a great deal of information to be gleaned from the heads of people without elaborate precautions or experimental methods. What we need at the beginning are ideas as to what general directions to take, not detailed logistic plans for the journey. What kinds of insights can introspection yield? Several examples are given in the text, but one at this point should serve to show the uses and abuses of the method. Some years ago a student of mine, in doing a classroom problem in PL/I, was having an inordinate amount of difficulty “Sigmund Freud, A General Introduction to Psychoanalysis. By permission of Sigmund Freud Copyrights, Ltd., London England. ™ William James, Psychology, Cleveland, The Living Library, 1948.

How Can We Study Programming? 29 with the statement shown in Figure 3-1. When he finally finished the program, | asked him to sit down with me and recall the thought processes —the difficulties—he had experienced with this statement. This student came up with the following list of problems:

1. He had trouble getting the parentheses matched, because there were so many of them.
2. The original ANGLES(I) was a structure, not an array, so the com- piler complained of an illegal subscript list, but he had not been able to find the problem because there were so many subscript lists.
3. When the program finally got into execution, there were these further difficulties: a. IND was a matrix on another page of the listing, and he had trouble finding it. b. Even though the parentheses were now matched, one pair was in the wrong place and was most difficult to find. c. The last difficulty turned out to be a problem of precision caused by capricious declaration of different data types, by the large ex- pression, and by the division—which would not have been neces- Sary if the two-argument form of ATAND had been used. Now, a single case such as this contains insights into many problems in computing: the proper size of statements, the choice of data structures, the arrangement of different parts of a program, the use of parentheses rather than other techniques for decomposition, the design of compiler and execution-time diagnostics, and techniques for learning and teaching programming. None of these insights could be obtained without intro- spection of this sort, yet thousands of programmers each day have these same problems and simply grumble that they are having “bugs.” Without the introspection, they will simply continue to have ‘‘bugs’’ until, in the end, the bugs will have them. However, even though we must start with the introspection, we can- not stop there, for then we would be practicing magic or religion, not science. Based on one example such as this, we can hardly conclude such general laws as:
4. The mind cannot cope with more than five levels of parentheses.
5. Compiler diagnostics should be more explicit.
6. PL/I precision rules are too complicated to use. ANGLES( 1 )=2ATAND(SQRT((S-ACINDO1,1)))#(S-ACIND(1,2))))/ (S(S-ACINDOI,3)))))); Figure 3-1 A long PL/I statement.

30 Programming as Human Performance Although each of these statements might turn out to be a “law” in the psychology of programming, writing it on the basis of a single intro- spective example hardly qualifies it as a law. To be a “law,” a principle must be explored so as to set limits to its application, because all laws have limits. Indeed, the limits are usually more important to know than the laws themselves, and it is only by investigating a multitude of cases that limits can be set. Thus, without introspection, investigation would be sterile, but without investigation, introspection will be of questionable value in application. OBSERVATION One way to follow up introspection is by observation of what people actually do, aS opposed to what they think they do. If, for instance, we want to study the question of levels of parentheses, we could observe the number of levels that other people use, and the difficulties they seem to have in using them. One of the problems with observation, how- ever, is that it tells us what people do do, but not necessarily what they can do. Thus, if we observed hundreds of programmers and never found any using more than five levels of parentheses, we could not conclude that people are unable to use six levels. Nevertheless, observations of this kind can lay to rest certain conjectures about what people cannot do, since if we find people successfully using six and more levels of parentheses, we settle the issue. Or at least we settle some issue. A second problem with observation is deciding just what it is we are observing. A programmer doesn’t use six levels of parentheses in just any context—there may be situations in which he really cannot use six levels successfully. Once we have observed the six levels, our work has only begun; next we have to de- limit the circumstances under which such behavior can be found and under which it cannot. How far we have to delimit depends upon the use we intend to make of the observation. For example, if we are de- signing a programming language and we would like—for reasons of efficient compilation—to limit the depth of parentheses’ nesting to five, we would like to have estimates of how often somebody would be in- convenienced by not using six. We would not necessarily have to know why he was using six. We must be extremely careful in making observations such as this, however, because programming is extremely complex behavior. Many languages, for instance, limit the number of dimensions that an array may have to three—not on any psychological grounds, but only on the casual observation that nobody seems to use more than three. Such an observation may be faulty for a number of reasons other than hap-

How Can We Study Programming? 31 hazardness. In the first place, if the programmer is working in a language that allows only three dimensions, we are not likely to observe more than three. Secondly, other features of the language may prevent the user from making effective use of more than three dimensions, even though the language permits them. Perhaps the printing format for higher arrays is not well chosen, or perhaps the specification of sub- scripts is so clumsy as to be confusing beyond three. The point, then, is that we must be most careful in trying to carry observations from one context over into another, because the contexts will always differ in several ways. A third problem with observation is the problem of interference of the observer with the subject being observed—a kind of uncertainty principle. Here we have much to draw on from the social sciences, which have long been concerned with this problem and have extracted such phe- nomena as the ‘Hawthorne Effect.” This effect was named after the Hawthorne Works of the Western Electric Company, where between 1924 and 1927 certain experiments in industrial psychology failed. They failed because no matter how conditions of work changed, productivity rose. Eventually, the experimenters came to realize that the missing factor was the pride the workers took in being subjects of all this attention. It was the act of observation itself which was producing the phenomenon being observed. The interference of the observed and observer occurs not only in in- dustrial psychology, of course, but in all sciences that study human behavior. The anthropolgists are constantly plagued in their field studies by the gnawing fear that the behavior being observed is that behavior appropriate for a visiting anthropologist. Through the method of “par- ticipant observation,” the anthropologist tries to make himself ‘‘invisible” to the people by becoming so much a part of the culture that he is not noticed, so that the culture can go on as if no outsider were there. The method of participant observation is certainly one which can be used in the study of programming; and our students, who are often working programmers, have made good use of it in obtaining some of the data reported in this book. Another way of being “invisible” is to observe in ways in which the people observed actually have no possibility of knowing that they are under observation. Many such methods exist, and they may all be Classified under the rubric of ‘‘unobtrusive measures.” There is no need for us to go into detail about such measures here, as a number of books and articles discuss not only the methods but the ethical and moral problems in using them. But we should take time to discuss one type of unobtrusive measure which is more or less unique to computer studies— the direct recording of user behavior by the computer. In a way, any computer system that does self-accounting is doing un-

32 Programming as Human Performance obtrusive observation of programming behavior. A number of explicit studies have been made, however, particularly of time-sharing systems to see how users behave at terminals. (See Sackman,* 1970, Chapter 2, for a summary of this work.) These studies have been in the main statistical, but because the computer records data in such detail, there is no reason why individual users cannot be studied, even over long periods of time. In fact, it is the enormous amount of detail that has forced most of these studies to be statistical, and one of the major problems faced by the would-be observer is what to throw away. One man sitting at a terminal for one hour generates considerably more data than a psychologist can fruitfully analyze in one hour, and when fifty users are sharing the system, the situation is certainly not improved. However, we must not be lulled by this enormous quantity into believing that such data, by its sheer bulk, must of needs contain the information we want. Although it is convenient to use the normal accounting or logging information generated by a system, it will usually be necessary to provide some hardware or software to get what we are truly after in a careful study. For example, the timing information needed for account- ing will usually not suffice for psychological purposes. In the first place, the units timed may not be the units we wish to study, and few systems log a time with every transaction. Secondly, the resolution level of the timing for accounting purposes may be adequate to a second or more, but for many psychological studies, we may want to get down to milliseconds. EXPERIMENT One way to reduce the cost of the bulk of data produced by observation is to design experiments. Through the use of experiments, fewer data can produce more information about the aspects of behavior which in- terest us. One of the first dangers of experiments, however, is precisely that they may be so refined that they bypass the most interesting data. The experimental situation might so constrain the subject’s behavior that we will never see what we saw when observing him in a natural setting. The constraint influencing the subject’s behavior may be either planned or unplanned. To take one example, Sackman cites studies comparing on-line and batch systems where the batch system is simulated by running the jobs on-line but holding the work for a fixed period before delivery to the user. The purpose of this simulation was to ensure that the two groups of programmers were working in the same language and had the same system facilities available.

• See the bibliography at the end of the chapter.

How Can We Study Programming? 33 A batch system simulated in this manner, however, may differ from a true batch system. For instance, the language used is tailored to the needs of terminal users. Some terminal-oriented system commands cCan- not be used, but are not replaced by batch-oriented commands. Diag- nostics are oriented to terminal use, and corrective measures that might be provided in a batch system are probably not present, since it is easier for a terminal user to make his own corrections. Finally, a more subtle effect may be introduced by the very regularity of the turnaround. A normal batch system does not give jobs back precisely two hours after they are submitted, and the variance in the turnaround is a major in- fluence on user behavior. By inadvertently eliminating the variance, the experimenters may have biased the study in one direction or the other. Only another study could tell which. Another study! Aye, there’s the rub! Studies cost money, and in pro- gramming they cost more than in other areas of human behavior. For instance, in one small study, we used one-quarter time of nine ex- perienced programmers for three months. Their average salary was $14,000, which with modest overhead came to say, $20,000 apiece, or $1250 apiece over the course of the study. Add to this the cost of the experimenters’ salaries, the computer time, and various supplies and secretarial services, and the bill comes easily to $30,000—for nine sub- jects. In. the classical psychological study, costs do not run so high. In the first place, computers are not used—though the modern trend is in that direction. More important, however, is that subjects are usually free. As one wag put it, ‘psychology is the psychology of 18-year-old college freshmen.”” What he meant was that the typical source of subjects is the freshman psychology course in a university, where one of the re- quirements for credit is participation as a subject in one experiment. Normally, such subjects are not paid at all, let alone $14,000 a year. In selecting subjects, inadvertent constraint may slip in. Whereas psychology may be the psychology of college freshmen, the psychology of programming could easily become the psychology of programmer trainees. And for the same reason—cost. Trainees are paid less, but, more important, they are not doing anything very critical, as experienced programmers seem to be. Thus, they are far more easily obtainable for experimentation, regardless of their suitability as subjects for a particular experiment. Later on, we shall discuss the question of professional versus amateur programming, and show how this confusion has always dom- inated and befuddled the thinking about programming. If, in the psy- chological study of programming, we insist on using trainees as a matter of cost or convenience, we shall only compound these confusions and give them the weight of ‘‘scientific evidence.”

34 Programming as Human Performance Undoubtedly it will turn out that some experiments can be done equally well on trainees, but the burden of proof is on the experimenter. More often, the results obtained from trainees will be limited in their application, so that they may be useful only for the design of training programs. In one study, we tried to determine the difficulty of using OS/360 Job Control Language—then, as now, the source of many com- plaints. In testing the class of OS/360 beginners, we found that the dif- ficulty did indeed exist, but more or less disappeared completely upon completion of the course of study because of the use of carefully de- signed exercises. If we had simply tested beginners and stopped at that, we would have indeed concluded that OS/JCL was very difficult to use, instead of the proper conclusion—that it is very difficult to /earn. Once the learning has taken place—whether through a course or through bitter experience—JCL is no more difficult to use than any other rather rigid programming language. Although the language perhaps should be redesigned, more fruitful results would probably be obtained from im- proved training methods. If we had just tested experienced programmers, we might well have wondered what the fuss was all about. Only by testing both groups, or—as we did—by following a group of trainees until they became experienced, could we put the problem of JCL in the proper perspective. Whether we use experienced programmers or trainees, however, we are always faced with another difficulty which these simple titles tend to conceal. In our study of JCL, some of the students were relative novices at programming, and others were rather old hands. All, however, were novices at JCL. Thus, if we had simply correlated our results with “years of programming’ as a measure of “programming experience,” we should have had no results at all. When the group was tested at the beginning of the course, experienced and inexperienced alike did poorly; at the end of the course, the groups did equally well. Thus, it was not “experience” in general that mattered in this problem, but the specific experience with JCL. The difficulty with simple measures of experience or inexperience is illustrated by an example taken from one of Sackman’s studies with “experienced” programmers. There, experience was measured, among other ways, by years of programming, ranging from 2 to 11 years. On one of the problems, the best and worst performance was obtained by the two 11-year programmers in the study. Obviously, there are factors at work other than sheer length of time in “programming.” It may be that the experiences in programming are so diverse as to make length of time a fruitless measure, or it may be that people don’t all learn the Same things from the same experiences. Before we leave the subject of experience and experiments, we must

How Can We Study Programming? 35 deal with one other problem, which is important because of a seldom- questioned view of programming—a view which this book will spend a great deal of time in questioning. That view is that programming is an individual activity, not a social activity. To the extent that experimenters have believed this, they have chosen the individual programmer—ex- perienced or not—as the suitable object for study. But there is much evidence, as we shall see presently, to argue that the proper study of programming is done at the level of the programming social unit. Not that the individual level is unimportant, but we might start by asking why, if the average programmer spends two-thirds of his time working with other people rather than working alone (yes, it’s true!), that 99 percent of the studies have been on individual programmers. One answer, of course, is that if studying programmers is expensive, studying groups of programmers is extravagantly so. Moreover, not just any group of programmers will do—not, for example, a collection of trainees put into a “team.” Putting a bunch of people to work on the same problem doesn’t make them a team—as the sloppy performance in all-star games should teach us. And furthermore, even studying teams as they are constituted today may not be sufficient, for these are teams which have grown up in an environment pervaded by the myth that programming is the last bastion of individuality. The neglect of social effects also casts doubt upon all individual studies, since these studies force the individual to separate himself from his normal working environment. In one of our studies, one of the programmers came to me when he had finished coding and asked who could check his work. In his home group, this was the standard practice at this point in development, but it was not allowed during the study —lest it should ‘“‘invalidate’’ the results. Looking back, it seems that forcing programmers to work in isolation is like trying to study the swimming behavior of frogs by cutting off their legs and putting those legs in the water to watch them swim. After all, everyone knows that the frog swims with its legs, so why complicate the experiments with all the rest of the frog? PSYCHOLOGICAL MEASUREMENT The use of “years of programming” as a measure of ‘“‘experience’”’ is but one example of the problems of measurement faced by programming psychologists. In any observation or experiment involving human behavior, there are literally thousands, if not millions of variables, which might be measured. We can record age or agility, blink rate or birth order, Cranial capacity or church preference, deductive reasoning or degrees

36 Programming as Human Performance awarded, electrocardiogram or elementary mathematics, frown rate or father’s profession, gestures made or grade-point average, height or hatreds, ideas or ideals, jitteriness or jargon, kindliness or knowledge, left-handedness or latent homosexuality, mother’s tidiness or muscle tension, native tongue or nervous tic, opinions held or objects touched, preferred foods or pupillary motion, questions asked or questions answered, response time or requests for assistance, source of support or slips of the tongue, time of day or tap of the foot, university degree or unconventional behavior, vocabulary or voting record, weight or wage, X-ray photographs or xenophobic index, yawns or years in college, zygosity or zodiacal sign. Surely the psychologist is the snoopiest of all the scientists. However, that is just the problem. Of all the sciences, psychology seems to have the most possible things to measure—more than any reasonable man can comprehend. But is this surplus unique to psychology? Not really, except that psychology—and the other behavioral sciences—is in its infancy. Maxwell, the great physicist, once said, ‘‘To Measure is to Know,’ and his words are often taken as a motto by other sciences. What Maxwell probably meant was ‘To know how to measure is to know,” or even better, ‘To know what to measure is to know.” In physics, the number of possible measurements seems smaller because physicists have learned, through long experience, what is worth meas- uring and what is not. Knowing what is worth measuring, the physicist can narrow his experimental field to those matters. In a sense, physics is the science of those things that physicists can measure. When we study human behavior, however, we are not blessed with this simplification. We do not define our subject by what we can measure, but by what we would like to know about—human behavior. We then must search among many possible measurements in hopes that one or a few will give us some insight which we can carry back to the problem with which we began. No clue is too tiny to be ignored in this search, no suggestion ridiculous a priori. Take the ‘‘zodiacal sign’ we mentioned earlier. Did that stand out as the one item on the list which was not legitimate? But how do we know? Not that we are arguing for astrology, but medical scientists, for instance, have found it prudent to be a bit humble about folk medicines, and the same prudence should be exercised in the study of human behavior. In fact, several recent studies have purported to show correlation be- tween birth month or season and certain psychological and physical variables. Whether they prove durable observations or simply spurious is beside the point for the present argument. What is important is to see how serious workers cannot rule out any variables in advance without

How Can We Study Programming? 37 risk of error. And since we are working so much in the dark, we most certainly cannot afford to pass up the “folk wisdom” or our subjects, even if they ultimately prove groundless, or even outright wrong. Folk wisdom is but one source of insight. Another is the existence of a tool presently used for some other purpose. Since the tool exists and something, at least, is known about it, we must allow it to influence the type of study we make. Thus, for instance, since “intelligence” tests exist, one of the first things we do is try them out on programmers, hoping to find some relationship between what they measure (‘intelligence’) and what we want to understand (“good programming’’). If we are lucky, we will find some relationship. The most normal course is to go the other way—from problem to tool. Sackman’s work is one example of this progression—he wanted to know about the efficacy of time-sharing systems, so he constructed a number of studies and measured numerous variables. Another example is the ACM Special Interest Group on Personnel Research. This loose federation is interested mainly in programming from the point of view of management, and, more particularly, personnel management. Thus, their studies tend to be in the area of selection and training, and the things they measure reflect this interest. Much of our own interest has been in the design of programming languages, and the kinds of measurements we make are influenced as much by that interest as by anything else. However, when you don’t know very much about a subject, you don’t know in advance what to measure. So your early experiments are not what they might seem to be—they are not to measure things, but to determine what can be measured and what is worth measuring. The difference here is well exemplified by the difference between the method- ologists of anthropology and of sociology. The sociologist is working in his own culture, about which he assumes that he is sufficiently knowledge- able to construct, say, a questionnaire as a measuring instrument. But the anthropologist cannot use a questionnaire because he does not know what questions will be meaningful in another culture. Roughly, then, we might say that the sociologist is looking for answers and the anthropologist is looking for questions. For the present, most of the work in the psychology of programming is going to have to be “looking for questions.” Although we may use standard psychological measuring instruments—surveys, tests, and the like—we are really exploring the instruments rather than the subjects. One of the consequences of this point is that it will restrain us from publishing such specious measurements as “the correlation between test score and experience was 0.72.” Although we may certainly define “test score’ to whatever number of decimal places we wish, we certainly

38 Programming as Human Performance have no way of measuring ‘experience’ to two decimal places, or even an order of magnitude. Thus, the “2” in 0.72 is of the nature of noise— and so, probably, is the “7”. We haven’t the space here to survey the entire field of psychological measurement or psychological statistics, but we have given several fine references. Nobody, but nobody, should attempt to draw hard conclusions from such data as are being obtained in this field without studying the pitfalls which the behavioral scientists have already experienced. We shall have occasion to comment on specific instances as we proceed with our presentation. USING BEHAVIORAL SCIENCE DATA Probably the main use of previous experience in behavioral science for us is in the field of methodology, as we have discussed. Because of the nature of programming, there is not much hope that actual results can be transferred directly from other fields to ours. Our use of results has to be for insights, not for answers. Where are we most likely to find our insights, and how? The best plan is to avoid getting fascinated with theories that are too specific, for they will ultimately turn out to be based on premises which programming does not satisfy. For example, there has been a great deal of psychologi- cal work on the subject of ‘problem solving” but this work in its details is not applicable to programming because the “problems” which are “solved” are simply too simple. They have had to be simple because almost all of the experimenting done in this area is based on the idea that there is one right ‘“‘solution’” to a ‘‘problem’’—and that the psycholo- gist knows it! In programming, however, there is no right ‘“‘solution’—we can be sure of that. But even if there were, we could be sure that we would not know it any better than our subjects. Time after time, the student or novice comes up with “solutions” which are superior in some regard to the teacher’s or veteran’s. Therefore we cannot draw directly on the problem-solving literature for results, but we can certainly look there for insights. The abundance of areas from which useful insights will be available makes it mandatory to impose a certain amount of organization on our presentation. Organization can help us to remember important ideas and suggest, by defects in the structure, other areas where aid might be found. We should try, however, not to take our organization too seriously, for it is only a convenience—a tool—and not some set of truths passed down on stone tablets from the mountain top.

How Can We Study Programming? 39 The social science that provides us with the most useful overall model for computer programming is anthropology. With a little artistic license and stretching of the imagination, we could imagine computer pro- grammers as having a culture—a shared set of beliefs and activities which shape their day-to-day activities. In our study of programming, we shall first examine the “social structure’ of that culture—the way pro- grammers relate to one another and to other people who are not pro- grammers. We shall find some surprising possibilities for improvement in this area over present practices, more even than in the second area of study—programming as an individual activity. Although stretching a bit to make the analogy, this study is related to the personality and culture studies of the anthropologist. We shall see how the individual lends his individuality to his program and how programming lends shape to the programmer himself. Whereas in the social structure discussion we shall draw heavily on social psychology, here we shall draw upon the results of individual psychology—especially the areas of problem solving and personality. Once these two sections have prepared the way, we should be in a proper frame of mind to examine the programmer’s tools—his languages, operating systems, and other devices—from a psychological point of view. These tools are the “material culture’ of programming—the artifacts which the programming archeologist would find and study to understand the dead civilization of programming. Also, as programmers are not born into this culture, we shall study how they are—or should be—recruited, and how they learn this complex way of life. Here the psychology of learning—and especially language learning—shall play a large, though not exclusive, role. Finally, as we pass successfully through all this, we may find ourselves in a sufficiently objective frame of mind to examine some of the myths of programming—the articles of programming faith. Faith, as Bertrand Russell pointed out, is the belief in something for which there is no evidence; and myths, as Ambrose Bierce once defined them, are the sacred beliefs of other people. Perhaps we shall simply carry away our old beliefs unchanged, but if just one unfounded programming myth should die as a result, this book will have been worth the effort. SUMMARY Because programming is such a rich and complex activity, we shall need all the richness of methods and results we can borrow from all of the behavioral sciences. But the borrowing process is not without danger, and as we prepare ourselves to make studies in the psychology

40 Programming as Human Performance of programming, we should particularly prepare ourselves to avoid the common pitfalls. Among the major problems are: Using introspection without validation. Using of observations on an overly narrow base. Observing the wrong variables, or failing to observe the right ones. . Interfering with the observed phenomenon. . Generating too much data and not enough information. Overly constraining experiments. Depending excessively on trainees as subjects. . Failing to study group effects and group behavior. 9. Measuring what is simple to measure. 10. Using unjustified precision. 11. Transferring borrowed results to inapplicable situations. ONAAROD But in any fledgling field, perhaps the greatest mistake of all is over- caution. Though we try to observe the proprieties, better to have experi- mented and lost than never to have experimented at all. QUESTIONS For Managers

1. What type of accounting system does your installation use? Do you use data from this system to evaluate programmer performance? Are the data intended for this use, and what can be done to improve the system so that it is more useful to you for this purpose?
2. Would you permit behavioral science experiments to be conducted with your experienced programmers? With your trainees? What would you do if a person objected to participating in such an experiment? What safeguards would you ask of the experimenters?
3. Do you ever read articles on psychology in such magazines as Har- vard Business Review, Reader’s Digest, Playboy, Think, or Psychology Today? Have you ever tried to put some of the reported results to use in your work? What kinds of experiences did you have in transferring data reported from other circumstances to the management of pro- grammers? For Programmers
4. When you have just found a bug, do you ever sit back and try to trace out the paths you took in your mind? Try doing this on the next bug you find, and write a brief report or outline of what you find.
5. Do you keep records on your performance? What kinds of things do you record, and how accurately do you record them? What picture do

How Can We Study Programming? 41 you get from these observations which you could not get by simple reflection on what you are doing? Does your installation accounting routine ever give you insights into your behavior, or the behavior of other programmers in your installation? 3. Would you have any objections to participating in psychological ex- periments to study programming? What kinds of assurances would be » required to overcome your objections? If you have ever participated in: a psychological experiment, write a small report or outline of the most significant things you remember about the experience. Do you see any reasons why the results of the study might not be valid, or valid for programming? Were you ever shown the results of the study, or did you have the reason for the study explained to you? BIBLIOGRAPHY Freud, Sigmund, A General Introduction to Psychoanalysis, Garden City, N.Y., Doubleday, 1953. Of great interest in and of itself, this book is also useful as a historical document on the way in which a new study of human behavior begins. Especially interesting in the present context is the way Freud has to convince his audience that the subject really exists. One of the ways he does this is through his discussion of errors—‘‘slips of the tongue” and the like. This particular discussion is of special interest to programming people. James, William, Psychology, Cleveland, The Living Library, 1948. Certain parts of this classic by the man who was “‘first to extend the methods and spirit of modern science to the human mind’”’ are outdated, but not many. As with Freud, the study of James bears many fruits for the modern student of the psy- chology of computer programming, and these two books might well be the proper starting place for a study of this subject, in order that historical perspective not be lost. Brown, J. A. C., The Social Psychology of Industry, Baltimore, Penguin Books, 1954. A very readable account of the practice of studying working groups, which gives a good historical view of such matters as the Hawthorne Effect. The book is recom- mended as an effective study for those entering the field called Industrial Psy- chology, which, although it may have limited relevance to work on tasks as com- plex as programming, must be studied by anyone seriously involved in this field. At the very least, Brown’s review of the early days of the field will warn us against certain excesses to which we might otherwise be prone. Junker, Buford H., Field Work—An Introduction to the Social Sciences, Chicago, University of Chicago Press, 1960. The author claims—with probably truth—that one cannot properly appreciate field work without experiencing it. Yet by acknowledging his limitations, he does as good a job as he can at disproving that statement, and the reader can obtain some sense of the various field work methods of sociology and anthropology. Golde, Peggy, ed., Women in the Field, Chicago, Aldine, 1970. The “Women” in the title is a bit misleading, since one of the things the book

42 Progra:saming as Human Performance demonstrates is how little difference the gender of a fieldworker actually makes. The book can be read at several levels—either as thoroughly enjoyable anecdotal material or as a comprehensive—if not organized—survey of anthropological field method. Hammond, Phillip E., Sociologists at Work, New York, Basic Books, 1964. A collection of anecdotal material on method, like Women in the Field, but ori- ented to the sociologist rather than the anthropologist. Several of the chapters are about studies, which are in themselves of interest to the psychologist of program- ming: The Dynamics of Bureaucracy, Men Who Manage, and Union Democracy. Webb, Eugine J., et a/., Unobtrusive Measures: Nonreactive Research in the Social Sciences, Chicago, Rand McNally, 1966. A survey of a specialized subset of psychological research techniques. Sackman, Harold, Man-Computer Problem Solving, Princeton: Auerbach Publishers, 1970. Although subtitled—Experimental Evaluation of Time-Sharing and Batch Proc- essing—the book is the first we have on methods of psychological research in programming. As such, it is essential reading for the serious student. Hyman, H. H., Survey Design and Analysis, New York, Free Press, 1955. Possibly the best and most thorough text on the theory and practice of public Opinion surveys. Cronbach, L. J., Essentials of Psychological Testing, 3rd ed., New York, Harper and Row, 1970. An elementary, but therefore useful as an introduction, account of the construction and application of psychological tests of various kinds. Hammond, K. R., and J. E. Householder, /ntroduction to the Statistical Method, New York, Knopf, 1962. One of the better of hundreds of books on psychological statistics, and as good as any as an introduction. Mathematically oriented programmers may be turned off by such books, which are written for the mathematically unwashed, but this one is quite palatable. In any case, it is important to know how psychologists think about quantitative data. Morgenstern, Oskar, On the Accuracy of Economic Observations, Princeton, Prince- ton University Press, 1963. Economics is another behavioral science, and although it does not have much direct bearing on the psychology of programming, we can learn something from its methodological problems. Morgenstern, co-inventor with John von Neumann of Game Theory, brings his steel-trap mind to the question of how good economic numbers really are. But the first half of the book is in no way really specific to economics, and should be required reading for all behavioral scientists. Proceedings of the Nth Annual Computer Personnel Research Conference, Asso- ciation for Computing Machinery, New York. As N has grown, so has the size and scope of these conferences. So also has their availability grown, and they should be known by every researcher in this field. Even though the early issues are difficult to obtain, they are perhaps even more valuable than the later ones in that they relate the earliest struggles to come to grips with problems that are not yet nearly solved, but about which we have become a bit more sophisticated. Parsons, Henry M., The Scope of Human Factors in Computer-Based Data Process- ing Systems, Human Factors, 12, 2 (1970), pp. 165-175.

How Can We Study Programming? 43 The appearance of this article in Human Factors probably signals the beginning of a new interest by human factors engineers in programming, although most of the article is concerned with nonprogramming aspects of computers. Not that non- programming aspects are unimportant—far from it—but only that they are beyond the scope of this book, and would probably require three or four books to survey. The article contains no new data, but gives a few references to work already done.

Silver Anniversary Notes Comments on Chapter 3: How Can We Study Programming? Perhaps programming is too complex a behavior to be studied and must remain largely a mysterious process. (p. 27) After a quarter-century of learning, | don’t think programming is too complex a behavior to be studied, but it may be too complex to be studied in a laboratory. | guess I’ve changed my ideas on how to study programming, or more properly, how / study programming. The academic process is essential, but it grinds too slowly for my mental metabolism. It also grinds too slowly for my current clients, who are interested in robust and practical ideas, but not necessarily in scientific ones. Over the years, I’ve observed that the best programmers are the most intro- spective. If they do something wrong, they examine the mental processes (or physical processes) that led to the problem; then, they do something to change the process. This is what we at Weinberg and Weinberg try to teach organiza- tions, under the title “root-cause analysis,” but this approach is not yet favored by very many people. Too many of them are still concerned with “blame analy- sis,” an approach that is guaranteed to motivate people to behave in ways that conceal root causes. Observation of behavior has long been a major focus of mine, in workshops and in books. In my Problem-Solving Leadership Workshop, for example, I’ve been accused of “only” teaching people to see things they didn’t see before, or

Silver Anniversary Notes <x i to see familiar things in a different way. What I’ve learned is that once they have this new way of seeing, most intelligent people are quite capable of changing what’s wrong and preserving what’s right about whatever they observe. The Hawthorne Effect (that the process of being observed often motivates people to better performance) can be the bane of scientific observation. Over the years, however, I’ve learned to use the Hawthorne Effect to great practical advantage in my consulting business. A striking example of this advantage occurred a few years ago when | observed a client’s technical review to help its developers determine what was going wrong with their process. The review | observed was highly productive, but the participants said it wasn’t typical. | asked them what might have caused it to be different, and they concluded it was because | was watching. With me watching, they were too embarrassed to behave badly. So, they took my photograph and enlarged it to make a life-size cardboard cutout of me to attend all their future reviews. It seems to be working. From this and other observations, I’ve learned a basic principle of the psychol- ogy of management: Managers who pay attention to people get good results. Many managers (especially those who were once programmers) want their peo- ple to behave like modules of code. These managers think people should func- tion as little black boxes that tasks are fed into and work comes out of, with no observation necessary, and especially no interaction.1 Many software develop- ment managers simply hate working with their staff, mostly because they have never received any training on how to do that task. Simply by observing, I’ve also found some new, rich sources of information that is just lying around. For example, by studying records of phone calls to the service line of several software companies, I’ve identified some strange new pat- terns of human behavior. I’ve also learned that the diversity of human behavior around computers is far greater than even | could ever have imagined twenty- five years ago. Experiment involving human behavior, however, has proved as costly and diffi- cult as | predicted. Most studies are carried out in university settings, and, though often fascinating, have had slow and minimal influence on actual pro- gramming practice. Even if the work is vital, and is worthy of recognition, the researchers must turn it into a worthy piece of communication. Instead, the implicit communications that universities send students generally run contrary to the university’s own research results. Many college-trained developers approach a project as they would a one-semester class assignment. They think about it or ignore it for a while, then suddenly start jamming on the keyboard to produce something that will earn a passing grade. Software projects done at 1 See Gerald M. Weinberg, “Overstructured Management of Software Engineering,” reprinted in Software State-of-the-Art: Selected Papers, eds. Tom DeMarco and Timothy Lister (New York: Dorset House Publishing, 1990), pp. 4-13.

WS Silver Anniversary Notes universities generally don’t have to be maintainable, usable, or testable by another person. Professional software needs to do more than show that the developer has a grasp of computer science concepts, and | salute those college professors who demand more of their students’ projects. Individual variation is, as it was twenty-five years ago, the bane of project pre- dictability. The social nature of team programming can be used to average out this variation—but such averaging prevents us from getting experimental infor- mation about individual programmers. The devastating cost of individual varia- tion on real projects has supported the validity of my prediction that the individual working alone is neither a fruitful unit of study, nor a productive component of programming project work. On page 36 in this chapter, | wrote that Maxwell said, “To Measure is to Know.” Nowadays, everyone seems to want to attribute some version of this quote to Lord Kelvin, perhaps to add the prestige of peerage. This desire to associate Lord Kelvin with the importance of measurement puzzles me, since Lord Kelvin also asserted that airplanes would never fly. A lot of attention has been given to measurement, but not much measurement is devoted to understanding how programming happens. Instead, much mea- surement seems to be designed to control individual programmers without inter- acting with them—that is, to force them to work according to some model that someone else thinks is the right way for them to program. This is not always a good idea. Perhaps this idea comes because we are still stuck with the belief that there is one and only one solution to a given programming problem—and that the teacher knows what it is. Lots of training is still based on this erroneous belief, but we are eroding the fringes. “With a little artistic license and wade of the imagination, we could imag- ine computer programmers as having a culture—a shared set of beliefs and activities which shape their day-to-day activities” (p. 39). So | wrote in 1971, and over time, cultural patterns have emerged—both in fact and in our conscious- ness. | have developed the idea of software culture in my Quality Software Management series, because | think culture is one of the few important ideas for the next twenty-five years. ,