31KB
Developing Your Own 32-Bit Operating System — Chapter-02: General Discussion and Background
- Source: /library/Developing Your Own 32-Bit Operating System/source-file.pdf
- PDF pages: 35–44
- Pages without text: none
General Discussion and Background
H Part I ■ Introduction This section discusses the actual chore of writing an operating system, as well as what may in- fluence your design. I wrote an operating system over a period of almost five years. Things change a lot in the computer industry in five years, but I'm happy to find that many of the concepts I was interested in have become a little more popular (small tight kernels, simplified memory management, client/server designs, and so on). Reading this chapter will help you understand why I did things a certain way; my methods may not seem obvious to the casual observer. I also give you an idea of the steps involved in writing an operating system. Where Does One Begin? Where does one begin when setting out to write a computer operating system? It's not really an easy question to answer. The history of MMURTL and an explanation of why certain de- cisions were made, however, can be valuable to you if you intend to dig into the source code, write your own operating system, or even just use MMURTL the way it is. A friend of mine and I began by daydreaming over brown-bag lunches in a lab environment. We bantered back and forth about what we would build and design into a microcomputer operating system if we had the time and inclination to write one. This lead to a fairly concrete set of design goals that I whittled down to a size I thought I could accomplish in two years or so. That was 1989. As you can see, Fud's Law of Software Design Time Estimates came into play rather heavily here. You know the adage (or a similar one): Take the time estimate, mul- tiply by two, and add 25 percent for the unexpected. That worked out just about right. Five years later, here it is. I hope this book will shave several years off of your development effort. In Chapter 1, “Overview,” I covered what my final design goals were. I have tried to stick with them as closely as possible without scrimping on power or overcomplicating it to the point of it becoming a maintenance headache. Far too often, a piece of software becomes too complex to maintain efficiently without an army of programmers. Usually software also tends to ex- pand to fill all available memory (and more!). You would not let this happen. You Are Where You Were When The heading above sounds a little funny (it was the title of a book that was popular a few years back, and it is borrowed with great respect, I might add). It does, however, make a lot of sense. MMURTL's design is influenced by all of my experiences with software and hardware, my schooling, my reading of various publications and periodicals, and what I have been introduced to by friends and coworkers. Even a course titled Structured Programming with FORTRAN 77, which I took online from the University of Minnesota in 1 982, has had an effect. I have no doubt that your background will have a major effect on your own design goals for such a sys- tem. Borrow from mine, add, take away, improve, and just make it better if you have the de- sire. What I don't know fills the Library of Congress quite nicely, and “where you were when”
Chapter 2 ■ General Discussion and Background will certainly give you a different perspective on what you would want in your own system. Every little thing you've run across will, no doubt, affect its design. My first introduction to computers was in 1976, on board a ship in the U.S. Coast Guard. I was trained and tasked to maintain a Honeywell DDP-516 computer system. It had a “huge” 8K of hand-wound core memory and was as big as a refrigerator (including ice maker). It was a 32-bit system, and it was very intriguing (even though punching in hundreds of octal codes on the front panel made my fingers hurt). The term “register” has a whole new meaning when you are looking for a bit that intermittently drops out in a piece of hardware that size. It whet my whistle, and from there I did everything I could to get into the “computer science” end of life. I bought many little computers (remember the Altair 8800 and the TRS-80 Model I? I know, you might not want to). I spent 20 years in the military and took computer science courses at schools when time allowed. In 1981 I was introduced to a computer system made by a small company named Convergent Technologies (since swallowed by Unisys, a.k.a. Burroughs and Sperry, merged). That system was an Intel 8086-based system and ran a multitasking, real-time operating sys- tem from the very start (called CTOS). Imagine an 8086-based system with an eight-inch, 10MB hard drive costing $20,000. This was a good while before I was ever introduced to PC-DOS or MS-DOS. In fact, the convergent hardware and the CTOS operating system reached the mar- ket at about the same time. After I got involved with the IBM PC, I kept looking for the rich set of kernel primitives I had become accustomed to for interprocess communications and task management, but, as you well know, there was no such thing. This lack was disappointing because I wanted a multitasking OS running on a piece of hardware I could afford to own! In this early introduction to CTOS, I also tasted “real-time” operation in a multitasking environment. The ability for a program- mer to determine what was important while the software was running, and also the luxury of changing tasks based on outside events in a timely manner, was a must. There was no turning back. Nothing about my operating system is very new and exciting. The messaging is based on theory that has been around since the early 1970s. It is just now becoming popular. People are just now seeing the advantages of message-based operating systems. Message-based, modular micro kernels seem to be coming of age. The Development Platform and the Tools My desire to develop an operating system to run on an inexpensive, popular platform, com- bined with my experience on the Intel x86 processors, clinched my decision about where to start: It would definitely be ISA 386-based machines (the 486 was in its infancy). The next major decision was the software development environment. Again, access and familiarity played a role in deciding the tools that I would use. MS-DOS was everywhere, like a bad virus. I had two computers at that time (a 386DX and a 386SX) still running MS-DOS and not enough ra
Part I 1 1 Introduction memory or horsepower to run UNIX or even OS/2, which at the time was still 16 bits anyway. I thought about other processors. I purchased technical documentation for Motorola's 68000 series, National Semiconductor's N80000 series, and a few others. The popularity of the Intel processor and the PC-compatible systems was, however, a lure that could not be suppressed. As you may be aware, one of the first tools required on any new system is the assembler. It's the place to start. Utilities and compilers are usually the next things to be designed or ported to a system. Not having to start from scratch was a major advantage. Having assemblers available for the processor from the start made my job a lot easier. Of course, in the process of using these other tools, I found some serious bugs in them when used for 32-bit development. The 32-bit capabilities were added to these assemblers to take advantage of the capabilities of the new processors, but not many people were using them for 32-bit development five or six years ago on the Intel-based platforms. Operating system developers, people porting UNIX systems, and those building 32-bit DOS extenders were the bulk of the users. I began with Microsoft's assembler (version 5.0) and later moved to 5.1. Two years later I ran into problems with the Microsoft linker and even some of the 32-bit instructions with the as- sembler; these problems pretty much forced the move to all-Borland tools. Then those were the 2.x assembler and 2.x linker. I'm not a Borland salesman, but the tools just worked better. There were actually several advantages to using MS-DOS in real mode during development. Being in real mode and not having to worry about circumventing protection mechanisms to try out 32-bit processor instructions actually lead to a reduced development cycle in the early stages. The Chicken and the Egg The egg is first... maybe. Trying to see the exact development path was not always easy. In fact, at times it was a quite difficult. I wanted the finished product to have its own develop- ment environment — including assembler, compiler, and utilities — yet I needed to have them in the MS-DOS environment to build the operating system. I wanted eventually to port them to the finished operating system. How many of you have heard of or remember ISIS or PLM-86? These were some of the very first environments for the early Intel processors — all but ancient history now. It was definitely a “chicken-and-egg” situation. You need the 32-bit OS to run the 32-bit tools, yet you need the 32-bit tools to develop the 32-bit OS. It can be a little confusing and frustrat- ing. Again, MS-DOS and the capability for the Intel processors to execute some 32-bit instruc- tions in real mode (while still in 16-bit code segments) made life a lot easier. I could actually experiment with 32-bit instructions without having to move the processor into protected mode and without having to define 32-bit segments. Memory access was limited, but the tools worked. I even put a 32-bit C compiler in the public domain for MS-DOS as freeware. It was limited to the small-memory model, but it worked nonetheless.
Chapter 2 ■ General Discussion and Background I really do hate reinventing the wheel; however, in the case of having the source code to an assembler I was comfortable with, I really had no choice. Some assemblers were available in the “public domain” but came with restrictions for use, and they didn't execute all the instructions I needed anyway. I even called and inquired about purchasing limited source rights to some tools from the major software vendors. Needless to say, the cost was prohibitive. (They men- tioned dollar figures with five and six digits in them. . .. Now that's a digital nightmare.) This lead to the development of my own 32-bit assembler. Developing this assembler was, in itself, a major task. That set me back four months, at least; but it gave me some serious insight into the operation of the Intel processor. The source code to this assembler (DASM) is included with this book on the CD-ROM. The prohibitive costs also led to the development of a 32-bit C compiler, a disassembler, and a few other minor utilities. The C compiler produces assembly language, which is fed to the assembler. There is no linker. That's right, no linker. It's simply not needed. To fully understand this, you will need to read Chapter 28, “DASM: A 32-Bit Intel-Based Assembler.” Assembly language has its place in the world, but there is nothing like a high-level language to speed development and make maintenance easier. I actually prefer Pascal and Modula over C from a maintenance standpoint. I was into Pascal long before I was into C. I think many people started there. Those who must maintain code or read code produced by others understand the advantages, if they get past the “popularity concept.” C still can be made almost unreadable by the macros and some of the language constructs, even though ANSI standards have fixed much of it. C's big strength lies in its capability to be ported easily. Most of the machine-dependent items are left completely external to the language in libraries that you may, or may not, want or need. I don't like the calling conventions used by C compilers — for example, parameters passed from right to left, and the fact that the caller cleans the stack. This is obviously for variable length parameter blocks, but it leads to some serious trouble on occasion. Nothing is harder to troubleshoot than a stack gone haywire. MMURTL uses the Pascal conventions (referred to as PLM calling conventions in the early Intel days). Parameters are passed left to right, and the called function cleans the stack. I mention this now to explain my need to do some serious C compiler work. I started with a public-domain version of Small-C (which has been around quite a while) and took a good look at Dave Dunfield's excellent Micro-C compiler, but I pretty much built from scratch. Some very important pieces are still missing from the compiler, but another port is in the works. The details of CM-32 (C-Minus 32) are discussed in Chapter 29, “CM32: A 32-Bit C Compiler.” m Early Hardware Investigation I taught computer electronics technology for two years at the Computer Learning Center of Washington, DC. I have worked for years with various forms of hardware, including most of the digital logic families (ECL, TTL, and so on). I dealt with both the electrical and timing
H Introduction characteristics, as well as the software control of these devices. The thought of controlling all of the hardware (timers, DMA, communications controllers, and so on) in the machine was actu- ally a little exciting. The biggest problem was accumulating all of the technical documentation in adequate detail to ensure it was done right. If you've ever looked though technical manuals that are supposed to give you “all” of the information you need to work with these integrated circuits, you know that it's usually not everything you need. You will need to accumulate a sizable library of technical documentation for your target hard- ware platform if you intend to write an operating system or even to port MMURTL to it. I'm not going to tell you I did some disassembly of some BIOS code; I plead the “Fifth” on this. Besides, IBM published theirs. BIOS code brings new meaning to the words spaghetti code. It's not that the programmers who put it together aren't top quality; it's the limitations of size (memory constraints) and possibly the need to protect trade secrets that create the spaghetti- code effect. Following superfluous JMP instructions is not my idea of a pleasant afternoon. I used many different books, documents, articles, and technical manuals throughout the early development of my operating system. The IBM PC-AT and Personal System/2 Technical Reference manuals were a must. They give you the overall picture of how everything in the system ties together. Computer chip manufacturers, such as Chips & Technologies, supply excellent little documents that give even more specific details on their particular implementa- tion of the AT series ISA logic devices. These documents are usually free. All of the ISA-type computer systems use VLSI (Very Large Scale Integration) integrated cir- cuits that combine and perform most of the functions that were done with smaller, discreet devices in the IBM AT system. I keep referring to the IBM AT system even though it was only an 80286 (16-bit architecture). I do so because its bus, interface design, and ancillary logic design form the basis for the 32-bit ISA internal architecture. In fact, as you may understand, the external bus interface on the ISA machines is still 16 bits. It is tied to the internal bus, which is 32 bits for the most part. SX machines still have 16-bit memory access limitations, but this is transparent to the programmer, even to the operating system writer. Many bus designs now out take care of this problem (PCI, EISA, and so on). This made the IBM designs the natural place to start. It was proven that PC-style computer manufacturers who deviated from this hardware design to the point of any real low-level soft- ware incompatibility usually weren't around very long. Even machines that were technically superior in most every respect died horrible deaths. (Remember the Tandy 2000?) MS-DOS really didn't have to worry about the specific hardware implementation because the BIOS software/firmware writers and designers took care of those details. As you may know, however, the BIOS code is not 32-bit (not even on 32-bit machines). It is also not designed to be of any real value in most true multitasking environments. A variety of more recent operat- ing systems go directly to much of the hardware just to get things done more efficiently, any- way (for example, OS/2 Version 2.x, Windows Version 3.x, all implementations of UNIX). I knew from the start that this would be a necessity. “Thunking” was out of the question.
Chapter 2 M General Discussion and Background Thunking is what occurs when you interface 32- and 16-bit code and data. An example would be making a call to a 32-bit section of code from a 16-bit code segment. This generally involves manipulating the stack or registers, depending on the calling conventions used in the destination code, as well as storage on the stack. Some proces- sors (such as the Intel 32-bit x86 series) force a specific element size for the stack. Much of the early development was a series of small test programs to ensure I knew how to control all of the hardware, such as Direct Memory Access (DMA), timers, and the Priority Interrupt Controller Unit (PICU). This easily consumed the first six months. During this time, I was also building the kernel and auxiliary functions that would be required to make the first executable version of an operating system. The second six months was spent moving in and out of protected mode and writing small programs that used 32-bit segments. If you use Intel- based platforms, this is a year you won't have to go through, because of what you get on this book's CD-ROM. My initial thought on memory management centered on segmentation, but not on a fully seg- mented model — two segments per program and two for the OS, all based on their own selec- tors. This would have given me a zero-based addressable memory area for all code and data access for every program. It would have appeared as a flat memory space to all the applications, but it presented several serious complications. The largest complication was that all memory access would be based on 48-bit (far) pointers. The thought was rather sickening, and some initial testing showed that it noticeably slowed down program execution. When the 386/486 loads a segment register, a speed penalty is paid because of the hardware overhead (loading shadow registers, and so on). I was after a “kinder, gentler” environment, anyway. The seg- mented memory idea was canned, and I went straight to fully paged memory allocation for the operating system. It has turned out very nicely, as you will see. One other consideration on the memory model is address aliasing. If two programs in memory must exchange data and they are based on different selectors, you must create an alias selector to be used by one of the programs. In a paged system, aliasing is also required, but it's not as complicated. In Chapter 5, “Memory Management,” I discuss several options you have. To get a simple model up and running took almost two years. This simple model could allo- cate and manage memory, and it had only the most rudimentary form of messaging (Send and Wait). There was no loader, no way to get test programs into the system. There wasn't even a file system! Test programs were actually built as small parts of the operating system code, and the entire system was rebuilt for each test. This was time-consuming but necessary to prove some of my theories and hunches. All of the early test programs were in assembler, of course. If you start from scratch on a different processor, you will go through similar contortions. If you H
H Introduction intend to use the Intel processors and start with MMURTL or the information presented in this book, you'll be ahead of the game. The Real Task at Hand (Pun Intended) All of the hardware details aside, my original goal was to create a multitasking operating system that was not a huge pig, but instead a lean, powerful, easy-to-use system with a tremendous amount of documentation. I had to meet my design goals, and the primary emphasis was on the kernel and all that that implies. If you write your own system, don't lose sight of the forest for the trees. I touched on what I wanted in Chapter 1, but putting it on paper and getting it to run are two very different things. A Starting Point If you have truly contemplated writing an operating system, you know that it can be a very daunting task. I wish there would have been some concrete information on really where to start, but there wasn't. The books I have purchased on operating system design gave me a vast amount of insight into the major theories and overall concepts concerning operating system design, but they just didn't say, “You start here, at the beginning of the yellow brick road.” In addition, it is difficult for me to advise you because I don't know exactly where you're com- ing from, how much knowledge you have, or how much time you really have to invest. But I'll try. The theory behind what you want is the key to a good design. Know your desired application. If you are writing an operating system for the learning experience (and many people do), the actual application may not be so important, and you may find that you really have something when you're done with it — or at least when you get it working; you'll never really be done with it. Your target application will help you determine what is important to you. For example, if you are interested in embedded applications for equipment control, you may want to concen- trate on the device and hardware interface in more detail. I was really interested in communi- cations. You'll probably be able to see that as you read further in this book. The easy way out would be to take a working operating system and modify it to suit you. I have provided such an operating system with this book. It is, however, by no means the only operating system source code available. Of course, I'm not going to recommend any others to you. If you choose to modify an existing operating system, study what you have and make sure you completely understand it before making modifications. I say this from experience. Many times I have taken and “gutted” a piece of software that I didn't fully understand, only to find out that it had incorporated many of the pieces I needed — I simply hadn't understood them at the time. If this has never happened to you, I envy you.
Chapter 2 ■ General Discussion and Background The Critical Path Along your journey, you'll find that there are major items to be accomplished, and there will also be background noise. If you've ever done any large-scale project management, you know that certain items in a project must work or that certain must be proved in order for you to continue along the projected path, also called the critical path. I'll give you a list of these items in the approximate order that I found were required while I wrote my system. 1 . Decide on your basic models. This includes the tasking model, memory model, and basic programming interface. You may need to choose a hardware platform before making your memory model concrete. Chapter 3, “The Tasking Model,” and Chapter 4, “Interprocess Communications,” will give you plenty of food-for-thought on the tasking model. 2. Select your hardware platform. It may be RISC, it may be CISC, but whatever you decide on, get to know it very well. Try to be sure that it will still be in production when you think you'll be done with your system. This may sound humorous, but I'm very serious. Things change rapidly in the computer industry. 3. Investigate your tools. Get to know your assemblers, compilers, and utilities. Even though you may think the programmers who wrote those tools know all there is to know about them, you will no doubt find bugs. I blamed myself several times for problems I thought were mine but were not. 4. Play with the hardware to be sure you understand how to control it. This includes understanding some of the more esoteric processor commands. Operating systems use processor instructions and hardware commands simply never used by applications and even device drivers. Document the problems, the discoveries — anything you think you'll need to remember. Chapter 6, “The Hardware Interface,” gives you some hints on this. With so many platforms out there, hints are about all I can give you, unless you go with the same hardware platform I did. 5. Go for the kernel. If you have to build a prototype running under another operating system, do it. Doing so will be worth the time spent. You can then “port” the proto- type. 6. Memory management is next. This can be tricky because you may be playing equally with hardware and software. The level of complexity, of course, will depend on the memory model you choose. In Chapter 5, I give you some options. I tell you the easiest option, and I also dive into some more challenging possibilities. 7. If you've gotten past the fun parts (kernel and memory), the rest is real work. You can take these pieces — everything from device drivers, garbage collection, and overall program management — as you see fit. H
H Part I 1 1 Introduction Working Attitude Set your goals, but don't set them too high. I had my wish list well in advance. Creating a realistic wish list may be one of the harder jobs. Don't wish for a Porsche if what you need is a good, dependable pickup truck. The Porsche is nice... but think of the maintenance (and can you haul firewood in it?). Make sure your tools work. Make certain you understand their capabilities and limitations. I spent a good six months playing with assemblers, compilers, and utilities to be sure I was “armed and dangerous.” After I was satisfied, I moved on from there. As I mentioned previously, I actually had to switch assemblers in midstream. Problems with programming tools were some- thing I couldn't see in advance. The tools you should have include documentation for the hardware and plenty of “theory food” for the engine (your brain). Work with small, easily digestible chunks. Even if you have to write four or five small test pro- grams to be sure that each added piece functions correctly, it's worth the time. Nothing is worse than finding out that the foundation of your pyramid was built on marshmallows. When writ- ing my memory- allocation routines, I must have written 30 small programs (many of which I did not intend to act correctly) to be sure I had what I thought I had. I'll admit that I had to do some major backtracking a time or two. Document as you go. I lay out my notes for what I want, I write the documentation, and then I write the code. Sounds backward? It is. I wrote major portions of my programming docu- mentation before the system was even usable, but doing so gave me great insight into what a programmer would see from the outside. Good luck. Really, it won't be luck. It will be blood, sweat, and maybe some tears, all the way.