71KB
Developing Your Own 32-Bit Operating System — Chapter-05: Memory Management
- Source: /library/Developing Your Own 32-Bit Operating System/source-file.pdf
- PDF pages: 77–102
- Pages without text: 77
Memory Management
E Part II ■ Architecture and General Theory This chapter provides you with some fairly detailed ideas for memory management from an operating system's “perspective.” An operating system's view of memory management is quite different than that of a single application. If you've ever written memory-allocation routines for compiler libraries, or other complicated applications, you know you are basically asking the operating system for chunks of memory that you will break down even further as required for your application. Just as your memory management routines keep track of what they hand out, so does an oper- ating system. There are some major differences though. These differences vary considerably based on how you choose to handle the memory on the system. Memory-management code in an operating system is affected by the following things (this list is not all inclusive, but touches on the important things): 1 . How the processor handles addressing. This greatly affects the next three items. 2. The memory model you choose. 3. Whether or not you use paging hardware, if available. 4. Whether you allow variable-sized chunks of memory to be allocated. I do make some assumptions about your processor. I can do this because most of the proces- sors afford roughly the same memory-management schemes. Some do it with internal paging hardware; some do it with an external page memory management unit (PMMU). Either way, it's all basically the same. Basic Terms Once again, I need to define some terms to ensure we are talking about the same thing. Physical memory is the memory chips and their addresses as accessed by the hardware. If I put address 00001 on the address bus of the processor, I am addressing the second byte of physical memory. Address 0 is the first. Linear memory is the address space a program or the operating system sees and works with. These addresses may or may not be the same as the physical addresses. This will depend on whether or not some form of hardware translation takes place. For instance, Intel uses this term to describe memory addresses that have been translated by paging hardware. Logical memory is more an Intel term than a generic term that can be applied across different manufacturers’ processors. This is the memory that programs deal with and is based around a selector (a 1 6-bit number that serves as an index into a table). With the Intel processors, a protected-mode program's memory is always referenced to a selector that is mapped in a table to linear memory by the operating system and subsequently translated by the processor. 1 will discuss segmentation in greater detail in this chapter for those of you who may want to use it.
Chapter 5 H Memory Management S Memory Model Do not confuse my term “memory model” for the MS-DOS/Intel term. It does not refer spe- cifically to size or to any form of segmentation. Instead, it refers to how and where programs are loaded, and how the operating system allocates and manages processor memory space. There are several basic memory models to choose from and countless variations. You may be restricted from using some of these options, depending on what your processor or external memory management hardware allows. I'll start with the easiest to understand and work from there. Keep in mind that the easiest to understand may not necessarily be the easiest to im- plement. Simple Flat Memory Simple flat memory means that the operating system and all programs share one single range of linear memory space, and that physical addresses and linear addresses are the same. No ad- dress translations take place in hardware. In this type of system, the operating system is usually loaded at the very bottom or very top of memory. As programs are loaded, they are given a section of memory (a range of address) to use as their own. Figure 5.1 shows this very simple shared-memory arrangement. FIGURE 5.1. Simple flat memory. Unallocated Operating System Controlled Physical/Linear Memory Program B Program A Operating System Code, Data & Stack Unless some form of protection is used, any program can reach any other program's memory space (intentionally or unintentionally). This can be good and bad. The good part is that no address translations are required for any kind of interprocess communications. The bad part is that an invalid pointer will more than likely cause some serious problems. Hardware memory paging would possibly provide a solution to the bad pointer problem. Pag- ing hardware usually affords some form of protection, at least for the operating system. If the
H Part II ■ Architecture and General Theory paging hardware provides only two levels of protection, this still means that program A can destroy program B as long as they both share the same linear space and are at the same protec- tion level. Considerations and variations of this simple model include paging, different memory alloca- tion schemes (top down, bottom up, and so on), and possible partitioning for each program. Partitioning means you would allocate additional memory for a program it would use as its unallocated heap. This would keep all of the program's memory in one range of linear addresses. As programs are loaded, they are partitioned off into their own space with a lower and upper bound. Paged Flat Memory Paged flat memory is a variation on simple flat memory. With this scheme, chunks of physical memory, called pages, are allocated and assigned to be used as a range of linear memory. This is still a flat scheme because all the programs will use the same linear addresses. They can still share and get to each other's address space if no protection is applied. This scheme requires the use of paging hardware. Figure 5.2 shows how the blocks of physical memory are turned into addressable chunks for the system. With paged memory, the physical memory can come from anywhere in the physi- cal range to fit into a section of linear address space. With this scheme, the linear address range can also be much larger than the amount of physical memory, but you can't actively allocate more linear address space than you have physical memory. For example, your linear address space may run from 0 to 1 gigabytes, but only 16 megabytes of this linear address range may be active if that's all the physical memory your system has. This gives the operating system some options for managing memory allocation and cleanup. Figure 5.2 also implies that the physical pages for the program were allocated from the “top down” to become its addressable area from the “bottom up” which is perfectly legitimate. FIGURE 5.2. Paged flat memory. Highest ^ Linear Address Unallocated Linear memory / Physical Pages Allocated to Program Unallocated Physical Memory Pages Program Operating System Pages Allocated to OS Linear Address 0^ -Highest Physical Address Physical
- Address 0
Chapter 5 U Memory Management H Demand-Paged Flat Memory Demand-paged flat memory is a major variation on paged flat memory. With demand-paged memory, you extend the active size of your linear memory beyond your physical memory's limits. For instance, if you have 16MB of physical memory, your system's actively- allocated linear address range may actually be extended to a much greater size (say 32, or even 64MB). The additional pages of physical memory will actually be contained on an external, direct-ac- cess storage device, more than likely a hard disk drive. The word demand means that they are sent out to the disk when not needed, and brought back into real physical address space on demand or when required. Figure 5.3 shows an example of the page translations that can occur. FIGURE 5.3. Demand-paged flat memory. Highest - Linear Unallocated Linear memory / EE Least recently SS used pages stored == on disk J%T mm — Unallocated — Physical — Pages Program C Program B ZZ Allocated ~ Physical ~ Memory ~ Pages Program A Operating System — Pages Allocated — to OS --Highest Physical Physical In this scheme, all the programs still use the same linear space. Address 30000h is the same to all programs and the operating system. Something else to note is that the actual values of these linear addresses may extend to the highest values allowed by the paging hardware. You can start your application addressing at the 3GB mark if you desire, and the hardware allows it. Virtual Paged Memory The term virtual usually means it's not real, but it sure looks like it is, as in virtual reality. The minute you apply this term to memory management, it means that you're lying to your pro- grams, and indeed you are. With paging hardware, you're already lying to them when you map in a physical page and it's translated to a linear page address they can use. So you can really call any form of memory management where address translations take place “virtual” memory. Some of the older systems referred to any form of demand paging as virtual memory. My definition of virtual is telling a big We (not a little one).
60 | Part II ■ Architecture and General Theory This means you can tell two programs they are both using linear address ranges that are the same! In other words, they are operating in “parallel” linear spaces. Look at Figure 5.4 for an example that will help you understand this technique. FIGURE 5.4. Virtual paged memory. Drogram A Program B Linear Address Linear Address Space Space Physical Memory Highest^” Linear Unallocated Pages
- Unallocated Linear Address — Linear — Address Free Space Physical — Space for Program A Pages — for Program
- B Pages \ forB \\\ ^NNNN\ Pages \x\v for A Lnea Operating System OS Pages _ Operating ~ System Address 0^ Like the paged flat memory, physically-allocated pages can be placed anywhere in a program's linear address space. Unlike paged flat memory, virtual paged memory is not flat. To a single program with its own address space, it may appear flat, which is the idea, but linear address 30000h to one program may not be the same physical memory addressed as linear address 30000h to another program (same address, different data or code). Notice I said may not, and not will not. This is because you can make the physical addresses match the linear addresses for two programs if you desire. The options here are almost mind boggling. You can have all pro- grams share a section of the same physical memory in a portion of their linear space, but make all the rest of their linear space completely independent and inaccessible by any other program. In Figure 5.4 you can see that both programs see the operating system pages at the same ad- dresses in their linear spaces, but the same linear range of addresses where program code and data are stored are completely independent. Note the advantages here. This means you can share operating system memory between all your programs, yet any memory they allocate can be theirs alone. Note also the disadvantages. The largest will be when you implement interprocess communi- cations. You will need some form of translation between addresses that are passed from one linear space to another. A way around this to use a shared memory area. Once again, there are many possible variations. No doubt you will think of, and implement, some of your own. In order to allow multiple, parallel linear spaces, the memory management hardware must support it. You will have to do some homework for your processor of choice.
Chapter 5 M Memory Management Demand-Paged Virtual Memory Just like demand-paged flat memory, demand-paged virtual memory is when you can allocate more pages than you physically have on your system. The same type of on-demand external storage is used as with demand-paged flat memory. The major difference is that these pages can be for any one of several independent linear address spaces. Demand-Paged Memory Management Management of demand-paged memory is no trivial task. The most common method is with a least recently used (LRU) algorithm. It's just what it sounds like. If you need to move someone's physical pages back into their linear space, the pages that are moved out to make room are the ones that haven't been used recently. The LRU schemes have many variations. The most common, and generally the most efficient, is to use a separate thread or task that keeps track of how often pages are used. The paged memory management hardware will tell you which ones have been used and which ones have been sit- ting idle since they were paged in or created. In fact, paging hardware will tell you whether a page has been accessed only for reading or has actually been modified. When a page has been written to, it is usually termed as dirty. The idea of a page being dirty has some significance. If you had read in a page and it was accessed for reading, and now you want to move it out to disk because it hasn't been used lately, you may already have a clean copy on the disk. This means you can save the time rewriting it. This can be a big time-saver when you're talking about thou- sands of pages being swapped in and out. This additional task will try to keep a certain percentage of physical space cleared for near term page-in or page creation (memory allocation) requirements. H Segmented Memory I will discuss segmented memory because the Intel processors allow this type of memory, and you need to understand the implications. You may even want to use it to make your system compatible with some of the most popular operating systems in use (OS/2 or Windows for example). This compatibility may be in the form of the ability to use code or object modules from other systems that use segmentation, and maybe to use the Intel Object Module format. Segmentation is really a holdover from the past. When this was a 1 6-bit world, or even an 8-bit one, we had some fairly limited address spaces. A multitude of software was written for 16-bit addressing schemes. When Intel built the first of its 32-bit processors (80386), Intel made sure that the 32-bit pro- cessor was downward-compatible with code written for the 1 6-bit processors. Likewise, when Intel built the 16-bit processors (8086, 8088, 80186, and 80286), the 16-bit processors were also compatible with object code for the 8-bit processors (8080 and 8085, and the popular Zilog Z80).
E Part II ■ Architecture and General Theory The 8-bit processors had 16 address lines. This meant you could have a whopping 64K of RAM. It also meant that when they built processors with more address lines, 20 to be exact, they needed a way to use all of the 8-bit processor code that used 16-bit addressing. What Intel came up with is called segmentation. They allowed multiple sequential, or overlap- ping 64-K sections of memory called segments. To make this work, they needed to identify and manage those segments. They used something called a segment register. This was actually several registers, one for code and several for data access, that would identify which particular 64K- addressable area you were working with. What they needed was a transparent way to use the extra four bits to give you access to a full 1MB of RAM. The segment registers gave you an addressable granularity of 16 bytes. All of the Intel products are very well documented, so I won't go into further detail. However, this introduction leads me to another very important point about segmentation that you need to understand. The segment registers in the 1 6-bit processors were a simple extension to the address lines. When the 80286 came on the scene, they added a very important concept called Protected mode. This made it possible to manage these segments with a selector. A selector is a number that is an index into a table that selects which segment you are working with. It is really a form of virtual memory itself. Protected mode operation with selectors carried over into the 32-bit processors. Another major change in the 32-bit processors that deal with segments is the fact that they can now be as large as all available memory. They are not limited to 64K. The selectors are managed in tables called the Global Descriptor Table (GUY) or Local Descrip- tor Tables (LDT). When you use a selector, the processor uses the table entry indicated by the selector and performs an address translation. The descriptor tables allow you to set up a zero- based address space that really isn't at linear address zero. This means you can locate code or data anywhere you want in the 4GB of linear memory, and still reference it as if it were at offset zero. This is a very handy feature, but it can become cumbersome to manage because it can force you to use something called far pointers to access almost everything. The far pointer is a pointer that includes the selector as part of the address. With a 32-bit system, this makes a far pointer 48 bits. This can be a real hassle. It also slows things down because the processor must do some housekeeping (loading shadow registers) each time the selector is changed. Memory Management After you've chosen a memory model, you'll face many decisions on how to manage it. Man- agement of memory involves the following things: ■ Initialization of memory management ■ Allocating memory to load applications and services ■ Allocating memory for applications
Chapter 5 M Memory Management ■ Tracking client linear memory allocations ■ Tracking physical memory, if paging is used ■ Handling protection violations, if your hardware supports protection I'll discuss each of these things, but not in the order I listed them. This is because you have to figure out how you'll manage memory before you know how to initialize or track it. H Tracking Linear Memory Allocation Two basic methods exist to track allocation of linear or physical memory. They are linked lists and tables. Before you decide which method to use, you should decide on your basic allocation unit. Basic Memory Allocation Unit The basic allocation unit defines the smallest amount of memory you will allow a client to allocate. Remember, you are the operating system. You make the rules. You can let applica- tions allocate as little as one byte, or you can force them to take 1 6 kilobytes at a time. You'll have to make a decision, and it will be based on many factors. One of the important ones will be whether or not you use hardware paging. If you do, you may want to make the allocation unit the same size as a page, which will be defined by the paging hardware. This will simplify many things that your memory management routines must track. Other factors include how much memory you have, the type of applications you cater to, and how much time and effort you can put into to the algorithms. I don't know of any theoretical books that take into account how much time it takes to actu- ally implement pieces of an operating system, but it's a very realistic part of the equation, espe- cially if you have a very small programming team — one person, for instance. I learned this the hard way, and my advice to you is not to bite off more than you can chew. Linked List Management Using linked lists is probably the most common and one of the easiest ways to manage memory. This type of algorithm is most common when a large linear address space is shared by one or more applications, such as I described earlier with the simple flat memory model. With this scheme, you have one or more linked lists that define allocated and/or free memory. Hundreds of algorithms have been designed for linked list memory management. I will give you a good idea what it's like, and then it's up to you to do some serious research to find the best method for your system. The linked lists are set up when the operating system initializes memory. How you store the links in this list can vary. The easiest and most efficient way is to store the link at the beginning of the allocated block. This way, each block holds its own link and can be located almost in- stantly with the memory reference. The links are small structures that contain pointers to other
r / Part II ■ Architecture and General Theory links, along with the allocated size and the memory block owner. I recommend you track which client owns each link for deallocation and cleanup purposes. In operating systems that don't provide any type of protection, when one application crashes, it usually takes the whole system down. In many systems, however, when one application crashes, it is simply terminated, and the resources it used are recovered. The word cleanup also implies that the memory space becomes fragmented as callers allocate and deallocate memory. This is another topic entirely, but I'll touch on it some. MS-DOS appears to use links in the allocated space. I haven't disassembled their code; I can tell this from the pointer values I received when I allocate memory from the MS-DOS operat- ing system heap. Ignoring the fact that MS-DOS uses segmented pointers, you can see when you do the arithmetic that there are a few conspicuous bytes missing from two consecutive memory allocations. You now have an idea where they went. In the simple linked list scheme in Listing 5.1, you have two structure variables defined in your memory management data. These are the first links in your list. One points to the first linked block of free memory (pFree), and the other points to the first linked block of allocated memory (pMemlnUse). Listing 5.1. Linked list memory management. /* Memory Management Variables */ struct MemLinkType /* 16 bytes per link */ { char *pNext; char *pPrev; /* a backlink so we can find previous link */ long size; long owner; }; struct MemLinkType pFree; struct MemLinkType pMemlnUse; struct MemLinkType *pLink; /* to work with links in memory */ struct MemLinkType *pNewLink; struct MemLinkType *pNextLink; struct MemLinkType *pPrevLink; long Client = 0; In the process of memory management initialization, you would set up the two base link struc- tures, pFree and pMemlnUse. For my description, let's assume you have 1 5MB of memory to manage beginning at the 1 MB address mark (lOOOOOh), and all of it is free (unallocated). You would set up the base struc- tures as follows: void InitMemMgmt(void) { pFree.pNext = 0x100000; /* first link in 15 megabyte array */ /* " " */ /* " ■ */ /* " " */
pFree.pPrev = 0; / pFree.size = 0; / pFree. owner = 0; / pMemlnUse.pNext = 0 / pMemlnUse.pPrev = 0 / pMemlnUse.size = 0 / pMemlnUse. owner = 0 / Chapter 5 M Memory Management No backlink */ No size for base link */ OS owns this */ No mem blocks allocated yet */ No backlink */ No size for base link / OS owns this / / Set up first link in free memeory / pLink = 0x100000; / Set structure pointer to new link / pLink->pNext = 0; / NULL. No next link yet. / pLink->pPrev = &pFree; / Backlink to pFree / pLink->size = 0xf00000 - 16; / 15 Megs-16 for the link / pLink->owner =0; / OS owns it now / } When a client asks for memory, you follow the links beginning at pFree to find the first link large enough to satisfy the request. This is done by following pLink . pNext until pLink . size is large enough, or you reach a link where pNext is null, which means you can't honor the re- quest. Of course, you must remember that pLink . size must be at least as large as the requested size plus 16 bytes for the new link you'll build there. The following case of the very first client shows how all allocations will be. Let's assume the first request is for 1MB. WantSize is the size the client has requested, and Client is the owner of the new block. The function returns the address of the new memory, or a NULL pointer if you can't honor the request. The example code in Listing 5.2 is expanded to show each operation. You could make this type of operation with pointers almost “unreadable” in the C programming language. If you use this code, I suggest that you leave it as is for maintenance purposes and take the hit in memory usage. I've gone back to some code I did five or six years ago and spent 30 minutes just trying to figure out what I did. Granted, changes in pointer usage were in an effort to save memory and make the code tight and fast, but it didn't seem like it was worth the headaches. H Listing 5.2. Linked list memory allocation routine. char AllocMem(long WantSize) { pLink = &pFree; / Start at the base link / / Keep going till we find one / while (pLink->pNext) / As long as we have a valid link.../ { pLink = pLink->pNext; / Next link please / if (pLink->size >= (WantSize + 16)) / This one will do! / { / Build a new link for the rest of the free block / / then add it to the free list. This divides the free / / block into two pieces (one of which we'll allocate) */
Architecture and General Theory Listing 5.2. continued /* Set up new link pointer, fix size and owner */ pNewLink = &pLink + (WantSize + 16); pNewLink->size = pLink->size - WantSize - 16; pNewLink->owner = 0; /* OS owns this for now */ /* Hook it into free links after pLink */ pNewLink->pPrev = &pLink; /* Backlink */ pNewLink->pNext = pLink->pNext; pLink->pNext = &pNewLink; pLink->size = WantSize; /* Fix size of pLink */ /* Remove pLink from the pFree list and put it in the */ /* allocated links! */ pPrevLink = pLink->pPrev; /* get previous link */ pPrevLink->pNext = &pNewl_ink; /* Unhook pLink */ pLink->pNext = pMemlnUse.pNext ; /* Put pLink in USE */ pMemlnUse.pNext = &pLink; /* Point to the new link */ pLink->size = WantSize; /* How much we allocated */ pLink->owner = Client; /* This is who owns it now */ return(&pLink+16) ; /* Return address of the NEW memory */ } } return(0); /* Sorry - no mem available */ } You will need a routine to free up the memory and also to clean up the space as it gets frag- mented. Believe me, it will get fragmented. The deallocation routine is even easier than the allocation routine. When they hand you a pointer to deallocate, you go to that address (minus 16 bytes) validate the link, change its owner, then move it to the pFree list. This is where the fragmentation comes it. The cleanup of fragmented space in this type of system is not easy. Even though the links them- selves are next to each other in memory, they can end up anywhere in the pFree list, depending on the order of deallocation. There are several thoughts on cleanup. You can add code in the allocation and deallocation routines to make it easier, or you can do it all after the fact. I personally have not done any research on the fastest methods, but much depends on the patterns of allocation by your cli- ents. If they allocate a series of blocks and then free them in the same order, the links may end up next to each other in the free list anyway. A simple scan of the list can be used to combine links that are next to each other into a single link. But you can't depend on this. Another method is to keep the pFree list sorted as you add to it. In other words, do a simple insertion at the proper point in the list. This will add time to deallocation routines, but cleanup
Chapter 5 ■ Memory Management will be a snap. You have the linear address, so you simply walk the list until you find the links that the newly freed memory belongs between, and you insert it there. Combining blocks can be done at the same time. This makes the most sense to me. Here is a deallocation routine that defragments the space as programs free up memory. This starts at the first link pFree is pointing to, “walks” up the links, and finds its place in linear memory order. You try to combine the newly-freed link with the one before it or after it, if at all possible. This provides instant cleanup. You should notice that the allocation algorithm ensured that there was always a link left at a higher address than the link you allocated. This will be useful in the routine in Listing 5.3, which returns memory to the free pool, because you can assume there will always will be a link at a higher address. Listing 5.3. A routine to return memory to the free pool. int FreeMem(char MemAddress) { / We will use pNewLink to point to the link to free up. / / It's actually a MemlnUse link right now. / pNewLink = MemAddress - 16; / point to the link to free up / / Some form of error checking should go here to ensure they / / are giving you a valid memory address. This would be the link / / validation I talked about. You can do things like checking to / / ensure the pointer isn't null, and that they really own the / / memory by checking the owner in the link block. / pNextLink = pFree.pNext; / Start at pFree / / scan till we find out where we go. We will stop when / / we find the link we belong between. / while ((pNextLink->pNext) && (&pNextLink < &pNewLink)) { pNextLink = pNextLink->pNext; } pPrevLink = pNextl_ink->pPrev; / this will be handy later / / If memory isn't corrupted, we should be there! / / We could just stick the link in order right here, / / but that doesn't help us cleanup if we can / / First let's see if we can combine this newly freed link / / with the one before it. This means we would simply add our / / size + 16 to the previous link's size and this link would / / disappear. But we can't add our size to pFree. It's a dummy / if (pNextLink ->pPrev != &pFree) / Can't be pFree */ { if (SpNewLink == (&pPrevl_ink + (pPrevLink->size + 16))) H
I
68
Part II ■ Architecture and General Theory
Listing 5.3. continued
{
/* add our size and link size (16), then go away! */
pPrevLink->size += pNewLink->size + 16;
return(0) ;
}
}
/* If we got here, we couldn't be combined with the previous. */
/* In this case, we must insert ourselves in the free list */
pPrevLink->pNext = pNewLink;
pNextLink->pPrev = pNewLink;
pNewLink->pNext = pNextLink;
pNewLink ->pPrev = pPrevLink;
pNewLink ->owner = 0; /* operating system owns this now!*/
/* Now we'll try to combine pNext with us! */
if ((&pNewLink + pNewLink ->size + 16) == &pNextLink)
{
/* We can combine them. pNext will go away! */
pNewLink ->size += pNextLink->size + 16;
pNewLink ->pNext = pNextLink->pNext;
pLink = pNextLink->pNext;
if (pLink) /* the next could have been the last! */
pLink->pPrev = &pNewLink; /* fix backlink */
}
/* If we didn't combine the new link with the last one, */
/* we just leave it were we inserted it and report no error */
return(0) ;
}
Memory Management with Tables
If you use paging hardware — or you decide to make your operating system's allocation unit a
fixed size and fairly large — you can use tables for memory management. In the case of the pag-
ing hardware, you will be forced to use tables of some kind anyway.
Paging hardware will define how the tables are organized. If you don't use the paging hard-
ware, you can decide how you want them set up. Tables are not as elegant as linked lists. Quite
often you resort to brute processor speed to get things done quickly. I go into great detail on
paging hardware and tables when I describe MMURTL memory management. It combines
the use of managing physical and linear memory with tables. It should give you a great number
of ideas, and all the information you need to use paging hardware on the Intel and work- alike
systems. If you want to use tables, and you're not using paging hardware, you can still look at
the setup of the tables for paging. You can expand on them if needed.
Chapter 5 B Memory Management Tracking Physical Memory If you use some form of paging, you will have two memory spaces to track. The two must be synchronized. In the memory-management routines earlier, you assumed you were working with a flat linear space. You can still use paging under the system described earlier; you will simply have to ensure that when you hand out a linear address, there are physical pages as- signed to it. This means your operating system will be tracking linear and physical memory at the same time. When you work with paging hardware, you will deal with a fixed size allocation unit. It will be the page's size. It will be greater if you keep a certain number of pages grouped together in a cluster. This will almost certainly lead to some form of table use. Paging hardware uses tables to track the relationship between physical and linear memory. Your operating system is responsible for ensuring they are updated properly. You could also use linked lists to manage the physical memory underneath the linear address space if you like. Something to remember about working with physical memory when paging is used, is that you can't address it until it has been identified in the hardware page tables that you manage. H Initialization of Memory Management I won't discuss loading the operating system here because it's covered in Chapter 7, “OS Ini- tialization,” but once the operating system is loaded, you need to ensure that you initialize memory so that it takes into account all memory used by the OS code and data. This is one of the first things you will do during initialization of memory management. You must consider where the operating system loads. You may want your operating system code and data at the very top of your linear memory address space. Wherever you load it, it becomes allocated memory right away. From there, you may also have to take into account any hardware addresses that may not be allocated to applications or the operating system. Some processors use memory addresses from your linear space for device I/O. Intel has a different scheme called port I/O. I'll discuss that in Chapter 6, “The Hardware Interface.” If you use a processor that uses hardware I/O from the memory space, it will generally be in a block of addresses. You will allocate these up front. If you use paging, you have a lot of work to do during the initialization process. You must set up tables, add the translations you need and are already using, and turn on the paging hard- ware. The hardware paging is usually not active when the processor is reset. Most paging hard- ware also requires that the non-paged addresses you are executing during initialization match the paged linear addresses when you turn on paging. This means that if you want your operat- ing system code and data in a linear address beyond the physical range, you will have to move
H Architecture and General Theory it all, or load the rest of it after you turn on paging. This can get complicated. I recommend you leave the initial operating system memory block in an address range that matches physical memory. But it's up to you. Another chore you'll have is to find out just how much memory you have on the system. Some hardware platforms have initialization code stored in ROM (executed at processor reset) that will find the total and place it somewhere you can read, such as battery backed-up CMOS memory space. You may not always be able to depend on this to be accurate. Batteries fail, and you should take that into consideration. Memory Protection Protected memory implies that the processor is capable of signaling you when a problem is encountered with memory manipulation or invalid memory address usage. This signaling will be in the form of a hardware trap or interrupt. The trap may not actually be signaling a problem, but may be part of the system to help you manage memory. For instance, demand-paged systems must have a way to tell you that some application is trying to address something in a page that isn't in memory right now. Other forms of traps may be used to indicate that an application has tried to access memory space that doesn't belong to it. It generally means the application has done some bad pointer math and probably would be a candidate for shut-down. If your processor gives you these types of options, it will generally be associated with page tables that you are using to manage your linear address space. In some cases, you will have independent tables for each application. You will have to investigate each type of interrupt, trap, fault or other signaling method em- ployed by the processor or paging hardware and determine how you will handle all those that apply to the type of system you design. An Intel-Based Memory-Management Implementation I chose the Intel processors to use for MMURTL, so the best thing I can do is provide you with a very detailed description of the memory model and my implementation. As described previously, I use a virtual paged memory model. I may add demand paging to it another time, but it serves my purposes quite well without demand paging. I depend on the paging hardware very heavily. It can be very complicated to use on any system. but how I use it may give you some ideas even I haven't thought of.
Chapter 5 B Memory Management A Few More Words on Segmentation Even though I continue to refer to Intel as the segmented processors, the concept of program segments is an old one. For instance, programs are quite often broken down into the code seg- ment, the initialized data segment, uninitialized data segment, and many others. I do use a very small amount of segmentation, and I even use the Intel segment registers, but in a very limited fashion. If you have no desire to use them, set them all to one value and forget them. If you are familiar with segmented programming, you know that with MS-DOS, programs generally have one data segment (which is usually shared with the stack) and one or more code segments. This is commonly referred to as a Medium Memory Model program. In the 80x86 Intel world, there are Tiny, Small, Compact, Medium, Large, and Huge models to accommo- date the variety of segmented programming needs. The variety of models is too complicated for me and is no longer necessary on these powerful processors. I want only one program memory model. The program memory model I chose is most analogous to the small memory model that has two segments. One is for code and the other is for data and stack. This may sound like a restriction until you consider that a single segment can be as large as all physical memory and even larger with demand page memory. I use almost no segmentation. The operating system and all applications use only three defined segments: the operating system code segment, the application code segment, and one data seg- ment for all programs and applications. The fact that the operating system has its own code segment selector is really to make things easier for the operating system programmer and to permit protection within the operating system pages of memory. Making the operating system code zero-based from its own selector is not a necessity, but nice to have. This could change in future versions, but the change will have no affect on applications. The “selectors” (segment numbers for those coming from real mode programming) are fixed. The selector values must be a multiple of eight, and I chose them to reside at the low end of the Global Descriptor Table. These will never change in MMURTL as long as they are legal on Intel and work-alike processors. The operating system code segment is 08h. The user code segment is 18h. The common data segment is lOh. MMURTL's memory management scheme allows you to use 32-bit data pointers exclusively. This greatly simplifies every program you write. It also speeds up the code by maintaining the same selectors throughout most of the program's execution. The only selector that will change is the code selector as it goes through a call gate into the operating system and back again. This means the only 48-bit pointers you will ever use in MMURTL are for an operating system call address (16-bit selector, 32-bit offset). m
H Part II ■ Architecture and General Theory How MMURTL Uses Paging MMURTL uses the Intel hardware-based paging for memory allocation and management. The concept of hardware paging is not as complicated as it first seems. Getting it straight took only half my natural life. MMURTL really doesn't provide memory management in the sense that compilers and lan- guage systems provide a heap or an area that is managed and cleaned up for the caller. I figured that an operating system should be a “wholesale dealer” in memory. If you want just a few bytes, go to your language libraries for this trivial amount of memory. My thought was for simplicity and efficiency. I hand out (allocate) only whole pages of memory as they are requested, and I return them to the pool of free pages when they are deallocated. I manage all memory in the processor's address space as pages. Apageis four kilobytes (4K) of contiguous memory. It is always on a 4K boundary of physical as well as linear addressing. Paging allows you to manage physical and linear memory address with simple table entries. These table entries are used by the hardware to translate, or map, physical memory to what is called linear memory. Linear memory is what applications see as their own address space. For instance, you can take the very highest 4K page in physical memory and map it into the application's linear space as the second page of its memory. This 4K page of memory becomes addresses 4096 through 8191, even though it's really sitting up at a physical 16MB address, if you had 16MB of RAM. No, it's not magic, but it's close. Page Tables (PTs) The tables that hold these translations are called page tables (PTs). Each entry in a PT is called a page table entry (PTE). There are 1 024 PTEs in every PT. Each PTE is four bytes long. (Aren't acronyms fun? Right up there with CTS — Carpal Tunnel Syndrome). With 1024 entries (PTEs) each representing four kilobytes, one 4K page table can manage 4MB of linear/physical memory. That's not too much overhead for what you get out of it. Here's the tricky part. The operating system itself is technically not a. job. Sure, it has code and data and a task or two; but most of the operating system code — specifically, the kernel — runs in the task of the job that called it. The kernel itself is never scheduled for execution. Because of this, the operating system really doesn't own any of its memory. The operating system is sharedhy all the other jobs running on the system. The page tables that show where the oper- ating system code and data are located get mapped into every job's memory space. Page Directories (PDs) The paging hardware needs a way to find the page tables. This is done with something called a page directory (PD). Every job gets its own PD. You could design your system with only one page directory if you desire.
Chapter 5 U Memory Management m Each entry in a PD is called a page directory entry (PDE). Each PDE holds the physical address of a page table. Each PDE is also four bytes long. This means you can have 1024 PDEs in the PD. Each of the PDEs points to a PT, which can have 1024 entries, each representing 4K of physical memory. If you get your calculator out, you'll see that this allows you to map the entire 4GB linear address space. 1024xl024x4K (4096) = 4,294,967,296 (4GB) You won't need this capability any time soon, right? Wrong. What you don't see, because I haven't explained it yet, is that you really do need most of this capability, but you need it in pieces. The Memory Map The operating system code and data are mapped into the bottom of every job's address space. A job's memory space actually begins at the 1GB linear memory mark. Why so high? This gives the operating system one gigabyte of linear memory space, and the same to each applica- tion. Besides, if I add demand paging to MMURTL's virtual memory, an application of a hundred megabytes or more is conceivable. The map of a single job and the operating system is shown in Table 5.1. The map is identical for every job and service that is installed. Table 5.1. MMURTL memory map. Description Address Range Linear Top Dead address space Linear Max Job Allocated Memory Data Code Initial Stack Job Memory DLLs (loadable shared code) Device Drivers OS Allocated memory OS Memory Linear Base 4GB - 1 byte 2GB - LinearTop 2GB (artificial maximum limit) 1GB + job memory 1 GB + stack page(s) + code pages 1 GB + stack page(s) 1GB 1 GB (initial stack, code, data) 0GB + operating system memory 0GB 0GB
H Part II ■ Architecture and General Theory Now the pundits are screaming: “What about the upper two gigabytes — it's wasted!” Well, in a word, yes. But it was for a good cause. (No, I didn't give it to the Red Cross.) In the scheme of things, the operating system has to know where to find all these tables that are allocated for memory management. It needs a fast way to get to them for memory-manage- ment functions. Sure, I could have built a separate table and managed it, but it wasn't needed. Besides, I wanted to keep the overhead down. Read on and see what I did. The processor translates these linear (false) addresses into real (physical) addresses by first find- ing the current page directory. It does this by looking at the value you (the OS) put into Con- trol Register CR3. CR3 is the physical address of the current PD. Now that it knows where the PD is, it uses the upper 10 bits of the linear address being translated as an index into the PD. The entry it finds is the physical address of the page table (PT). The processor then uses the next lower 10 bits in the linear address as an index into the PT. Now it's got the PTE. The PTE wthe physical address of the page it's after. Sounds like a lot of work, but it does this with very little overhead, certainly less overhead than this explanation. The operating system has no special privileges as far as addressing physical memory goes. The operating system uses linear addresses (false ones) just like the applications do. This is fine until you have to update or change a PDE or PTE. You can't just get the value out of CR3 and use it to find the PT because it's the physical address {crash — page fault). Likewise, you can't just take a physical address out of a PDE and find the PT it points to. Finding the PD for an application is no problem. When I started the application, I built the PD and stored the physical address in the Task State Segment field for CR3, then I put the linear address of the PD in the Job Control Block. This is fine for one address per job. How- ever, now you're talking dozens or even hundreds of linear addresses for all the page tables that you can have, possibly with several for each application. The Task State Segment (TSS) is a processor-defined structure used to manage a task. It may be expanded by the operating system to define other items needed for task management. The Job Control Block (JCB) is defined by the operating system and used to manage programs that are running on the system. This is how I use the upper 2K of the page directories. I keep the linear address of all the PTs there. Two kilobytes doesn't sound like a lot to save, but when you start talking 1 0, 20, or even 30 jobs running, it starts to add up. I make this upper 2K a shadow o$ the lower 2K. If you remember, each PDE has the physical address of each PT. MMURTL needs to know the physical address of a PT for aliasing ad- dresses, and it needs it fast.
Chapter 5 ■ Memory Management Exactly 2048 bytes above each real entry in the PD is MMURTL's secret entry with the linear address of the PT. Well, the secret is out. Of course, these entries are marked “not used” so the operating system doesn't take a bad pointer and try to do something with it. Page Directory Entries (PDEs) I know you're trying to picture this in your mind. What does this make the page directory look like? Below, in Table 5.2 is a list of the entries in the PD for the memory map shown in Table 5.1. The table assumes the operating system consumes only a few PTEs (one-page table maxi- mum). Table 5.2. Page directory example. Entry Number Description 0 Physical address of operating system PT (PDE 0) 1 Empty PDE 256 Physical address of job PT 257 Empty PDE 512 Linear address of operating system PT (Shadow — marked not present) 513 Empty Shadow PDE 768 Linear address of job PT (Shadow — marked not present) 769 Empty Shadow PDE 1023 Last Empty Shadow PDE This table doesn't show that each entry only has 20 bits for each address and the rest of the bits are for management purposes, but you get the idea. It's 20 bits because the last 12 bits of the 32-bit address are below the granularity of a page (4096 bytes). The low-order 12 bits for a linear address are the same as the last 12 bits for a physical address. As shown, all the shadow entries are marked not present, in fact, all of the entries with nothing in them are marked not present. They simply don't exist as far as the processor is concerned. If I desired, I could move the shadow information into separate tables and expand the operating system to address and H
H Architecture and General Theory handle 4GB of memory, but I'm more interested in conservation at this point in time. If I decided to do it, it would be transparent to applications anyway. Something else the example doesn't show is that the entry for the physical address of the oper- ating system PT (0) is actually an alias (copy) of the page tables set up when memory manage- ment was initialized. I don't keep duplicate operating system page tables for each job. That would really be a waste. Allocation of Linear Memory You now know the mechanics of paging on the Intel processors, and how I use the processor's paging hardware. Now you need to know how MMURTL actually allocates the linear space in each job or for the OS. This is accomplished with three different calls, depending what type of memory you want. AllocPage ( ) , AllocOSPage ( ) , and AllocDMAPage ( ) are the only calls to allo- cate memory in MMURTL. AllocPage () allocates contiguous linear pages in the job's address range. This is 1GB-2GB. The pages are all initially marked with the user protection level Read/Write. AllocOSPage ( ) allocates contiguous linear pages in the operating system address range. This is 0-1 GB. The pages are all initially marked with the system protection level Read/Write, and the entries automatically show up in all jobs’ memory space because all the operating system page tables are listed in every job's page directory. AllocDMAPage ( ) allocates contiguous linear pages in the operating system address range, but it ensures that these pages are below the 16MB physical address boundary. Direct Memory Ac- cess hardware on ISA machines can't access physical memory above 16MB. AllocDMAPage ( ) also returns the physical address needed by the users of DMA. The pages are all initially marked with the system protection level Read/Write. All AllocPage ( ) calls first check to see if there are enough physical pages to satisfy the request. If the physical memory exists, then they must find that number of pages as contiguous, free entries in one of the PTs. If enough free PTEs don't exist in a row, it will create a new PT. All AllocPage ( ) calls return an address to contiguous linear memory, or will return an error if it's not available. With a 1GB address space, it's unlikely that the allocation routine won't find a contiguous section of PTEs. You are likely to run out of physical memory (the story of my life). Deallocation of Linear Memory When pages are deallocated (returned to the operating system), the caller passes in a linear address, from a previous AllocPage ( ) call, along with the number of pages to deallocate. The caller is responsible for ensuring that the number of pages in the DeAllocMem( ) call does not exceed what was allocated. If it does, the operating system will attempt to deallocate as many pages as requested which may run into memory that was allocated in another request, but only from this caller's memory space. If so, there will be no error, but the memory will not be
Chapter 5 ■ Memory Management available for later use. If fewer pages than were allocated are passed in, only that number will be deallocated. The caller will never know, nor should it try to find out, where the physical memory is located, with the exception of DMA users (device drivers). I've discussed how MMURTL handles linear addresses. Now comes that easy part — managing physical memory. Allocation of Physical Memory The fact that the processor handles translation of linear to physical memory takes a great deal of work away from the OS. It is not important, nor do you even care, if pages of memory in a particular job are physically next to each other (with the exception of DMA). The main goal of physical memory management is simply to ensure you keep track of how much physical memory there is, and whether or not it's currently in use. Physical memory allocation is tracked by pages with a single array. The array is called the Page Allocation Map (PAM). The PAM is similar to a bit allocation map for a disk. Each byte of the array represents eight 4K pages (one bit per page). This means the PAM would be 5 1 2 bytes long for 1 6MB of physi- cal memory. The current version of MMURTL is designed to handle 64MB of physical memory which makes the PAM 2048 bytes long. Now if I could only afford 64MB of RAM. The PAM is an array of bytes from 0 to 2047, with the least significant bit of byte 0 representing the first physical 4K page in memory (physical addresses 0 to 4095). For AllocPage() and AllocOSPage(), you allocate physical memory from the top down. For AllocDMAPage( ) you allocate physical memory from the bottom up. This ensures that even if you install a device driver that uses DMA after your applications are up and running, there will be physical memory below 16MB available (if any is left at all). The PAM only shows you which pages of memory are in use. It does not tell you to whom they belong. To get this information, you must go to the PDs and PTs. Loading Things Into Memory Applications, system services, device drivers, and DLLs must all be loaded into memory some- where. Each application (job) gets its own PD. This is allocated along with the new job control block (JCB). It also gets as many pages as it needs for its code, initial stack, and data. It's loaded into these initial pages. Message-based system services are exactly like applications from a memory standpoint. They are simply new jobs. Device drivers have code and data but no stack. They become pan of the operating system and are reached through the standard entry points in the call gates. They are actually loaded into the operating system's memory space with freshly allocated operating system pages. They become part of the operating system in the operating-system address space accessible to everyone. m
H Part II ■ Architecture and General Theory Dynamic link libraries are the weird ones. They have only code, no data and no stack. Some systems allow DLLs to contain data. This can introduce re-entrancy problems. I wanted them to be fully re-entrant with no excuses. They are there to be used as shared code only. DLLs are also loaded into operating system address space, but the pages they occupy are marked executable by user level code. This means they can be accessed from the user's code with near calls. This also means that the loader must keep track of the PUBLIC names of each call in a DLL and resolve references to them after you load the applications that call them, but this gets away from memory management. Operating system page tables are aliased as the first tables in each job's PD and are marked as supervisor. PDs and PTs are always resident (no page swapper yet, how could they be swapped?). This is 8K of initial memory management overhead for each job. It will still only be 8K for a 4MB application. Memory overhead per application is one thing you will have to consider in your design. Messaging and Address Aliases If you design your memory management so that each application has its own independent range of linear addresses, you'll have to tackle address aliasing. This means you will have to translate addresses so one program can reach another's data. With a page-based system such as MMURTL, an alias address is actually just a shared PTE. If two programs need to share memory (as they do when using the interprocess communications, such as request/ respond messaging), the kernel copies a PTE from one job's PT to another job's PT. Instantly, the second job has access to the other job's data. They share physical memory. Of course they probably won't be at the same linear address, which means you have to fix up a pointer or two in the request block, but that's trivial. The fixup involves just a few instruc- tions. There is no new allocation of physical memory, and the service doesn't even know that the pages don't actually belong to it, as they “magically” appear inside its linear address space. Of course, if it tries to deallocate them, an error will occur. Paging makes messaging faster and easier. An aliased PTE is marked as such and can't be deallocated or swapped until the alias is dissolved. Aliasing will only occur for certain operating system structures and messaging other than the aliased page tables in the user's page directory for the operating system code and data. If you decide you want to use the full segmentation capabilities of the Intel processors, then you can also alias addresses using selectors. The Intel documentation has a very good descrip- tion of this process. Memory and Pointer Management for Messaging If you haven't read Chapter 4, “Interprocess Communications,” you should do it before you read this section. If you read that chapter first, this section will make more sense.
Chapter 5 • Memory Management When an application requests a service, the kernel allocates a request block from the operating system and returns a handle identifying this request block to the caller. This request block is allocated in operating system memory space, but at the user's protection level so the service can access it. The user's two pointers, pDatal and pData2, are aliased into the services memory area and are placed in the request block. The memory management aspects of the request/respond messaging process work like this: 1 . The caller makes a request. 2. The Request ( ) primitive (on the caller's side) does the following: ■ Allocates a request block. ■ Returns a request handle to the caller. ■ Places the following into the RqBlk: linear address of pDat a 1 (if not 0) linear address of pData2 (if not 0) sizes of the pDatal and pData2 pointer to caller's Job Control Block service code response exchange dDataO and dDatal ■ Places a message on the service's exchange with the Request handle in it. ■ Schedules the service for execution. ■ Reevaluates the ready queue, switching tasks if needed. 3. The Wait ( ) primitive (on the service's side) does the following: ■ Adds aliases to the service's Page Table(s) for pDatal and pDatal. ■ Places aliased linear addresses into RqBlk. ■ Returns the message to the service. 4. The service does its thing using the aliased pointers, reading and writing data to the caller's memory areas. When it's done its work, it responds. 5. The Respond ( ) primitive (on the service's side) does the following: ■ Removes the aliased memory from the service's PTs. ■ Places the message on the caller's response exchange. ■ Reevaluates the ready queue (switch tasks if needed). 6. The Wait ( ) primitive (back on the caller's side) passes the response message to the caller. H
I
80
Part II ■ Architecture and General Theory
Summary of MMURTL Memory Management
The key points to remember and think about when pondering my memory-management scheme
(and how you can learn from it, use it, or use pieces of it) are these:
■ One page directory (PD) for each job.
■ Linear address of the PD is kept in the job control block.
I One or more page tables (PT) for each job.
■ One or more PTs for the OS.
■ OS PTs are mapped into every job's page directory by making entries in the job's PD.
■ OS uses upper 2K of each PD for linear addresses of PTs.
■ Physical memory is tracked with a bit map.
What does all this mean to applications programmers? Not much I'm afraid. They don't need
to know any of this at all to write a program for MMURTL. Only those of you who will brave
the waters and write your own operating system will have to consider documentation directed
at the programmers. How they see the interface is very important. For MMURTL, the appli-
cation programmer needs to know the following:
■ The minimum amount of allocated memory is 4K (one page).
■ Memory is allocated in 4K increments (pages).
■ Jobs can allocate one or more pages at a time.
■ Jobs can deallocate one or more pages at a time.
■ The programmer tracks his own memory usage.