d a v i d j w a l l i n g . c o m

Custom Operating System Series

Part Four : Entering Protected Mode

This article is part four in a series introducing system software concepts for Intel x86 or compatible processors in protected mode. In this series we are developing a simple protected mode operating system in assembly language. We are using the Netwide Assembler (NASM) to assemble our code and VMware Player as our test platform.

In our first three articles, we configured VMware Player to launch a virtual machine that loaded a boot sector from a floppy disk image file, called our boot sector code which searched the disk image file for an operating system program, loaded that program into memory and called it, which displayed a message to our screen.

In this article, we will extend our operating system program to setup the required tables, interrupt vectors and handlers to support protected mode operation. In addition, we will introduce 32-bit code, data and stack segments to define a protected mode task that will write a message to the screen. Our original operating system program will actually become a 16-bit loader program that positions the 32-bit operating system kernel into memory, calls the BIOS to enter protected mode and launch our 32-bit console task by performing a far JMP instruction to a Task State Segment (TSS). Later articles in this series will extend this core architecture by adding additional interrupt service routines and console functionality.

1. The Boot Process

In our last article, we introduced a complete boot sector that loaded an operating system file into memory from disk. By way of review, the following graphic illustrates the loading and execution of the boot sector code by the BIOS. When the virtual machine is powered on, the BIOS loads the first sector of the boot disk image file (os.flp) into memory at address 7C00. The boot sector code reads each directory sector, searching for the operating system loader os.com. If this file is found, the FAT is read into memory and used to locate and read each os.com sector into memory. Once the entire program is loaded, control is transferred to the start of os.com by way of a simple JMP instruction. We introduced the first version of os.com in our last article, where it simply produced a message, "Starting ...".

The diagram below illustrates what storage areas are used by our program, both on the boot disk image and when code is loaded into memory by the boot sector. On disk, the boot sector is immediately followed by two copies of the file allocation table (FAT) and the disk directory. All disk storage after the disk directory is available to store files. Our disk image file contains one logical "file", os.com. Our boot sector code copies one copy of the FAT into memory and uses a second buffer to hold directory sectors while it searches for the directory entry for os.com. Once this entry is found, the os.com loader file (os.com) is copied from disk into memory immediately after the FAT. Once os.com is loaded, the boot sector transfers control to it by jumping to its starting address. Once os.com has control, the boot sector and the FAT are no longer needed and these memory areas could be reused.

2. Naming Conventions

Beginning with this article, our custom operating system will grow rapidly. At this stage, it is helpful to document naming conventions for variables, constants, macros, routines, etc. Keeping to a regular naming convention can improve program readability. We document our naming conventions in the program source code itself.

3. Additional Equates

This article introduces a large number of new symbolic constants, or "equates". First, we define an additional BIOS interrupt, for "miscellaneous" services, such as entering protected mode. We also begin to define constants for character codes belonging to the American Standard Code for Information Interchange, or "ASCII". Before the introduction of Unicode, ASCII was commonly used to encode printable and control characters using 7-bits per character. IBM introduced an "extended" ASCII using all 8-bits of each byte to encode an additional 128 symbols or control values. Next, we begin to define reserved codes used by the Intel processors to specify segment types. These codes are contained in segment "descriptors".

This article introduces input/ouput (I/O) to several devices. The next section of new equates defines commonly-used constants for some of these devices. Constants are provided for the 8253 Programmable Interrupt Timer (PIT), the 8259 Peripheral Interrupt Controller (PIC), the 6845 CRT Controller, and the NEC 765 Floppy Disk Controller (FDC). In modern computers, many of these devices have been integrated together and are no longer individual microchips. Nevertheless, support for the original interfaces to them is still supported.

The last set of new equates defines constants related to the operating system kernel itself. Segment selectors are really offsets into descriptor tables, the "global" descriptor table (GDT) or a "local" descriptor table (LDT). Each descriptor, as we'll see, is eight bytes in length. Console identifiers set size and positioning information for displaying text to the console.

4. Structures and Macros

This article introduces the use of structures in NASM. The structure "OSDATA" defines a map or template of memory at the start of memory (0:0h). Our operating system reuses some of this area. Since our interrupt vectors have been remapped, the default real-mode interrupt vectors are no longer relevant. Note that some BIOS routines use memory areas from 40:0h, however.

Protected mode operation uses descriptor tables to track memory areas. These descriptors identify addresses and lengths of memory areas. As we continue to add code to our program, it will not be practical to try and hard-code all the addresses of our interrupt handling routines, for example. Changing a handler will almost surely change the addresses of any code labels that follow our changes. What we need is a mechanism that can automatically determine the correct entry addresses for our interrupt handlers at assembly time. We are going to use macros to solve this problem. One macro, "menter", will be used to define the start of each interrupt handler. This macro will generate a symbolic name that represents the value of the interrupt's entry address. Our two other macros, "mint" and "mtrap", will be used to define the IDT descriptors. We will link the macros together by defining a symbolic name for each interrupt and using that name when invoking the macros. NASM macro definitions begin with the reserved work "%macro" and end with "%endmacro".

The "menter" macro takes one paramter. That is why we code "1" after the the macro name. Inside the macro, we refer to that parameter as "%1". The "menter" macro has only one line. That line defines a new label by prepending our parameter with a question mark. That symoblic name is assigned a value that represents the difference between the current assembly address and the start of the current section. This section-relative address, or "offset", is calculated using the "($-$$)" syntax. So if we coded "menter keyboard", our macro would create the symbol "?keyboard" having the value of the address of that label.

Having this mechanism to refer to the entry address of any label, we can now manipulate these values, embedding them into descriptors within the IDT. The "mint" and "mtrap" macros do this. Each of these macros also require one parameter. This parameter must also be the name of an interrupt handler identical to one used as a parameter to "menter". The macros refer to the "?" variants of the parameters to access the addresses of the interrupt handlers. Therefore, we cannot use "mint" or "mtrap" unless there is also a corresponding "menter" macro defined for the given parameter.

The macros also refer to other symbolic names we have already defined, including the selector of our interrupt section, "ESELOSCODE", and access bytes that define the descriptor as either a trap or interrupt type, "EACCTRAP" and "EACCINT". The output produced by "mint" and "mtrap" is a quad-word defining the complete descriptor for the interrupt. Now we can use these macros to define the IDT itself.

For this article, the boot sector code has not changed and is not reproduced here, although it follows next in our source code.

5. Updated File Allocation Table (FAT) and Directory

In our last article, we introduced a very simple File Allocation Table (FAT) in our disk image file. In that article, our operating system program only required a single disk cluster. In this article, our operating system program is considerably larger. Therefore, we need to map each contiguous cluster on our disk image file that will be used by our operating system file. The program comments explain how the FAT table entries are structured. Remember that the first three bytes (two entries) are reserved. Starting with logical cluster 2, (byte offset 4 into the FAT), each entry points to the next logical cluster. The final cluster ends the chain with a value of 0xFFF.

For this article, we still have only one entry in the directory. However, the size of the os.com program is significantly larger and must be adjusted in the directory entry. The new size of os.com is 5200h.

6. The Loader

In our last article, we introduced a simple loader that displayed a message. For this article, our loader will consist of two parts. The first part is 16-bit code that receives control from the boot sector. This code prepares memory areas for protected mode operation and calls the BIOS to enter protected mode. The second part is 32-bit code that acts as the kernel (or core) of the operating system.

The Loader begins by setting DS and ES to the current section address held in CS. Then a message is displayed to the screen indicating that we have successfully reached the Loader code.

Before we try to enter protected mode, we need to verify that the CPU is capable of operating in protected mode. We will introduce a subroutine, GetCPUType, that determines whether the CPU is an Intel 80386 or higher processor. If the returned value in AL is less than 3, we will display an error message, wait for a keypress and restart. Otherwise, we will continue with preparations for protected mode operation. Note how NASM supports a "cpu" assembler directive that instructs the assembler whether instructions available on a given CPU can be generated.

Having verified our CPU, we now prepare system memory areas for our switch to protected mode. Protected mode operation uses several global tables. The first table we are concerned about is the Global Descriptor Table, or GDT. This table contains a series of 8-byte "descriptors" that define areas of memory that can be read from, written to, or executed. We will examine this table in more detail later. For now, what we need to make sure of is that our current code segment - our 16-bit loader code - is defined in the GDT before we enter protected mode. Our Loader will edit the descriptor that will be referenced by our CS when we enter protected mode. The descriptor must contain the absolute memory address of our 16-bit loader code, as well as a few other fields that we will examine later. After fixing up the descriptor for our Loader code, we display a message to the screen.

Now, before calling the BIOS to enter protected mode, we want to place our 32-bit operating system kernel at a specific location in memory that corresponds to the memory areas mapped in our GDT. We could place the operating system just about anywhere we'd like. But we will put the kernel a bit lower in memory than our boot sector code has initially put us. This ensures that we have a consistent location for our operating system kernel. Had we run OS.COM from a DOS environment, for example, we might not have our kernel ideally positioned for efficient memory use. Furthermore, we would have to fix up a lot of descriptors in our GDT to match where OS.COM happened to be loaded. We position the 32-bit operating system kernel and display a message to the screen.

To enter protected mode, we will use function 89h of BIOS interrupt 15h. This BIOS service requires setting a few parameter values in registers. ES:SI must point to the GDT. We set ES to the segment of where our kernel was moved. Since the GDT is at the start of our kernel, ES:SI will point to the GDT when SI is zero. SS:SP must point to where we want our initial stack to be when we enter protected mode. We will reuse the real mode interrupt vector address space for our stack. BH and BL are used to map our hardware interrupts (IRQ 0 through IRQ 0Fh) to our interrupt vectors. We will be reserving the first 32 interrupts (0-1Fh) for CPU interrupts. We will map IRQ0-F to interrupts 20h through 2Fh. These hardware interrupts are divided among our two 8259 Peripheral Interrupt Controllers (PIC). BH must point to the first interrupt number for the primary PIC. BL must point to the first interrupt number for the second PIC. Finally, AH must contain the function code 89h to instruct our BIOS interrupt to enter protected mode.

Just before calling our BIOS interrupt, we issue a loop. This loop is intended to allow any pending interrupts to complete. After returning from our BIOS call, we are in protected mode. Then we enable all maskable interrupts.

One of the powerful features of protected mode is the ability to switch between tasks - entire register and segment mapping contexts - simply by calling or jumping to a "selector" - the offset of a descriptor, either in the GDT or in a Local Descriptor Table (LDT). We have defined in our kernel an initial task and have defined two required descriptors in our GDT - one for a "task" segment, which defines the initial register state of the task, and one for an LDT for the task. The following two instruction, LTR and LLDT, load the CPU's task register (TR) and local descriptor table register (LDTR) with selectors for our initial task. Now we merely need to JMP to our task state segment, referenced by its selector. Note that we can use a far JMP syntax instead of hard-coding the op code.

We have defined two subroutines used by our Loader. The first subroutine, "GetCPUType", determines the type of our CPU. This routine is primitive. It only detects whether the CPU is an 8086, 80186, 80286 class of CPU or is an 80386 or higher.

The second subroutine, "PutTTYString", is our familiar code to write characters to the screen using the BIOS TTY output function.

Our Loader code ends with constant values and work areas used by our Loader.

7. The Kernel

Immediately following our 16-bit loader code is the operating system kernel. All of os.com that follows the loader is 32-bit code and data that is moved to address 100:0 in memory, or physical location 1000h. Before introducing this code and data, we define a few symbolic values. Of special note, ESELDAT, ESELCGA and ESELOSCODE reference offsets to GDT descriptor entries. EKRNADR and EKRNSEG equates both represent the address of the kernel after it is relocated. The descriptor access codes are used in descriptor tables to specify what kind of memory area is being defined. The particular bit definitions of the access bytes are beyond the scope of this lab note but will be addressed in more detail in a later note.

The first code generated as part of this kernel is the Global Descriptor Table (GDT). As described above, this table and all Local Descriptor Tables (LDT), define memory areas that can be accessed. When the CPU is operating in protected mode, it will issue a CPU interrupt if any task attempts to read or write memory or execute code that the task is not authorized to access. Therefore it is very important that the GDT, and all tables for that matter, are correctly defined and maintained at all times. Sections of memory defined in the GDT are referenced by "selectors", which are the offset addresses of the descriptors. In other words the first descriptor is referenced by selector "0", the second by selector "8", etc. LDT selectors differ slightly as we will see below.

The comments in the code for the GDT describe the structure of each descriptor. Simply put, each descriptor contains an address, a size, a set of flags and an "access" byte. Since all of these values can be known at assembly-time, we've defined the GDT descriptors as fixed-value quad-words. We must simply be sure that the address parts of each descriptor matches the actual physical memory location of the operating system components when we enter protected mode.

The CPU requires that the first descriptor be set to all nulls. The BIOS routine that initializes protected mode expects to find "alias" descriptors for the GDT and the Interrupt Descriptor Table (IDT) as the second and third descriptors. An alias descriptor describes an area of system memory, such as a descriptor table, as a data area in order for tasks to read or write to these memory areas. The BIOS will use the fourth descriptor, at 0018h into the GDT, as our initial data segment. In other words, DS will contain 0018h. Since we are in protected mode, the CPU knows to interpret this value as a selector into the GDT rather than as a real mode segment. The selector 20h will be placed into ES. Our operating system uses the descriptor at 20h to map video memory at B8000h, compatible to the Color Graphics Adapter's text video modes. The selector 28h will be placed into SS and will serve as our initial stack. The selector 30h will be placed into CS and will be our initial code segment. Remember that the descriptor at selector 30h was "fixed up" by our Loader before entering protected mode so that it's address portion matched the code segment of our 16-bit Loader code. BIOS expects selector 38h to contain a descriptor defining BIOS ROM at "FF0000" and the descriptor at selector 48h to define a BIOS data area at 400h. Our operating system will reserve selector 48h for a descriptor that defines code residing at 2000h. This code will be our 32-bit interrupt handlers, accessible by any task. Selectors 50h and 58h are LDT and TSS descriptors, respectively, for our initial "task", which will manage console I/O. Selectors 60h and 68h also define LDT and TSS descriptors. However, these descriptors merely point to unused memory areas. We define these selectors so we can load our task and LDT registers with valid selectors prior to issuing a task switch to the console.

The next system table that we must define is the Interrupt Descriptor Table (IDT). Like the GDT, this table contains 8-byte descriptors. Each descriptor defines an address, various flags, and a type. Essentially, the IDT serves the same function as the real-mode interrupt vector table. Each descriptor corresponds to an interrupt number. In protected mode, when an interrupt is invoked, the CPU determines the address of the interrupt handling routine by inspecting the corresponding descriptor in the IDT.

The IDT must contain descriptors for all expected interrupts. Interrupts fall into three major categories: Processor, Hardware and Software. Processor interrupts are issued by the CPU itself when various conditions are detected, such as a division by zero or a general protection fault. Hardware interrupts are issued by the CPU when an external device, such as the clock timer or keyboard, signals the CPU on an Interrupt Request (IRQ) channel. Software interrupts are issued by application or system software using the INT instruction.

In protected mode, interrupt descriptors can be of different types, known as "gates". The types of gates include trap gates, interrupt gates, task gates, etc. Each type of gate behaves slightly differently than the other types. For example, calling a trap gate causes the flags and return address to be stored on the stack and disables interrupts on entry. An interrupt gate is similar to a trap gate but does not disable interrupts automatically on entry. A task gate causes an entire task switch, storing the callers complete register state in the calling tasks TSS and loading new register values from the called task's TSS. Our IDT will use only trap and interrupt type gates.

To see the actual descriptor bytes generated by these macros, refer to the complete program listing referenced by a link at the end of this lab note. In this lab note we will only be implementing one hardware interrupt and one software interrupt. In later lab notes we will implement more interrupt handlers as we require them.

The IDT only defines a table of descriptors that point to interrupt handling code. Now we need to define the code for these handlers. We put this code in a section that is located at physical address 2000h. This code is accessible through the GDT descriptor at selector offset 48h. In other words, loading 48h into CS causes code references to be relative to the address defined in the descriptor at GDT+48h. The address in this descriptor is 2000h.

Note that we specify a "vstart=0" for this section. This is necessary because the data areas used by the interrupt handlers are relative to the start of our data segment, not our code segment. Interrupt handlers will access system-wide data through GDT selector 18h, which defines memory starting at address 0:0. Therefore if we define constants such as tables or strings within our interrupt handling section, the offset addresses of those labels must be assembled relative to the start of our data descriptors address.

Now comes the code for our Processor and Hardware interrupt handlers. Again, we only implement code for one Hardware interrupt handler, the clock-tick interrupt. This hardware interrupt signals the CPU on interrupt request line zero. The interrupt handler increments a time-of-day counter and handles turning off the floppy disk drive motor if a latch counter reaches zero.

For this article, we define one hardware interrupt, for the clock timer. This interrupt triggers approximately 18.2 times per second. An internal counter, shared by the BIOS, is updated. Also, the floppy disk controller motor is turned off if it is time to do so.

The remaining hardware interrupts are not yet implemented. We will implement a keyboard interrupt handler in the next article. Important to note that at the end of a hardware interrupt, the peripheral interrupt controller chip must be sent an explicit end-of-interrupt signal.

We will define one software interrupt, interrupt 30h. This interrupt will be used as a single entry point for a library of operating system service routines. We will use register AL as a function code. The function code will be checked on entry to the interrupt handler to make sure the function code is valid. If the function code is valid, we will convert it to an index into an array of entry points for the services supported by the interrupt. The table of entry points, tsvc, is built using a new macro, "tsvce". This macro takes one parameter, which must be a defined code label. The macro does two things. First, it defines a symbolic code for the service, named by prepending an "e" to the code label. This symbol can be used as the function code put in AL prior to invoking the interrupt. Secondly, the macro generates a double-word containing the offset address of the label passed as the macro parameter. The "maxtsvc" symbol defines a value used in the range test to check the validity of the function code in AL.

In this lab, we define three services supported by our software interrupt: ClearConsoleScreen, PutConsoleString, and PlaceCursor. These services will be used by our console task to output data to the console, clear the screen and set the cursor position. In addition, several helper functions are defined as kernel routines. These may be called by service routines or by interrupt handlers. Note that these routines may not be called directly from the console task below. This is because the code segment used by the console task is defined in the console task LDT and is different from the code segment used by the kernel. We define macros to simplify calling our library functions. By defining these macros to call our library functions, we can ensure that the correct registers are set before calling our services and that the service routine interrupt (30h) is called.

Here is the logic for the service routines. We will expand this section greatly in later lab notes. This section will form the core of our operating system function library.

8. The Console Task

The remaining sections of our operating system file, OS.COM, define the Console Task. In protected mode, a task is executed by calling or jumping to a selector that references a Task State Segment (TSS) descriptor. The TSS, in turn, references a Local Descriptor Table (LDT) selector, which contains descriptors for memory areas accessible by the task. Tasks can access memory areas defined in both the GDT and the task's LDT.

The Console Task consists of a stack, LDT, TSS, code and a data area called a "message queue". The descriptors in the Console Task LDT define all of these memory areas. In addition, the LDT includes a data descriptor for operating system and BIOS shared data at 0:0. This descriptor is functionally equivalent to the GDT descriptor 18h. Using an LDT allows a task to maintain local control of its own data areas without having to expose the existing of its memory areas to other tasks.

The Task State Segment (TSS) defines what the initial contents of registers will be on entry to the task. In addition, the TSS supports up to four different stack addresses, one for each "ring level" in protected mode. The ring levels implement a hierarchical security framework of privilege levels where "ring 0" is the highest level and "ring 3" is the lowest level. Code executing at a lower level is restricted from accessing code or data at higher levels, except through specific, controlled access mechanisms. Typically, more sensitive code such as the operating system kernel will run at a higher privilege level than user or application level code. In this lab, all code operates at privilege level 0. In later lab notes, we will introduce logic to traverse different privilege levels

We will use the "message queue" section, conmque, as a first-in-first-out (FIFO) queue for passing messages between the keyboard interrupt handler and the Console Task. For this lab, we will simply define the section. In a later lab we will add logic to respond to keyboard events and place "messages" in the message queue to be read and processed by our Console Task code.

Our Console Task code section is very simple. After calling the local subroutine ConInitializeData, it makes four calls to operating service routines defined above using macros. The first call uses the macro, clearConsoleScreen. Next, the putConsoleString macro is invoked twice, once to display a title string and again to display a prompt character ("#"). Lastly, the placeCursor macro is called to position the console cursor after the prompt character.

To see the macro expansion code, review the complete listing of this program available at the end of this lab note.

As we have seen before, our code terminates in a HLT loop. In our next article, we will expand this Console Task code significantly, adding logic to process messages that arrive in the message queue from interrupt handlers. Instead of simply entering a HLT loop, the Console Task will loop through a "GetMessage" type of call to wait for user input to process.

Our code for this lab note ends with the constants required by the Console Task and the remaining unused disk image space.


In this article we have expanded our operating system program, "os.com", to enter protected mode and start a console task that displays a message on the screen.

In our next article, we will expand on this code, adding a keyboard interrupt handler that will add messages to our Console Task message queue. We will also add code to our Console Task to process messages from the message queue, reposition the cursor, recognize commands entered by the user and respond to a set of commands.

Program Notes

Here is a link to the entire listing of the code and data from this lab.

Program Listing

Revised 10 October 2014