When visiting real-time operating system (RTOS) vendors in their booths at design conferences, it’s easy to assume that all RTOSes do the same basic things: allocate processor time, trick the application into believing it can do activities on more than one processor, and split an application into pieces to speed development. But despite many apparent similarities between RTOS solutions, there are definite functional differences, some subtle and some major, reflecting a variety of factors such as time to market and return on investment. What are the causes of those differences: random or evolutionary changes due to response to market influences; specific target application requirements; RTOS designer motivations, education, and experience; or some combination of all of these? as someone who grew up with the embedded systems industry and who lives and breathes–and designs–RTOSes, William “Bill” Lamie is well positioned to provide some valuable insight into this question. Bill has personally architected, developed and distributed three successful commercial RTOS products since 1990–Nucleus RTX, Nucleus PLUS, and ThreadX, the latter two still active in today’s market. His RTOSes now run in more than 2.5 billion embedded devices, including market-leading consumer products such as the Apple iPhone and Hewlett Packard’s ink-jet printers. Bill also invented and implemented preemption-threshold, a new approach to eliminate unnecessary context switches while managing critical sections. What’s ironic in this story is that Bill didn’t start out intending to build an RTOS. In fact, he didn’t even intend to go into computer science. His dad, a computer science Ph.D. and university professor, had tried to interest his son in computers when Bill was still in high school, but to no avail. Bill was mainly interested in sports and rejected all attempts to interest him in the world of computer nerds. When he started college at San Diego State University, he took one course in computer science to “get my dad off my back for good,” he says. But a week into that one course, Bill fell in love with computer science, despite the dreariness of key punch cards and UNIVAC systems–technology’s finest at the time. he got his first taste of embedded computing in his junior year when he signed up for a small computer lab where he built programs with a C compiler on a Unix-based system and then downloaded S-records to a Motorola 68000. The next summer, Bill got a job helping to develop an operating system for a military application. some years later, Bill was contracting with a company that had paid big bucks for an RTOS for an AMD29k processor, but the RTOS well. “It was ridiculous that this RTOS was really expensive when it didn’t even work, was royalty bearing, and included no source code.” Bill was sure he could do better. certainly, he couldn’t do worse. So Bill spent his evenings trying to fix the RTOS that wouldn’t work right. I fixed the operating system by inserting patches into its instruction area at run time. since we had no source code, we had to copy it to RAM and patch the machine code there. once I had the client’s RTOS working, I designed and wrote what eventually became Nucleus RTX with the primary goal of making it simple and easy to use–a direct contrast to the existing RTOS I’d been working with, which, in addition to having problems with functionality, had a terrible API. When he designed Nucleus RTX in 1990, Bill’s intent was to provide the baseline functionality developers needed and wanted. he also wanted to make source code available so his customers could extend RTOS functionality and because he had disliked the black box non-source code RTOS that he had been working with. Influenced largely by his experience working on the client implementation on the AMD29k, Bill kept things simple in Nucleus RTX. In meeting just the baseline needs, he avoided complexities like dynamic object creation, variable length messages, and priority inheritance. Developers were confined to creating all the threads in a statically defined C array. once the system was running, no more threads could be created. When Nucleus RTX was ready for customer use, Bill and partner Neil Henderson founded Accelerated Technology and began to market the RTOS for embedded systems. Bill eventually recognized that his ignorance of broader market needs had caused him to miss the mark on some fundamentals of real-time control. The number and capabilities of processors grew, and they became able to tackle larger, more complex control issues. Motorola had extended the 68K line and introduced ColdFire and PowerPC. Others like Hitachi, Fujitsu, MIPS, and Texas Instruments had introduced new 32-bit RISC processors. The market had evolved to designs with needs for greater real-time control and ports that harnessed the superior capabilities of these processors. And developers wanted an RTOS with more guts–one able to conquer these increasingly complex challenges. Responding to the changing market and the need for more advanced technology, Bill went all out with his next RTOS, ultimately named Nucleus PLUS. “In Nucleus PLUS, I included dynamic object creation, as well as ways to send variable length queue messages, fixed-length queue messages, all sorts of different options on queue messages.” Bill contrasts his first and second RTOS solutions, “The original Nucleus RTX, functionality-wise, was a much closer fit to my old client’s operating system that didn’t work and the needs of their application. When I started work on Nucleus PLUS, I began by first studying every single RTOS publicly available at that time.” Using a list of all the fun
ctionalities of every RTOS, Bill came up with a feature set that tried to address everything: “With Nucleus PLUS, I went exactly in the opposite direction from RTX and offered everything under the sun. The API involved hundreds of services instead of the 20-30 of the original Nucleus. while the first time I hadn’t really understood the varied needs of the embedded market, I now had several years of experience with Nucleus RTX and I certainly knew what all the other products had. “The homework paid off. Nucleus PLUS, released in 1993, proved a much more successful RTOS than its predecessor Nucleus RTX. Mentor Graphics is still selling Nucleus with–I believe–pretty much the same functionality that Nucleus PLUS had when I left Accelerated in 1995. There’s maybe only a handful of APIs that have been added. It’s lasted more than 15 years and has found its way into many, many systems.” But this time Bill had overcorrected. After Nucleus PLUS was released he realized this approach was overkill. Not only did all that extra functionality result in extra overhead to handle fringe use cases, but the code was more complex, a lot bigger, and harder to maintain. “Shortly after releasing Nucleus PLUS, I had two primary regrets. The complexity that I had created with the number of APIs, and an interrupt architecture that I’d created to try to address interrupt latency.” To address the API complexity, Bill realized he should have simplified his design. instead of offering a dozen ways to deal with messages, for instance, Bill realized that fixed-length messages would have been fine and any variable length messages could simply have been passed through a pointer. Similarly, Nucleus PLUS used a segmented interrupt architecture that deferred some of the ISR processing to the thread level. while Bill’s first RTOS could call thread services from the ISR, it would cause interrupt lockouts at the thread level, locking out the whole API. To correct this in Nucleus PLUS, Bill decided he wanted to leave interrupts enabled a little bit longer. To do so, he segmented the interrupt processing so that the ISR could be stalled if it collided with something the thread level was doing. After releasing Nucleus PLUS, Bill realized the shortcoming of this approach: “Unfortunately, my solution required all sorts of overhead with full context save-and-restore on every interrupt, and the need to schedule the piece of the ISR that runs at the thread level.” although Nucleus PLUS was a very successful product, Bill recognizes its shortcomings and attributes them to human error: “I fell into the trap of overdesigning. I had a problem and I compensated for that problem, and then I found that I had gone too far in the other direction, but I realized this only after we had already gone to market with it.” Years later, in 1995, presented with a buyout offer from his business partner, Bill decided to sell his share of Accelerated Technology. In so doing, Bill opened the door to an opportunity very few developers get: he could start from scratch and correct the weaknesses of his earlier designs. After fulfilling the noncompete agreement between himself and Accelerated Technology, Bill set about the goal of creating a better RTOS than his first two, and this time he had no compatibility issues to constrain him. Third time’s a charm Bill’s third RTOS, ThreadX, released by his new company, Express Logic, in 1997, represented, in his words, “the balanced sweet spot between Nucleus RTX’s overly simple and Nucleus PLUS’s overly complex designs.” Bill summarizes his three primary improvements in ThreadX as reducing the number and complexity of the API, optimizing for the 90% use case instead of attempting to cover all fringe possibilities, and changing the interrupt architecture dramatically to improve efficiency. Bill feels that “such simplification resulted in lower overhead, fewer processor cycles, and code that was easier to read, easier to maintain, and smaller in size.” gone were the multiple ways to handle queue messages and an interrupt architecture segmented between the ISR and thread levels. Bill’s new interrupt structure relied on very short times with interrupts disabled. “In ThreadX, I realized that I could reduce the interrupt latency with small areas with interrupts disabled, and the ability to throw away some work when ISR calls to the kernel collide with an interrupt call. It’s kind of involved, but the net result is that it’s much easier to write ISRs, the overhead is way less, and you can do optimizations that you can’t do with a segmented operating system.” Preemption-threshhold Bill knew that just about every RTOS has the ability to turn off the scheduler when nothing else should run at the thread level. Typically, it’s done by making an API call that stops all thread scheduling until you make another API call that allows scheduling to happen again. In implementing ThreadX, Bill realized that it would be “kind of neat to make this a little more selective.” It was his experience that only a certain class of threads or priorities should be prevented from running. Preemption-threshold allows the developer to select a priority range where scheduling is turned off whil&
#101; still allowing threads of higher priorities to interrupt and preempt at the same time this critical section is going on.” Bill likes the control this offers programmers: “It’s a more granular critical section mechanism than other operating systems use that I just happened to think of during the development days of ThreadX. It’s turned out to be a nice benefit that a lot of embedded system developers now use.” asked how important it was to the creation of ThreadX that he had already written Nucleus RTX and Nucleus PLUS, Bill states that both were hugely important. In fact, he takes it back one step further, “Had I not responded to an ad on a 3×5 index card on a San Diego State bulletin board to help write an operating system for a military application, I wouldn’t have had the initial operating system experience and would have never even thought of writing an operating system myself.” that experience paved the way for Nucleus RTX, which in turn led to Nucleus PLUS. ThreadX represents the summation of all of these experiences. Today’s need for middleware and tools although Bill notes that the basic definition of an RTOS remains unchanged since 1990 when he created Nucleus RTX, hardware changes and market demands have altered expectations for RTOS technology. while all of Bill’s RTOSes were designed to be portable and to deliver rapid real-time performance, the impact of these goals shifted dramatically after 1990. The number of processors and tools grew significantly, and customers lined up for off-the-shelf ports. And, with microprocessors packing more punch than a desktop from just a few years earlier, real time got a whole lot tighter. Battery-powered gadgets needing to preserve power in the field definitely required a level of power conservation unheard of in the past. But the biggest difference between 1990 when Bill wrote his first RTOS and the present is connectivity. “When I designed RTX, the number of embedded gadgets that actually had to talk to one another was really few,” remembers Bill. “Then 90% of the customers just needed a small real-time operating system scheduler and that’s it. I didn’t even think about creating a network stack for RTX.” Bill figures that most embedded devices now need to be connected. The need for connectivity and tighter time to market has driven many changes for RTOS vendors. According to Bill, today’s RTOS provider cannot just provide a kernel. Today’s developer needs middleware and help analyzing real-time applications that debuggers don’t provide. (see sidebar “Get connected” for more on connectivity tools)
Getting Connected To provide connectivity, Bill first partnered with connectivity providers since it wasn’t clear which standards were going to win. however, as TCP/IP emerged, Bill tightly coupled ThreadX with a network stack (NetX) to offer the same low overhead, efficiency and ease of use and to reduce the support burden of partnering with other vendors whose definition of quality software was not the same. Demands for USB connectivity—unheard of in 1990—sparked the creation of another product, USBX. as part of his mandate to reduce the developer’s burden, Bill has also recently tackled the development of analytical tools that reach beyond the purview of typical debuggers. To Bill, the day has passed when all that a developer needs is a just an RTOS and traditional development tools. Bill’s first such tool (see Figure 1) delivers all the punch of a software logic analyzer, fully instrumenting and graphically displaying all RTOS activities thread by thread. The benefits? “Invariably, you see there’s something there you weren’t expecting—three context switches between here and here, for instance, or an unbounded priority inversion. You realize you can rearrange your priorities, avoid the inversion, and reduce overhead.” The tool also serves as a system black box. Developers can set it up to provide a memory dump of what goes on in their system immediately prior to a crash. The second tool (Figure 2) is patent pending and analytically determines how much memory should be allocated for a system stack. It produces a call tree, as well as other summaries of system structure, all derived from an .elf file. Tracing through the call tree, this new tool determines how much memory is needed for a stack in the worst case scenario, avoiding what Jack Ganssle refers to as the “guess and hope” approach of the past, and the notorious challenge of tracking down overwritten variables or return addresses. “The developer’s job is getting tougher and tougher all the time,” he says. “They need more turnkey solutions with operation out of the box and without going to partners or anything else. Time to market and trying to offload the hellacious tasks our customers have in trying to get these embedded gadgets done on time and budget drives the creation of our middleware and tool options.”
Click on image to enlarge. Click on image to enlarge. Proof in the pudding Toss a coin in the air and you’d think that you’d always get a head or a tail. Not so, it turns out. Sometimes the darn thing flips out of your hand and goes skittering around until it comes to rest edge up against the leg of a chair. It turns out that the same is true of RTOS technology. some changes since Bill’s RTOS beginning
in 1985 definitely track to that age-old wisdom. Bill is the first to admit that his three RTOSes certainly represent that all too human pendulum of going from too simple to too complex, before eventually settling into a middle “just right” position. some changes clearly arise from market changes and new customer demands. Previously unthought-of network stacks, processor speeds, and time-to-market pains drive the current RTOS vendor into avenues never believed necessary before. But, what are the edge factors? those personal signature lines that imprint themselves on every piece of software a designer ever creates? For Bill, the edge comes from the KISS principle (Keep it simple, stupid). He’s absolutely committed to providing core functionality that’s the easiest thing in the world to use. “I always start by writing the user manual first. I want to look at the problem from the user’s perspective from the get go. I’ve found that if I can’t explain how the API will work to a user, then it’s too complex. And, since I’m describing what will happen versus what’s already been written, it’s easy to change. When you write the documentation first, you can go to great lengths to making the software easy for them to use.” of course, Bill concedes that “easy” is in the eye of the developer. When it gets down to quantifying it, Bill acknowledges that it’s really just a matter of style. is the API consistent? Are the names good? can you tell from just looking at the code what it does? Are the basic services well represented in the API? how easy is it to port? “I try to not overly complicate my code.” This concern for simplicity extends to business. Bill believes developers should get the source code with their RTOS, they should pay no royalties, require no license keys or software dongles, and have a license agreement that’s a page and a half. For Bill, the bottom line has always been developing an RTOS he would like to use. has he succeeded? It appears that developers who use Mentor Graphic’s Nucleus in 2.11B devices and those using Express Logic’s ThreadX for 750 million devices think so. Bernard Cole is the site editor for Embedded.com and the aquisitions editor for Embedded Systems Design magazine. since the mid-1970s, he has worked as a technology journalist for most of the electronics trade magazines and as a freelance editor for many of the major semiconductor hardware and software companies. he is a partner in Techrite Associates, LLC. Toni McConnel is a contributing editor for Embedded Systems Design and Embedded.com and is a partner in Techrite Associates, LLC.
