As technology continues to advance, issues of signaling, timing, power, and noise become increasingly important. At high signaling frequencies wires can no longer be considered equipotential regions and must be modeled as transmission lines. The number and speed of gates on a chip have increased faster than the number and speed of pins, making inter-chip communication a system bottleneck and placing a premium on efficient signaling and timing conventions. With reduced supply voltages and higher currents, power distribution becomes a more challenging engineering problem. At high frequencies timing conventions must be carefully designed to prevent skew and jitter of clock signals from degrading system performance. Because of these trends, the techniques described in this book, once used only in supercomputers, are now essential to the correct and efficient operation of any type of digital system.

We were motivated to write this book for two reasons. First, in our research building high-speed digital systems (parallel computers, network switches, and high performance graphics systems) we developed a number of signaling, timing, and power distribution techniques that overcame the limitations of conventional methods. We were eager to share these methods and the engineering science behind them with other practicing engineers.

Our second motivation was to avoid repetitions of many disasters we encountered in our interactions with industry. Over a year was spent at one company chasing noise problems before a system would operate reliably. Another system went through several iterations of ASICs due to timing problems. A third system failed periodically due to on-chip power supply fluctuations. A fourth system product was delayed by six months because of a subtle failure in the flip-flops used throughout a custom chip design. These problems delayed system delivery by months to years and in some cases directly contributed to the failure of companies. Band-aid fixes rarely exist for these types of problems; however, they could have been easily avoided by proper design if the engineers involved had been knowledgeable about noise, signaling, timing and power. By writing this book we hope to help eradicate the widespread ignorance, and often misinformation, in these areas, and in doing so, help avoid disasters of this kind in the future.

Organization

The book begins by describing the major engineering problems of digital systems: power, noise, signaling, and timing. It presents the best known techniques for dealing with these problems and describes working circuits that implement these techniques. At all levels material is developed rigorously, deriving equations from first principles. The book avoids the handbook approach of describing how things are usually done. While it gives a good description of current industrial practice, it emphasizes why things are done, points out the limitations of current approaches, and describes emerging techniques, such as low-voltage signaling and closed-loop timing, that are likely to be how things are done in the future.

The book opens introducing the topic of digital systems engineering by describing the major engineering problems associated with digital systems and the technology trends affecting them. The book then spends three chapters laying the groundwork for the treatment of these problems by describing computer elements: wires and circuits. These chapters examine the components and interconnections used to build modern digital systems and develop engineering models of these components to facilitate the study of systems engineering problems in later chapters.

After this preamble, the book deals with the problem of power distribution in Chapter Five. Digital logic requires a stable, quiet, DC supply voltage while drawing a large AC current with frequency components comparable to signal rise times. This chapter develops circuit models for both on-chip and off-chip power distribution networks and loads. Off-chip distribution methods including bypass capacitors, local regulation, shunt regulators, and clamps are discussed, as are on-chip distribution methods including power grid layout, on-chip regulation, and symbiotic bypass capacitors. The chapter closes with the presentation of an example distribution system. Since power supply noise is one of the largest noise sources in digital systems, this chapter lays the groundwork for the treatment of noise in Chapter 6 which in turn lays the groundwork for the discussion of signaling conventions in Chapter 7.

Noise, the topic of Chapter 6, is one of the most important factors in the engineering of digital systems, yet it is also one of the least understood. Most noise in digital systems is created by the system itself, and hence is truly interference and not noise. This chapter gives an in-depth treatment of major noise sources including power supply noise, signal return coupling, crosstalk, inter-symbol interference, and parameter offsets. Lower-level noise sources including alpha particles, thermal noise, shot noise, and flicker noise are briefly treated. The engineering models of these noise sources are then used to introduce the use of noise budgets, the concept of noise immunity, and the calculation of error rate based on statistical noise models. This treatment of noise prepares the reader to understand the major problem in the design of signaling systems in Chapters 7 and 8.

Signaling, the method used to transmit a bit of information from one location to another, is central to the design of digital systems. A signaling convention involves encoding information into a physical quantity (typically current or voltage), providing a reference against which this quantity is measured, the design of transmitters to couple signal energy into the transmission medium and terminators to absorb energy and prevent unwanted reflections, and a method for controlling signal transitions to limit the spectrum of transmitted energy. The signaling convention determines to a large extent the reliability, speed, and power consumption of a system. A good signaling convention isolates a signal from noise, providing noise immunity, rather than attempting to overpower the noise with noise margin. Most signaling conventions in common use are quite poor, based on standards that are accidents of history rather than on careful design to isolate noise sources. For this reason, many modern systems define their own signaling conventions rather than employ these standards.

Chapter 7 deals with the basics of signaling over LC transmission lines and lumped LRC interconnect. This chapter introduces the concepts of voltage-mode and current-mode transmission, unipolar and bipolar signaling, series and parallel termination, references, and differential signaling. Undertermination, rise-time control, pulsed signaling, and multi-level signaling are also described. We develop methods for calculating the signal magnitude required to transmit a signal reliably in the presence of noise and shown that, with adequate noise isolation, very low voltage signal swings (100mV) are sufficient.. Chapter 8 continues this topic describing more advanced concepts including techniques for dealing with lossy RC and LRC transmission lines, simultaneous bidirectional signaling, and AC signaling.

A digital system uses a timing convention, along with a signaling convention to govern when to place a new symbol on a signal line and when to sample the line to detect the symbol. A good timing convention is one that maximizes performance by reducing and compensating for sources of timing noise, skew and jitter. It is the uncertainty in the timing of a signal, not the delay, that limits the rate at which a system can operate. Chapter 9 discusses the fundamentals of timing conventions. The methods used to encode timing on signals are described and the concepts of timing noise and timing budgets are introduced. The chapter goes on to develop methods for minimizing skew and jitter using both open-loop and closed-loop timing. A discussion of clock distribution techniques closes the chapter.

Closely related to timing is the problem of synchronization, determining the relative order of two events, which is discussed in Chapter 10. Synchronization is required when sampling asynchronous inputs into a synchronous system, or when a signal traverses the boundary between two clock domains. When a signal is synchronized there is some probability that the synchronizer will be driven into a metastable state, unable to resolve which event came first, and stay in that state long enough to cause a synchronization failure. This chapter introduces the problem of synchronization, describes metastability, synchronization failure, and gives methods to calculate the probability of synchronization failure. A section on synchronizer design recognizes several special cases of synchronization and describes how to build fast synchronizers that exploit these special cases to avoid the normal synchronizer delays. Finally the chapter describes methods for asynchronous design that avoid the problem of synchronization.

Circuits for signaling and timing are described in Chapters 11 and 12 respectively. Chapter 11 presents working circuits for the transmitters, receivers, and terminations needed to build the signaling systems described in Chapters 7 and 8. Particular attention is paid to techniques for managing noise and compensating for process, voltage, and temperature variations. Timing circuits are described in Chapter 12 including flip-flops, delay lines, VCOs, phase comparators, and clock drivers. The emphasis here is on minimizing skew and jitter. Each of these chapters closes by describing an example system that ties together the concepts of Chapters 7 through 10 with the circuit details of Chapters 11 and 12.

Teaching Digital Systems Engineering

The best way to empower engineers with mastery of digital systems engineering is to teach this material as part of the undergraduate curriculum in electrical engineering and computer engineering. There is currently a gap in the curriculum between circuit design, which covers the electrical design of individual logic and memory elements, and logic design and architecture, which deal with the logical organization of digital systems. System-level electrical issues, while critically important, are absent from the curriculum.

This book is intended to fill this gap in the electrical engineering and computer engineering curriculum. It is written as a textbook with the material presented in an order suitable for teaching and with exercises at the end of each chapter. The material is at a level appropriate for seniors or 1st year graduate students in electrical engineering. Drafts of the book have been used to teach digital systems engineering courses at MIT (by Dally) and Washington University (by our colleague Fred Rosenberger). Starting with Autumn Quarter 1998, a course on digital systems engineering based on this book, EE273, will be offered at Stanford University.