06.5 Verification & Testing

Lecture from: 16.03.2023 | Video: YT

Agenda

• Circuit Verification & Testing
  • How to know a circuit works correctly
  • Functional verification & testing
  • Timing verification & testing
  • Offline vs. online testing

Verification

Continuation from last lecture

Part 3: Circuit Verification

This section focuses on circuit verification, addressing how to ensure a designed circuit works correctly, both functionally and in terms of timing.

How Do You Know That A Circuit Works?

• When you design a circuit, key questions arise:
  • Is it functionally correct? (Does it perform the intended logic?)
  • Even if logically correct, does the hardware meet all timing constraints? (Will it operate reliably at the desired speed?)
• How can you test for:
  • Functionality? (Logical correctness)
  • Timing? (Performance and reliability)
• Answer: Simulation tools!
  • Formal verification tools (e.g., SAT solvers): For mathematical proof of correctness.
  • HDL timing simulation (e.g., Vivado): To simulate and analyze timing behavior.
  • Circuit simulation (e.g., SPICE): For detailed, low-level circuit simulation.

Testing Large Digital Designs

• Testing can be the most time-consuming stage in the design process.
  • Ensuring functional correctness across all logic paths.
  • Verifying timing, power, and other aspects for all circuit elements.
• Low-level (circuit) simulation is significantly slower than high-level (HDL, C) simulation.
• Solution: Split responsibilities to manage complexity and simulation time:
  1. Check only functionality at a high level (e.g., C, HDL).
     • High-level simulation is relatively fast, enabling high code coverage.
     • Easier to write and run tests at a higher level of abstraction.
  2. Check only timing, power, etc., at a low level (e.g., circuit).
     • Skip functional testing at this level for efficiency.
     • Instead, test for functional equivalence to a verified high-level model.
       • This is harder than high-level functional testing but easier than exhaustive functional testing at the low level.
• We have various tools to handle different levels of verification:
  • Logic synthesis tools: Guarantee functional equivalence between high-level logic and the synthesized circuit-level description.
  • Timing verification tools: Check all circuit timings against specified constraints.
  • Design rule checks: Ensure that physical circuits are buildable according to manufacturing rules.
• The task of a logic designer is to:
  • Provide functional tests to ensure the logical correctness of the design.
  • Provide timing constraints (e.g., desired operating frequency) to guide implementation and verification.
• Tools and circuit engineers then determine if the design can be physically built and meet specifications.

Part 4: Functional Verification

Functional Verification

• Goal: To check the logical correctness of the design. Does the circuit perform the intended Boolean function?
• Physical circuit timing ($t_{setup}/t_{hold}$) is typically ignored at this stage, focusing solely on logical behavior.
  • Simple checks might be implemented to catch obvious functional bugs.
  • Timing verification is addressed separately later.
• Two primary approaches to functional verification:
  1. Logic simulation: Using test routines in languages like C/C++/Verilog.
  2. Formal verification techniques: Employing mathematical methods to prove correctness (not covered in this course).
• In this course, we primarily use Verilog for functional verification, leveraging its simulation capabilities.

Testbench-Based Functional Testing

• Testbench: A dedicated module created specifically to test another design module (the DUT - Device Under Test).

• Testbench Operation:
  • Test Pattern Generator: Provides input stimuli (test patterns) to the DUT. These patterns can be:
    • Hand-crafted values: Specific test cases designed manually.
    • Automatically generated: Sequential or random values generated by the testbench itself.
  • DUT (Device Under Test): The circuit module being tested.
  • Output Checking Logic: Compares the outputs of the DUT against expected correct outputs. The expected outputs can be:
    • Hand-crafted values: Pre-calculated expected outputs for given inputs.
    • A “golden design”: Output from a known-correct reference model of the circuit.

Testbench Characteristics

• A testbench is primarily HDL code (Verilog in our case) but can also involve circuit schematics in some contexts.
• Crucially, a testbench is not designed for hardware synthesis. It is purely for simulation and verification purposes.
  • Testbenches run only in simulation environments, such as HDL simulators (like Vivado Simulator) or circuit simulators (like SPICE).
• Testbenches utilize simulation-only constructs that are not synthesizable into hardware, such as:
  • Time delays (#10ns, wait 10ns).
  • Ideal voltage/current sources.
• Testbenches are therefore not suitable for physical implementation; their sole purpose is to facilitate verification through simulation.

Common Verilog Testbench Types

Verilog testbenches can be categorized based on their approach to input/output generation and error checking:

Example DUT

To illustrate different testbench types, we consider a simple example DUT, a sillyfunction module implementing the boolean function: $y=(\overline{b}\cdot \overline{c})+(a\cdot \overline{b})$

module sillyfunction(input a, b, c, output y);
  wire b_n, c_n;
  wire m1, m2;
 
  not not_b(b_n, b);
  not not_c(c_n, c);
 
  and minterm1(m1, b_n, c_n);
  and minterm2(m2, a, b_n);
  or out_func(y, m1, m2);
 
endmodule

This DUT, while structurally verbose, serves as a straightforward example for demonstrating testbench methodologies.

Useful Verilog Syntax for Testbenching

Verilog provides specific syntax constructs that are particularly useful in testbenches:

module example_syntax();
  reg a;
 
  initial 
  begin 
    a = 0;             // set value of reg: use blocking assignments
    #10;              // wait (do nothing) for 10 ns
    a = 1;
    $display("printf() style message!"); // print message
  end
 
endmodule

• initial block: Executes only once at the start of simulation, unlike always blocks which can execute repeatedly.
• #delay: Introduces a time delay in simulation time units (e.g., #10 waits for 10 time units).
• $display("message");: A system task similar to printf() in C, used to display messages during simulation.

Simple Testbench

• Verilog Code Example: A testbench1 module demonstrates a simple testbench approach.

module testbench1(); // No inputs, outputs
  reg a, b, c;       // Manually assigned
  wire y;          // Manually checked
 
  // instantiate device under test
  sillyfunction dut (.a(a), .b(b), .c(c), .y(y) );
 
  // apply hardcoded inputs one at a time
  initial begin
    a = 0; b = 0; c = 0; #10; // apply inputs, wait 10ns
    c = 1; #10; // apply inputs, wait 10ns
    b = 1; c = 0; #10; // etc .. etc..
    c = 1; #10;
    a = 1; b = 0; c = 0; #10;
  end
 
endmodule

• Output Checking: In a simple testbench, output verification is typically done manually by examining waveform diagrams generated by the simulator.

• Pros of Simple Testbench:
  • Easy to design and implement.
  • Effective for testing a few specific input combinations, especially corner cases.
• Cons of Simple Testbench:
  • Not scalable to handle a large number of test cases.
  • Output verification is manual, making it time-consuming and error-prone for complex designs.

Self-Checking Testbench

• Verilog Code Example: testbench2 module illustrates a self-checking testbench.

module testbench2();
  reg a, b, c;
  wire y;
 
  sillyfunction dut(.a(a), .b(b), .c(c), .y(y));
 
  initial begin
    a = 0; b = 0; c = 0; #10; // apply input, wait 10ns
    if (y !== 1) $display("000 failed."); // check result
    c = 1;             #10;
    if (y !== 0) $display("001 failed.");
    b = 1; c = 0;       #10;
    if (y !== 0) $display("010 failed.");
  end
 
endmodule

• Pros of Self-Checking Testbench:
  • Still relatively easy to design.
  • Good for testing specific inputs and corner cases.
  • The simulator automatically prints an error message whenever a test fails, automating the error detection process.
• Cons of Self-Checking Testbench:
  • Remains not scalable for millions of test cases.
  • Still prone to errors in hardcoded expected values within the testbench.
  • Debugging can be challenging when errors occur, as it’s not always clear if the issue lies in the DUT or the testbench itself.

Self-Checking Testbench using Testvectors

• Testvector File: A more scalable approach involves using a testvector file, which lists input combinations and their corresponding expected outputs.
• Example Testvector File (testvectors.tv):

$ cat testvectors.tv
000_1
001_0
010_0
011_0
100_1
101_1
110_0
111_0
...

• Testbench with Testvectors Design: A “clock signal” is used to synchronize the application of inputs and the checking of outputs, with one testvector processed per “clock cycle.”

• Verilog Code Example (Testbench3): A more complete Verilog testbench example (testbench3) demonstrates how to read testvectors from a file, apply them to the DUT, and automatically check the outputs.

• Self-Checking Testbench with Testvectors - Pros:
  • Still relatively easy to design.
  • Good for both specific and more extensive testing.
  • Automated error detection.
  • No need to modify hardcoded values in Verilog for different test sets; testvectors are managed externally in a file.
• Self-Checking Testbench with Testvectors - Cons:
  • Can still be error-prone, depending on how testvectors are generated and verified.
  • Scalability is improved but still limited by the size and management of the testvector file, especially for very large designs.

Automatic Testbench

• Golden Models: To achieve fully automatic verification, an automatic testbench employs a “golden model” – an independent, highly trusted model of the circuit’s intended behavior.

module golden_model(input a, b, c, output y);
  assign y = ~b & ~c | a & ~b; // high-level abstraction
endmodule

• In an automatic testbench, the DUT’s output is directly compared against the output of the golden model for the same inputs.

• Verilog Code Example:

module testbench1();
  ... // variable declarations, clock, etc.
 
  // instantiate device under test
  sillyfunction dut (a, b, c, y_dut);
  golden_model gold (a, b, c, y_gold);
 
  // instantiate test pattern generator
  test_pattern_generator tgen (a, b, c, clk);
 
  // check if y_dut is ever not equal to y_gold
  always @(negedge clk)
  begin
    if(y_dut !== y_gold)
      $display(...);
  end
 
endmodule

• Automatic Testbench - Pros:
  • Output checking is fully automated, enabling extensive testing.
  • Potential for timing comparison using a golden timing model (though not detailed here).
  • Highly scalable, allowing for as much simulation time as feasible to achieve high input space coverage.
  • Promotes better separation of design and verification roles.
• Automatic Testbench - Cons:
  • Creating a correct and reliable golden model can be very difficult, potentially as complex as the DUT itself.
  • Generating effective and comprehensive testing inputs remains a significant challenge.

However, Even with Automatic Testing…

• Exhaustive testing (brute-force) is often infeasible, especially for circuits with a large number of inputs.
• For example, testing a 32-bit adder would require an astronomical amount of time to cover all $2^{64}$ possible input combinations.
• Therefore, even with automatic testbenches, smart test strategies are needed to prune the testing space and focus on critical or representative test cases.
• Verification remains a fundamentally hard problem, often requiring a combination of techniques, including formal verification, intelligent test case generation, and extensive simulation.

Part 5: Timing Verification

Timing Verification Approaches

• High-level simulation (e.g., C, Verilog):
  • Can model timing by embedding delays using constructs like #delay statements directly within the DUT’s Verilog code.
  • Useful for early, hierarchical modeling to get a general sense of timing behavior.
  • However, timing models at this level are typically not as accurate as real circuit timing, as they are estimations and not based on physical layout and circuit characteristics.
• Circuit-level timing verification:
  • Requires synthesizing the design to actual circuits first to obtain a detailed gate-level netlist.
  • Timing verification tools then analyze this netlist, considering specific technology libraries and layout effects.
  • This approach is very design flow specific, often relying on special tools provided by FPGA/ASIC vendors (e.g., Xilinx Vivado) or specialized EDA tool vendors (e.g., Synopsis/Cadence).

The Good News

• Modern EDA tools are designed to assist designers in meeting timing constraints automatically.
  • Tools attempt to optimize the design to satisfy setup times, hold times, and minimize clock skew.
• These tools usually provide detailed ‘timing reports’ or ‘timing summaries,’ which are crucial for identifying and resolving timing issues:
  • Worst-case delay paths: Identify critical paths that limit the circuit’s speed.
  • Maximum operation frequency: Estimate the highest clock frequency at which the circuit can reliably operate.
  • Any timing errors that were found: Report violations of setup and hold time constraints.

The Bad News

• Despite advancements in EDA tools, they are not always perfect and can fail to find a timing-correct solution.
  • The desired clock frequency might be too aggressive for the given design and technology.
    • This can lead to setup time violations on particularly long paths that the tool cannot optimize sufficiently.
  • Too much logic on clock paths can introduce excessive clock skew, making timing closure difficult.
  • Timing issues related to asynchronous logic are notoriously hard to resolve automatically.
• However, timing verification tools are invaluable in debugging and optimizing timing issues.
  • Reports generated by these tools can be very helpful, as they pinpoint paths that fail to meet timing requirements.
  • These reports provide a starting point for designers to focus their debugging and optimization efforts.
• Q: How can we fix timing errors? This often requires manual intervention and design adjustments.

Meeting Timing Constraints

• Meeting timing constraints, especially stringent ones in high-performance designs, is often a manual, iterative process.
• Strategies for addressing timing violations:
  • Iterate synthesis and place-and-route with different tool options: Experiment with various synthesis and implementation settings, optimization strategies, and even different random seeds to explore the design space.
  • Manually provide hints for place-and-route: Guide the tools by providing specific constraints or directives to influence placement and routing decisions.
  • Manually optimize the reported problem paths: Focus on the critical paths identified by timing reports and manually refine the design:
    • Simplify complicated logic: Reduce the complexity of logic gates and expressions in critical paths.
    • Split up long combinational logic paths: Break long combinational paths by inserting pipeline registers to reduce the logic depth in a single clock cycle (pipelining).
    • Recall: fix hold time violations by adding more logic! Sometimes, surprisingly, adding buffers (more logic) to short paths can resolve hold time violations by increasing the minimum delay.

Meeting Timing Constraints: Principles

• Fundamental Principle: Clock cycle time ($T_c$) is fundamentally limited by the maximum logic delay that the circuit can accommodate without violating timing constraints.
• Good design principles to improve timing and meet constraints:
  • Critical path design: Focus on minimizing the delay of the critical path, as this directly determines the maximum achievable performance (clock frequency).
  • Balanced design: Aim for a balanced design where maximum logic delays are evenly distributed across different parts of the system. This prevents bottlenecks and reduces wasted time in faster paths waiting for slower ones.
  • Bread and butter design: Prioritize optimization for common-case scenarios to maximize performance for typical operations. Ensure that less frequent, non-common cases do not negatively impact the overall design performance by creating timing bottlenecks.

Lecture Summary

• Timing in combinational circuits:
  • Covered propagation delay, contamination delay, and glitches.
• Timing in sequential circuits:
  • Discussed setup time, hold time, and how they determine the maximum operating frequency.
• Circuit Verification & Testing:
  • Explored methods to ensure a circuit works correctly, including functional verification and testing, and briefly touched upon timing verification and testing.

Continue here: 07 Von Neumann Model & Instruction Set Architectures