Focus Your Testing Resources for Higher Quality, On-Time Product Delivery

Overview

Run-Time Visibility into Devices Under Test
Wind River Test Management is a test execution optimization system for embedded devices. It enables unprecedented run-time visibility into devices while under test. The system provides the operational feedback that test teams and their management need to achieve the highest possible quality in the least amount of time.

Wind River Test Management is a scalable system that adds value to your existing test and development environments to enhance automation, control, traceability, and feedback. It leverages unique, dynamic instrumentation technology to measure test coverage, map test-to-code traceability, profile performance, enable white-box access, and speed diagnostics of complex devices—all at run-time:

• Get higher-quality devices to market faster, at less cost.
• Optimize testing efforts to save time and resources.
• Test more software more often.
• Dramatically accelerate defect resolution.
• Enhance your existing environment for better testing.

Device Test Execution Optimization
Wind River Test Management lets you optimize test execution while providing run-time visibility into device operation under test. It provides the information you need to improve testing, reduce test times, and find bottlenecks and defects before your customers do:

• Identify the portions of device software that are not tested.
• Determine automatically what tests need to be run when code changes.
• Identify performance regressions between builds or releases.
• Automate test of critical exception conditions.
• Isolate and repair run-time defects faster to avoid costly schedule delays.

Optimize Testing with Run-Time Analytics

A "perfect storm" of factors is challenging test teams today: more software to test, far more complex architectures being used, and much tighter production schedules. These factors are exacerbated when agile or iterative development methodologies are added to the mix. For most embedded companies, this combination of market and technology trends is outpacing the capabilities of their traditional testing systems and processes.

Wind River Test Management leverages operational feedback from devices under test to achieve the highest quality in the least amount of time

As the gap between testing requirements and testing capabilities grows, embedded companies are running increased risk of product failures occurring in the field. To avoid these costly, disruptive and damaging incidents, executives and team leaders need to recognize and address the problem.

The Wind River Test Management solution is designed to address these problems. It does so by enabling a fast and effective transition from "black-box" to "white-box" testing. Most testing today is limited to a black-box view of the device under test. Testers work from the "outside" and observe results through whatever pass/fail information, static logs, or error codes are available. With the white-box approach provided by Wind River Test Management, teams gain views into actual device operations when under test. They can gather critical information from running equipment, providing a clearer picture of test effectiveness, performance, defects, and more. The net result is testers can gather the key feedback they need from complex devices and store and manage this information in sync with test execution.

Find Untested Code

• Gather critical information about code that is not being tested with run-time test coverage.
• Dynamically deploy instrumentation to selected modules and functions to determine whether expected software is really executed under test.
• Validate the effectiveness of your test suites and focus development on filling the gaps.

Change-Driven, Optimized Testing

• Identify changes between software builds automatically (changed, new, and deleted code).
• Gain the capabilities needed by test teams to handle the demands of agile or iterative development environments—support rapid-fire build cycles and fast and frequent feedback loops.
• Generate a traceability map that shows the linkage between test cases and code at run-time.
• Leverage a traceability map to automatically determine which test cases are required to run when code has changed.
• Verify traceability requirements all the way to the code.

Identify Performance Regressions

• Profile software under test with low-impact probes.
• Identify bottlenecks or performance regressions between builds with graphical tools.
• Validate subsystem performance against requirements.

Validate Fault Handling and Exception Conditions

• Utilize automated white-box access to code and data to inject faults or force error states for testing.
• Quickly drive devices to desired state with direct access to software internals at run-time.
• Simulate I/O, automate manual input, or stub out interfaces.
• Save time and money while improving test coverage and quality.

Isolate, Repair, and Validate Run-Time Defects

• Deploy dynamic instrumentation to log data during test execution to isolate faults faster.
• Avoid time-consuming software rebuilds with incremental technology that lets users probe devices for forensic data without stopping them.
• Hot-patch proposed fixes into lab devices to validate repair before committing to code base.
• Dramatically reduce the time and effort required to find and repair defects.

Leverage Unique Dynamic Instrumentation
Wind River Test Management employs a unique dynamic instrumentation—sensorpoints—to probe, tag, access, or patch software running in the device. Unlike disruptive source-code-based instrumentation techniques, sensorpoints allow test and diagnostics instrumentation to be deployed in production software, not special test builds, giving you the best information about device operation. Dynamic instrumentation is deployed and removed on demand and on the fly without requiring the device to be stopped, reloaded, or rebooted.

Optimize Test Execution

Wind River Test Management includes an open text execution engine that can run tests from any source or synchronize the gathering of run-time analytics with external test harnesses.

Wind River Test Management change-driven testing leverages binary differencing and test-to-code traceability to help save time and better focus regression testing efforts in agile or iterative development environments

Import or Synchronize Tests

• Synchronize with externally stored tests or use as a repository.
• Import test cases into the system using a range of individual or batch import options.
• Use open interfaces to synchronize external test harnesses with the system's run-time analytics rather than moving tests from legacy systems.

Hybrid Test Cases

• Enable sophisticated device test sequences and facilitate migration from manual to automated testing.
• Utilize a hybrid test case type that can include both manual and scripted test steps.
• Let teams use automated scripts within manual test processes and execute them interactively along with the manual steps in a managed environment.

Component-Based Test Modularity

• Simplify test suites by organizing test scripts that are used repeatedly in multiple test cases.
• Compartmentalize tests along with related "setup" and "cleanup" sequences.
• Make it easy for test developers to build more self-contained tests with a modular structure and user-defined, reusable scripted test components.

Custom Fields

• Align with standard, in-house test templates through the addition of unique attributes.
• Create customized test case templates with the addition of user-definable fields to the base test case definition.
• Customize test execution status descriptions.

File Attachments for Data-Driven and GUI Testing

• Attach any type of file to a test case or to the individual test steps within an interactive test sequence, allowing development of sophisticated, data-driven tests for all types of devices.
• Move test datasets easily to external files to separate reusable or changing test data from the actual test scripts.
• Capture correct GUI screens easily to enable testers to compare current vs. expected results.
• Facilitate migration from manual to automated testing through easier handling of Microsoft Word or Excel file attachments.

Run Any Test

• Capture and execute manual or scripted tests, whether external or embedded.
• Use any scripting or programming language.
• Configure the system easily to manage and execute any type of test.

Assemble and Manage Test Suites

• Create purpose-built test suites, selecting any test cases from the repository into optimal groupings for test runs on reserved devices.
• Align test suites with specific software builds of the target application and validate that the correct software build is loaded on the assigned test device.
• Execute the scripted tests automatically, targeting the designated device.
• Walk testers through the sequence of instructions to execute manual test on a designated device and require input of pass/fail results and comments for each test case.
• Collect and store test results, stdio, coverage, performance, diagnostics, or other logs for each test run in a historical database.

Optimize Execution for Changed Code

• Leverage a unique test-to-code traceability map to auto-generate test suites for new builds.
• Identify any function that has changed between two binary software builds using the system's build comparison facility.
• Create lists of recommended test cases that need to be run based on identified code changes.
• Minimize the time and resources needed to verify new builds and eliminate guesswork.

Parallel, Distributed Operation

• Choose the test case sequence execution order to allow sequential or parallel execution.
• Distribute test execution over multiple engines covering different labs or devices.
• Use the system's scripted test case mechanism to manage and execute command scripts that initialize test equipment or simulators as part of a test sequence.

Manage Test Devices and Labs

Many companies have capital equipment spread across many test labs in different locations. That makes it hard for testers to know if devices with the correct configuration are available and difficult to provision them with the right software for testing.

Wind River Test Management includes a virtual lab manager (VLM) that provides tight integration of test target management with test execution and analytics, simplifying lab setup and management and providing a full life cycle test-device audit trail. The VLM saves time and hassle when working with test lab devices and allows maximum utilization of people and capital equipment resources.

Organize and Reserve Networked Devices

• Enable easy setup of virtual lab groupings of test equipment in local labs or across corporate networks.
• Facilitate fast and effective searches for needed equipment.
• Make it easy to reserve and check out testing equipment.
• Provide an effective tracking mechanism for equipment utilization.

Remotely Provision Builds

• Enable a "push-button" ability to track and load new software builds.
• Save time while ensuring correct test configurations.
• Leverage terminal servers or power controllers to remotely access and reboot devices.

Manage Multi-core Devices and Test Beds

• Control and run tests against multiple cores or devices simultaneously.
• Manage built-in run-time analytics to gather white-box data from the live device under test.
• Deploy test coverage analysis, performance measurement, fault injection, or dynamic diagnostics instrumentation across multiple cores or processors.

Tight Integration with Test Execution

• Avoid hard-coding IP addresses into test scripts.
• Assign test suites to available devices at run-time.
• Automatically map test execution to selected devices.
• Record and track the results in the system's database.

Mine and Manage Test Results

Wind River Test Management is built around a choice of scalable, relational databases that allow storage and management of all test results and analytics for mining, analysis, and archiving.

Unified Test Data Management

• Retain all test information and store it in an underlying database for easy mining.
• Store specific projects, builds, test cases, test suites, and device information with their associated test results.
• Capture any logs that are generated during the test process and store them along with pass/fail results.

Interactive Dashboards

• Provide easy and flexible access to key data via a web browser or the system's user interface and interactive graphical dashboards.
• Enable users to select specific test runs, modules, projects, etc.
• Offer instant access to current or cumulative data displays.

Test Log Parsing and Subresults Display

• Allow users to create parser scripts that post-process custom or third-party logs and expose summary data from the log file to the system.
• Display summary results in a new subresults area of the system's Test Run tab so that users can easily drill down to see subresults of a test run.
• Let users generate subresults as a batch operation at the end of a test run, or leverage a TMCLI command and user-created script to update the subresults periodically.
• Give users visibility of intermediate points of execution or results during complex, automated test runs.

Custom Reporting

• Use industry-standard JasperReports as the system's built-in reporting engine.
• Use standard reporting tools such as iReports to create custom reports from stored data.
• Enable defined report formats to be stored and run from the Reports tab via a web browser.
• Use new Build Change Summary report to show exactly how the code was changed between builds, helping managers to assess code churn and do change-based risk assessment in iterative or agile processes.
• Help managers to assess the thoroughness of their testing with reports such as Coverage Trends and Functions Not Covered.
• Automatically generate many popular report formats, including PDF, XLS, RTF, and HTML, which facilitates easy sharing and viewing of files.

Multiple Deployment Options

Wind River Test Management leverages open interfaces within your existing test environment. You can connect run-time analytics into your existing test systems, let Wind River Test Management control your legacy test beds, or migrate your test and execution environment into Wind River Test Management. Select the appropriate strategy for your projects based on life cycle stage, business needs, and state of existing test assets.

Open CLI
The system can perform automated or batch operations as commanded by external scripts or applications. For example, when a new build comes available, Wind River Test Management can be driven to set up and run regression test suites on the new build automatically. An open command-line interface (CLI) provides access to server commands, allowing integration with third-party applications or external scripts for automated control.

Web-Based Platform
Based on industry-standard J2EE application servers, a relational database, and downloadable device agents, Wind River Test Management provides a scalable, distributed architecture that brings together the development and testing players into a common environment. Users interact through Web-browser-based applications and command-line or Eclipse-based developer tools.

Wind River Sensorpoint Technology
Wind River's sensorpoint technology lets users probe or patch running devices without rebuilding the device software or stopping the device. Sensorpoints allow developers and testers to inject code into a running executable to gather data, force faults, add functionality, diagnose problems, gather run-time analytics information, or do white-box testing. Sensorpoints are minimally intrusive on device performance and footprint so they can be used in actual devices under test.

Broad Platform Support
Wind River Test Management was designed for embedded device system test and aims to support the widest range of device processor, operating system, host, and infrastructure software combinations with the product. This list is continuously expanding to cover Wind River Linux, Wind River VxWorks, and non–Wind River platforms. Contact your Wind River representative for details.