General information on Software QA & Testing
(Disclaimer: some of the general technology information on this page might have been shared with and from other third party sources)
Definition of 'Software Quality Assurance' (QA):
Software quality assurance (QA) involves the entire process of software development - including monitoring and improving the process, fully complying with the requirements, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. Software QA is oriented towards 'prevention'.
Definition of 'Software Testing':
Software testing involves operation of a system or application under controlled conditions and evaluating the results (e.g., 'if the user is in interface A of the application while using hardware B, and does C, then D should happen'). The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should. It is oriented towards 'detection'.
Organizations vary considerably in how they assign responsibility for QA and testing. Most of the time they are the combined responsibility of one group or individual. Also common are project teams that include a mix of testers and developers who work closely together, with overall QA processes monitored by project managers. It depends on what best fits an organization's size and business structure.
Possible reasons for management not serious about quality assurance:
Solving problems is a high-visibility process; preventing problems is low-visibility. Many organizations do not realize the danger of an unknown but possible problem. Since many software bugs and defects do not show up unless a sequence of events happen, management of many organizations is reluctant to spend resources on QA.
Reasons for software bugs and defects:
- Miscommunication or no communication - as to specifics of what an application should or shouldn't do (the application's requirements).
- Complexities of software - the complexity of current software applications can be difficult to comprehend for anyone without experience in modern-day software development. Windows-type interfaces, Web based interactive applications, client-server and distributed applications, data communications, enormous relational databases, and sheer size of applications have all contributed to the exponential growth in software/system complexity. And the use of object-oriented techniques can complicate instead of simplify a project unless it is well engineered.
- Programming errors - programmers, like anyone else, can make mistakes.
- Changing requirements (whether documented or undocumented) - the customer may not understand the effects of changes, or may understand and request them anyway - redesign, rescheduling of engineers, effects on other projects, work already completed that may have to be redone or thrown out, hardware requirements that may be affected, etc. If there are many minor changes or any major changes, known and unknown dependencies among parts of the project are likely to interact and cause problems, and the complexity of coordinating changes may result in errors. Enthusiasm of engineering staff may be affected. In some fast-changing business environments, continuously modified requirements may be a fact of life. In this case, management must understand the resulting risks, and QA and test engineers must adapt and plan for continuous extensive testing to keep the inevitable bugs from running out of control.
- Time pressures - scheduling of software projects is difficult at best, often requiring a lot of guesswork. When deadlines loom and the crunch comes, mistakes will be made.
- Poorly documented code - it's tough to maintain and modify code that is badly written or poorly documented; the result is bugs. In many organizations management provides no incentive for programmers to document their code or write clear, understandable, maintainable code. In fact, it's usually the opposite: they get points mostly for quickly turning out code, and there's job security if nobody else can understand it ('if it was hard to write, it should be hard to read').
- Software development tools - visual tools, class libraries, compilers, scripting tools, etc. often introduce their own bugs or are poorly documented, resulting in added bugs.
Verifications and validations:
Verification typically involves reviews and meetings to evaluate documents, plans, code, requirements, and specifications. This can be done with checklists, issues lists, walkthroughs, and inspection meetings. Validation typically involves actual testing and takes place after verifications are completed. The term 'IV & V' refers to Independent Verification and Validation.
Definition of 'walkthrough':
A 'walkthrough' is an informal meeting for evaluation or informational purposes. 'Walkthrough' hardly requires any preparations.
Definition of 'inspection':
An inspection is more formalized than a 'walkthrough', typically with a number of people including a moderator, reader, and a recorder to take notes. The subject of the inspection is typically a document such as a requirements spec or a test plan, and the purpose is to find problems and see what's missing, not to fix anything. Attendees should prepare for this type of meeting by reading thru the document; most problems will be found during this preparation. The result of the inspection meeting should be a written report. Thorough preparation for inspections is difficult, painstaking work, but is one of the most cost effective methods of ensuring quality. Employees who are most skilled at inspections, have very detailed knowledge about the total functionality and working of the system. Their skill may have low visibility but they are extremely valuable to any software development organization, since bug prevention is far more cost-effective than bug detection.
Different types of software testing:
- Black box testing - not based on any knowledge of internal design or code. Tests are based on requirements and functionality.
- White box testing - based on knowledge of the internal logic of an application's code. Tests are based on coverage of code statements, branches, paths, conditions.
- Unit testing - the most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses.
- Incremental integration testing - continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers.
- Integration testing - testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.
- Functional testing - black-box type testing geared to functional requirements of an application; this type of testing should be done by testers. This doesn't mean that the programmers shouldn't check that their code works before releasing it (which of course applies to any stage of testing.)
- System testing - black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.
- End-to-end testing - similar to system testing; the 'macro' end of the test scale; involves testing of a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate.
- Sanity testing or smoke testing - typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or corrupting databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state.
- Regression testing - re-testing after fixes or modifications of the software or its environment. It can be difficult to determine how much re-testing is needed, especially near the end of the development cycle. Automated testing tools can be especially useful for this type of testing.
- Acceptance testing - final testing based on specifications of the end-user or customer, or based on use by end-users/customers over some limited period of time.
- Load testing - testing an application under heavy loads, such as testing of a web site under a range of loads to determine at what point the system's response time degrades or fails.
- Stress testing - term often used interchangeably with 'load' and 'performance' testing. Also used to describe such tests as system functional testing while under unusually heavy loads, heavy repetition of certain actions or inputs, input of large numerical values, large complex queries to a database system, etc.
- Performance testing - term often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.
- Usability testing - testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.
- Install/uninstall testing - testing of full, partial, or upgrade install/uninstall processes.
- Recovery testing - testing how well a system recovers from crashes, hardware failures, or other catastrophic problems.
- Fail-over testing - typically used interchangeably with 'recovery testing'
- Security testing - testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques.
- Compatibility testing - testing how well software performs in a particular hardware/software/operating system/network/etc. environment.
- Exploratory testing - often taken to mean a creative, informal software test that is not based on formal test plans or test cases; testers may be learning the software as they test it.
- Ad-hoc testing - similar to exploratory testing, but often taken to mean that the testers have significant understanding of the software before testing it.
- Context-driven testing - testing driven by an understanding of the environment, culture, and intended use of software. For example, the testing approach for life-critical medical equipment software would be completely different than that for a low-cost computer game.
- User acceptance testing - determining if software is satisfactory to an end-user or customer.
- Comparison testing - comparing software weaknesses and strengths to competing products.
- Alpha testing - testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.
- Beta testing - testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.
- Mutation testing - a method for determining if a set of test data or test cases is useful, by deliberately introducing various code changes ('bugs') and retesting with the original test data/cases to determine if the 'bugs' are detected. Proper implementation requires large computational resources.
Common problems in the software development process:
- Poor requirements - if requirements are unclear, incomplete, too general, and not testable, there will be problems.
- Unrealistic schedule - if too much work is crammed in too little time, problems are inevitable.
- Inadequate testing - no one will know whether or not the program is any good until the customer complains or systems crash.
- Feature overloading - requests to pile on new features after development is underway; extremely common.
- Miscommunication - if developers don't know what's needed or customer's have erroneous expectations, problems are guaranteed.
Possible common solutions to software development problems:
- Detailed requirements - clear, complete, detailed, cohesive, attainable, testable requirements that are agreed to by all players. Use prototypes to help nail down requirements. In 'agile'-type environments, continuous coordination with customers/end-users is necessary.
- Realistic schedules - allow adequate time for planning, design, testing, bug fixing, re-testing, changes, and documentation; personnel should be able to complete the project without burning out.
- Adequate testing - start testing early on, re-test after fixes or changes, plan for adequate time for testing and bug-fixing. 'Early' testing ideally includes unit testing by developers and built-in testing and diagnostic capabilities.
- Stick to initial requirements as much as possible - be prepared to defend against excessive changes and additions once development has begun, and be prepared to explain consequences. If changes are necessary, they should be adequately reflected in related schedule changes. If possible, work closely with customers/end-users to manage expectations. This will provide them a higher comfort level with their requirements decisions and minimize excessive changes later on.
- Communication - require walkthroughs and inspections when appropriate; make extensive use of group communication tools - e-mail, groupware, networked bug-tracking tools and change management tools, intranet capabilities, etc.; insure that information/documentation is available and up-to-date - preferably electronic, not paper; promote teamwork and cooperation; use proto-types if possible to clarify customers' expectations.
Meaning of software 'quality':
Quality software is reasonably bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable. However, quality is obviously a subjective term. It will depend on who the 'customer' is and their overall influence in the scheme of things. A wide-angle view of the 'customers' of a software development project might include end-users, customer acceptance testers, customer contract officers, customer management, the development organization's management /accountants /testers /salespeople, future software maintenance engineers, stockholders, magazine columnists, etc. Each type of 'customer' will have their own slant on 'quality' - the accounting department might define quality in terms of profits while an end-user might define quality as user-friendly and bug-free.
Meaning of 'good software code':
'Good software code' is code that works, is bug free, and is readable and maintainable. Some organizations have coding 'standards' that all developers are supposed to adhere to, but everyone has different ideas about what's best, or what is too many or too few rules. There are also various theories and metrics, such as McCabe Complexity metrics. It should be kept in mind that excessive use of standards and rules can stifle productivity and creativity. 'Peer reviews', 'buddy checks' code analysis tools, etc. can be used to check for problems and enforce standards.
For C and C++ coding, here are some typical ideas to consider in setting rules/standards; these may or may not apply to a particular situation:
- Minimize or eliminate use of global variables.
- Use descriptive function and method names - use both upper and lower case, avoid abbreviations, use as many characters as necessary to be adequately descriptive (use of more than 20 characters is not out of line); be consistent in naming conventions.
- Use descriptive variable names - use both upper and lower case, avoid abbreviations, use as many characters as necessary to be adequately descriptive (use of more than 20 characters is not out of line); be consistent in naming conventions.
- Function and method sizes should be minimized; less than 100 lines of code is good, less than 50 lines is preferable.
- Function descriptions should be clearly spelled out in comments preceding a function's code.
- Organize code for readability.
- Use white space generously - vertically and horizontally.
- One code statement per line.
- Coding style should be consistent through a program (e.g., use of brackets, indentations, naming conventions, etc.)
- In adding comments, err on the side of too many rather than too few comments; a common rule of thumb is that there should be at least as many lines of comments (including header blocks) as lines of code.
- No matter how small, an application should include documentation of the overall program function and flow (even a few paragraphs is better than nothing); or if possible a separate flow chart and detailed program documentation.
- Make extensive use of error handling procedures and status and error logging.
- For C++, to minimize complexity and increase maintainability, avoid too many levels of inheritance in class hierarchies (relative to the size and complexity of the application). Minimize use of multiple inheritances, and minimize use of operator overloading (note that the Java programming language eliminates multiple inheritance and operator overloading.)
- For C++, keep class methods small, less than 50 lines of code per method are preferable.
- For C++, make liberal use of exception handlers.
Meaning of 'good design':
'Design' could refer to many things, but often refers to 'functional design' or 'internal design'. Good internal design is indicated by software code whose overall structure is clear, understandable, easily modifiable, and maintainable; is robust with sufficient error handling and status logging capability; and works correctly when implemented. Good functional design is indicated by an application whose functionality can be traced back to customer and end-user requirements. For programs that have a user interface, it's often a good idea to assume that the end user will have little computer knowledge and may not read a user manual or even the on-line help; some common rules-of-thumb include:
- The program should act in a way that least surprises the user.
- It should always be evident to the user what can be done next and how to exit.
- The program shouldn't let the users do something stupid without warning them.
Definitions of SEI, CMM, CMMI, ISO, IEEE, ANSI:
- SEI = 'Software Engineering Institute' at Carnegie-Mellon University; initiated by the U.S. Defense Department to help improve software development processes.
- CMM = 'Capability Maturity Model', now called the CMMI ('Capability Maturity Model Integration'), developed by the SEI. It's a model of 5 levels of process 'maturity' that determine effectiveness in delivering quality software. It is geared to large organizations such as large U.S. Defense Department contractors. However, many of the QA processes involved are appropriate to any organization, and if reasonably applied can be helpful. Organizations can receive CMMI ratings by undergoing assessments by qualified auditors.
- Level 1 - characterized by chaos, periodic panics, and heroic efforts required by individuals to successfully complete projects. Few if any processes in place; successes may not be repeatable.
- Level 2 - software project tracking, requirements management, realistic planning, and configuration management processes are in place; successful practices can be repeated.
- Level 3 - standard software development and maintenance processes are integrated throughout an organization; a Software Engineering Process Group is in place to oversee software processes, and training programs are used to ensure understanding and compliance.
- Level 4 - metrics are used to track productivity, processes, and products. Project performance is predictable, and quality is consistently high.
- Level 5 - the focus is on continuous process improvement. The impact of new processes and technologies can be predicted and effectively implemented when required.
Perspective on CMM ratings: During 1997-2001, 1018 organizations were assessed. Of those, 27% were rated at Level 1, 39% at 2, 23% at 3, 6% at 4, and 5% at 5. (For ratings during the period 1992-96, 62% were at Level 1, 23% at 2, 13% at 3, 2% at 4, and 0.4% at 5.) The median size of organizations was 100 software engineering/maintenance personnel; 32% of organizations were U.S. federal contractors or agencies. For those rated at Level 1, the most problematical key process area was in Software Quality Assurance.
- ISO = 'International Organization for Standardization' - The ISO 9001:2000 standard (which replaces the previous standard of 1994) concerns quality systems that are assessed by outside auditors, and it applies to many kinds of production and manufacturing organizations, not just software. It covers documentation, design, development, production, testing, installation, servicing, and other processes. The full set of standards consists of: (a) Q9001-2000 - Quality Management Systems: Requirements; (b) Q9000-2000 - Quality Management Systems: Fundamentals and Vocabulary; (c) Q9004-2000 - Quality Management Systems: Guidelines for Performance Improvements. To be ISO 9001 certified, a third-party auditor assesses an organization, and certification is typically good for about 3 years, after which a complete reassessment is required. Note that ISO certification does not necessarily indicate quality products - it indicates only that documented processes are followed.
- IEEE = 'Institute of Electrical and Electronics Engineers' - among other things, creates standards such as 'IEEE Standard for Software Test Documentation' (IEEE/ANSI Standard 829), 'IEEE Standard of Software Unit Testing (IEEE/ANSI Standard 1008), 'IEEE Standard for Software Quality Assurance Plans' (IEEE/ANSI Standard 730), and others.
- ANSI = 'American National Standards Institute', the primary industrial standards body in the U.S.; publishes some software-related standards in conjunction with the IEEE and ASQ (American Society for Quality).
- Other software development/IT management process assessment methods besides CMMI and ISO 9000 include SPICE, Trillium, TickIT, Bootstrap, ITIL, MOF, and CobiT.
Meaning of 'Software life cycle':
The life cycle begins when an application is first conceived and ends when it is no longer in use. It includes aspects such as initial concept, requirements analysis, functional design, internal design, documentation planning, test planning, coding, document preparation, integration, testing, maintenance, updates, retesting, phase-out, and other aspects.
Possible impacts of automated testing:
- Possibly automated software testing can make life easier. For small projects, the time needed to learn and implement them may not be worth it. For larger projects, or on-going long-term projects they can be valuable.
- A common type of automated tool is the 'record/playback' type. For example, a tester could click through all combinations of menu choices, dialog box choices, buttons, etc. in an application GUI and have them 'recorded' and the results logged by a tool. The 'recording' is typically in the form of text based on a scripting language that is interpretable by the testing tool. If new buttons are added, or some underlying code in the application is changed, etc. the application might then be retested by just 'playing back' the 'recorded' actions, and comparing the logging results to check effects of the changes. The problem with such tools is that if there are continual changes to the system being tested, the 'recordings' may have to be changed so much that it becomes very time-consuming to continuously update the scripts. Additionally, interpretation and analysis of results (screens, data, logs, etc.) can be a difficult task. Note that there are record/playback tools for text-based interfaces also, and for all types of platforms.
- Other automated tools can include:
- Code analyzers - monitor code complexity, adherence to standards, etc.
- Coverage analyzers - these tools check which parts of the code have been exercised by a test, and may be oriented to code statement coverage, condition coverage, path coverage, etc.
- Memory analyzers - such as bounds-checkers and leak detectors.
- Load/performance test tools - for testing client/server and web applications under various load levels.
- Web application test tools - to check that links are valid, HTML code usage is correct, client-side and server-side programs work, a web site's interactions are secure.
- Other tools - for test case management, documentation management, bug reporting, and configuration management.
Specific product information on some automated software testing tools and suites:
G&G is doing automated software testing using tools from Segue Software, IBM Rational, Compuware, Mercury, Worksoft, Empirix, Keynote, McCabe Associates and others. Among automated software tools, our software QA team has experience in or familiarity with Segue's SilkCentral Test Manager, SilkCentral Issue Manager, SilkCentral Performance Manager, SilkPerformer, SilkPerformer Component Test Edition, SilkTest, IBM Rational Functional Tester for Java and Web, Performance Tester, IBM Rational Functional Tester Extension for Terminal-based Applications, IBM Rational Robot, IBM Rational Test RealTime, Compuware's QARun, QALoad, TestPartner, QADirector, TrackRecord, Reconcile, Mercury LoadRunner, Mercury Tuning (formerly ProTune), Mercury Diagnostics & Monitoring, Mercury TestDirector, Mercury Functional Testing, Mercury Business Process Testing, Application Delivery Dashboard, Worksoft Certify, Empirix Web Testing solutions (e-TEST suite for Web Services, e-TEST suite for .NET testing, e-TEST™ suite integrated, comprehensive Web application testing solution, e-Load load and stress testing, e-Manager Enterprise test management, and Voice over IP [VoIP] testing), Keynote's LoadPro, PerformanceTune, WebIntegrity, LoadSuite, and TestPerspective, McCabe's IQ (Integrated Quality) tools.
SilkCentral Test Manager
Automate your testing process for optimal quality and productivity
SilkCentral™ Test Manager is an all-inclusive test management system that builds quality and productivity into the testing process to speed the delivery of successful enterprise applications. It lets you plan, document and manage each step of the testing cycle from capturing and organizing key business requirements, tracing them through execution … designing the optimal test plans … scheduling tests for unattended execution … tracking the progress of manual and automated tests … identifying the features at risk … and assessing when the application is ready to go live.
- Simplify test management via Web-based scheduling, execution, reporting and defect tracking.
- Speed the deployment of complex, distributed applications.
- Enhance decision-making by increasing visibility into the testing process.
- Allocate resources according to feature, priority, business risk or other defined process.
- Reduce cost by finding and fixing problems early in the testing process.
- Minimize risk by defining exit criteria.
- Deploy high quality applications that meet key business requirements.
- Increase productivity and optimize quality via integration with Segue's automated testing tools.
- Integrate with Segue's SilkCentral Issue Manager tool for automated issue tracking.
SilkCentral Issue Manager
Resolve issues quickly & reliably by automating the tracking process
An estimated 80% of all software costs is spent on resolving application defects. With SilkCentral™ Issue Manager (formerly SilkRadar), you can reduce the cost and speed the resolution of defects and other issues throughout the entire application lifecycle. SilkCentral Issue Manager features a flexible, action-driven workflow that adapts easily to your current business processes and optimizes defect tracking by automatically advancing each issue to its next stage. Its Web user interface provides 24x7x365 access to a central repository of all defect-related information - simplifying usage among geographically dispersed groups and promoting collaboration among different departments. Meanwhile insightful reports enable you to determine project readiness based on the status of important issues.
- Reduce the cycle time for bug fixes and enhancements by automatically advancing issues toward resolution
- Know the status of each issue, its next steps, and who's responsible for its resolution
- Track defects and enhancement requests from any location at any time
- Ensure high quality software projects that meet users expectations
- Minimize operation costs by automating your current tracking process with an easily adaptable workflow engine
- Enhance cross-departmental communication via Web-based access to all issue-related information
- Integrate your issue tracking process with Segue's test management, functional testing and load testing tools
SilkCentral Performance Manager
Maximize the availability, accuracy & performance of live applications
SilkCentral™ Performance Manager (formerly SilkVision) is a production testing system that evaluates the health of live applications based on three critical end-user metrics - availability, accuracy and performance. In addition to capturing infrastructure metrics, SilkCentral Performance Manager monitors end-user transactions using either live or simulated traffic, while simultaneously testing the functional integrity of those transactions. By correlating both end-user and infrastructure metrics, SilkCentral Performance Manager lets you accurately identify the root cause of errors, regardless of whether they reside in the application, network or back-end systems. And by providing historical data for trend analysis, it enables you to anticipate and avoid errors, reduce downtime and optimize system resources.
- Get a bird's eye view of system health on a 24x7x365 basis
- Validate Service Level Agreements
- View an accurate picture of the end-user experience
- Gain clear visibility into the root cause of performance problems
- Diagnose errors by proactively monitoring system behavior
- Maximize uptime by predicting and avoiding errors
- Gain anytime/anywhere access to your information repository
SilkPerformerMaximize the load and performance testing of enterprise applications
SilkPerformer is the industry's most powerful - yet easiest to use - automated load and performance testing system for maximizing the performance, scalability and reliability of enterprise applications. With SilkPerformer, you can accurately predict the "breaking points" in your application and its infrastructure before it is deployed, regardless of its size or complexity. SilkPerformer has the power to simulate thousands of simultaneous users working with multiple computing environments and interacting with various application environments such as Web, client/server, or ERP/CRM systems - all with a single script and one or more test machines. Yet its visual approach to scripting and root-cause analysis makes it amazingly simple and efficient to use. So you can create realistic load tests easily, find and fix bottlenecks quickly, and deliver high-performance applications faster than ever.
- Optimize performance and quality by gaining clear insight into your application
- Reduce testing costs by simulating thousands of simultaneous users from a single location using standard hardware resources
- Improve productivity with visual scripting, a built-in workflow and integrated wizards
- Delegate load testing to non-programmers by capitalizing on SilkPerformer's ease-of-use
- Shorten test cycles by finding the root cause of bottlenecks quickly and easily
- Minimize costs by accurately predicting system capacity
- Test your application against service levels to validate its ability to meet SLAs in production
- Reduce the cost of defects by testing early in the development cycle
- Reuse test scripts to monitor production systems from the end-user perspective
SilkPerformer Component Test Edition
Reduce costs by testing remote components early in development
As the central building blocks of a distributed application, remote software components are key to ensuring application quality. These components include middleware such as:
- Web services
- Enterprise Java Beans
- Java RMI Objects
- .NET Serviced Components
- .NET Remoting Objects
SilkPerformer Component Test Edition lets you test the functionality, performance and interoperability of these critical remote components early in the application lifecycle - even before client applications are available. Unlike unit testing tools, SilkPerformer Component Test Edition tests these components under realistic server conditions by exposing them to concurrent user access. What's more, its visual scripting technology is quick and easy to use. As a result, even non-programmers can detect potential problems that occur only under real-world conditions … resolve them early in development before they become costly realities … and release high quality applications on time.
- Enhance application quality by testing components under realistic server conditions
- Avoid costly redevelopment and speed development by finding and fixing defects early in the development cycle
- Eliminate the risk of deadlock situations and other functional errors that occur only under concurrent access
- Maximize the performance of remote components
- Ensure that all remote components work with all clients
- Offload developers by delegating component testing to non-programmers
- Improve communication between QA and development
- Leverage your investment by migrating to SilkPerformer for enterprise-class load testing
Build high performance into Web applications cost effectively
SilkPerformer Lite is the ideal solution for small and mid-size organizations seeking an affordable yet fully functional solution for testing the performance, scalability and reliability of Web applications before they go live. SilkPerformer Lite enables you to predict the "breaking points" in your Web application, as well as in its supporting infrastructure, with the utmost accuracy. It's simple to use, too. With just a single standard computer, intelligent visual scripting and built-in workflow, you can quickly and easily create a load test that simulates the realistic behavior of up to 100 simultaneous users under dynamic load conditions. Visual root-cause analysis tools and management reports help you find bottlenecks easily, fix them quickly and launch high-performance Web applications on time.
- Optimize performance and quality by gaining clear insight into your Web application
- Reduce testing costs by simulating up to 100 concurrent users from just a single test machine
- Improve productivity with visual scripting, a built-in workflow and integrated wizards
- Delegate load testing to non-programmers by capitalizing on SilkPerformer Lite's ease-of-use
- Shorten test cycles by finding the root cause of bottlenecks quickly and easily
- Minimize costs by accurately predicting system capacity
- Test your application against service levels to validate its ability to meet SLAs in production
- Ensure a smooth upgrade path to SilkPerformer for enterprise-class load testing
Meet the time-to-market & quality goals of enterprise applications
SilkTest is the industry-leading automated tool for testing the functionality of enterprise applications in any environment. It lets you thoroughly verify application reliability within the confines of today's short testing cycles by leveraging the accuracy, consistency and time-saving benefits of Segue's automated testing technology. Designed for ease of use, SilkTest includes a host of productivity-boosting features that let both novice and expert users create functional tests quickly, execute them automatically and analyze results accurately. With less time spent testing, your QA staff can expand test coverage and optimize application quality. In addition to validating the full functionality of an application prior to its initial release, users can easily evaluate the impact of new enhancements on existing functionality by simply reusing existing test cases.
- Accelerate time-to-market by automating the functional testing process
- Maximize application reliability by increasing test coverage and consistency
- Improve productivity with workflow bars and wizards for creating and customizing tests quickly and easily
- Avoid costly redevelopment by finding and fixing bugs early
- Minimize the risk of undetected errors by running tests unattended 24x7x365
- Shorten the testing cycle by distributing tests over multiple computers
- Optimize application quality through accurate results analysis
SilkTest International (Ensure the reliability of multi-lingual enterprise applications)
When it comes to localized versions of global applications, companies traditionally resort to second-class manual testing - a time-consuming and costly process which leaves a large margin of error. SilkTest International changes all that by providing a quick, accurate and fully automated way to test localized applications.
Test smarter, finish faster. Improve testing scope and depth while shortening test time with Rational testing products, online tester resources and services.
Ease the team into test automation by first using IBM Rational® TestManager to manage manual testing and reporting, and IBM Rational® Robot to automate regression tests. Take advantage of integrated defect tracking, test management, and requirements traceability in the IBM Rational® Team Unifying Platform to uncover more defects, earlier, when they are least expensive to fix. And improve testers' automation skills on-the-job via the Rational Developers Network, Rational University web-based courses, and the browser-based Rational Unified Process.
Rational offers the following products for automated system testing:
- IBM Rational Functional Tester for Java and Web: Virtually eliminates script maintenance by creating resilient, reusable test scripts - in Java - with ScriptAssure™.
- IBM Rational Functional Tester Extension for Terminal-based Applications: Extends IBM Rational Functional Tester for Java and Web to test 3270 (zSeries™) and 5250 (iSeries™) terminal-based applications.
- IBM Rational Robot: Automates functional, regression and configuration testing for a wide range of application types, including .NET.
- IBM Rational Performance Tester: Uncovers and corrects production-environment performance problems in the test lab, before deployment.
- IBM Rational Team Unifying Platform: Integrates all the testing activities for one application with centralized test management, defect tracking, and version control.
Rational offers the following products for developer testing:
- IBM Rational PurifyPlus: A complete set of runtime analysis tools designed for improving application reliability and performance.
- IBM Rational Rose XDE Developer Plus: Offers software designers and developers a rich set of model-driven development and runtime analysis capabilities for building quality software applications. It offers complete visual design and development environments that address the needs of organizations targeting both J2EE-based and .NET-based systems.
- IBM Rational Test RealTime: A cross-platform solution for component testing and runtime analysis, it was designed specifically for those who write code for embedded, real-time, and other types of cross-platform software products.
QACenter for Enterprise Servers
QACenter for WebSphere MQ
1) Automated functional and regression testing (QARun/TestPartner)
2) Test management (QADirector)
3) Requirements management (Reconcile)
4) Defect & project tracking (TrackRecord)
5) Test data management (File-AID/CS)
6) QALoad (load testing)
Note: QALoad allows you to easily measure the application's scalability and performance under unlimited traffic. You can predict an application's response before you go live, giving you confidence to release your application into production.
Get the advantage in scalability testing, using QACenter Performance Edition to:
- emulate, from a single point, any number of users and their impact on your system
- perform large-scale tests with minimal hardware resources
- adjust volume levels through dial-up/dial-down virtual users
- create repeatable test scripts through an easy-to-use visual interface
- find and correct scalability problems early in the development process
- correlate poor response times with virtual user levels
MercuryMercury Performance Center
- Reduce system downtime
- Plan, validate, and optimize capacity and performance
- Meet service-level agreements (SLAs) at the lowest possible cost
Mercury Performance Center provides the first lifecycle approach to optimizing application performance. It's the only solution that scales from project-based performance testing to a 24x7 globally accessible solution for Performance Centers of Excellence. It enables you to manage the risk of deploying mission-critical applications by telling you exactly what to expect once your application is in production. You can pinpoint bottlenecks ahead of time to help increase application performance. It also enables you to improve your infrastructure performance and ensure standardized best practices across your organization.
Mercury Performance Center Offerings
- Application Delivery Dashboard
Gain a real-time business view of key performance indicators across all quality and performance projects.
- Mercury LoadRunner
The industry-standard performance testing product for predicting system behavior and performance.
- Mercury Tuning (formerly ProTune)
Identify, isolate and resolve infrastructure bottlenecks.
- Mercury Diagnostics & Monitoring
Isolate application performance problems and reduces the mean time to resolution (MTTR) of performance bottlenecks.
- Mercury Capacity Planning
Provides simulation modeling of real-world production environments to optimize the cost-to-risk ratio in IT investment decisions.
- Mercury Global Management (formerly LoadRunner TestCenter)
The only enterprise-class performance testing solution designed for Performance Centers of Excellence.
- Application Delivery Foundation
Emulate your production environment while still in pre-production; continuously monitor quality applications; efficiently share data across IT teams.
- Mercury Performance Center Services
Achieve efficient deployment of Mercury Performance Center and fastest route to successful implementation.
Mercury Quality Center
Mercury Quality Center™ provides a complete, Web-based, integrated system for performing quality assurance across a wide range of IT and application environments. It includes an integrated suite of role-based applications and best-practices as well as an open, scalable, and extensible foundation - all designed to optimize and automate key quality activities, including requirements, test, and defects management; functional testing; and business-process testing.
Mercury Quality Center Offerings
- Mercury TestDirector
Provides a consistent, repeatable process and support for gathering requirements, planning and scheduling tests, analyzing results, and managing defects.
- Mercury Functional Testing
Mercury QuickTest Professional - Capture, verify, and replay user interactions automatically to identify defects. Mercury WinRunner - Reduce testing time by automating repetitive tasks and optimizing testing efforts.
- Mercury Business Process Testing
Provide automated functional testing capabilities to business analysts so they can be part of the quality process.
- Application Delivery Dashboard
Real-time business view of key performance indicators across all quality and performance projects.
- Mercury Quality Center Services
Achieve efficient deployment of Mercury Quality Center and fastest route to successful implementation.
WorksoftWhat is Certify ?
Certify is a next generation automated software testing solution that is easier to learn, faster to deploy and more efficient to maintain than traditional software testing tools.
Certify requires no programming. No programming means no recording, testing, debugging and easier maintenance of test cases. Without the need to write a programming language business analysts are now able to test software applications along with QA.
With only a few easy steps, just point and click to document and automate test cases - you can significantly reduce the time and cost of automation by 60% or more and increase ROI .
Certify is the future of test automation. In the past, automation required that test cases be documented by business and quality analysts, then translated into a specialized scripting language by programmers.
Certify changes all that. There are no scripts. Now analysts can point and click to document and automate their test cases in one easy step, reducing the time and cost of automation by 60% or more. No scripts means no recording, programming, testing, debugging and - especially - maintenance of scripts. Certify allows you to focus on coverage instead of coding.
Certify's open architecture also provides a cross-platform solution that delivers seamless end-to-end testing across multiple systems, providing a uniform user interface regardless of the underlying technology. You are free to select the best of breed tools for each project without compromising consistency.
And Certify is more than an automation solution. Its powerful, proven methodology optimizes your test process to reduce risk effectively and efficiently. Its extensive reporting capabilities enable management to track progress and measure results throughout the complete test life cycle. Certify delivers tangible, rapid results.
Certify. It's about time. Certify is the right choice for these economic times, when you are forced to do more with fewer resources and less time to market.
Organize is where you create your Certify user access rights, identify the applications and versions under test, and organize the platforms and environments you will be using. Certify allows multiple versions of a single application to be active at any time, and it allows multiple applications to work together - even across different platforms.
Plan allows your team leader to organize the certification plan for each application by specifying the processes, cycles and schedules. Each phase is documented and can be assigned to team members for development. This module also provides you with flexible ways to trace, sort and report any criteria you choose. Special report options produce attractive, analyst-friendly documentation, or permit you to export to any format you choose.
Define is where you manage your application map and data. The application map is an inventory of your software's interface, including all screens, windows or pages and the fields or objects within them. This map not only guides development, it manages changes and their impact on your repository. This module also allows analysts to create or import the data values that supply the inputs or verify the outputs that are key to certifying results.
Perform provides application analysts with a powerful editor to detail each process into individual steps through a series of simple drop-down lists. These steps are self-documenting, maintainable and automatically executable. Manual tests can also be developed that prompt the user and capture results. Results from each and every step - automated or manual - are logged precisely and provide a complete audit trail.
Analyze gives management a tool to collect and analyze information about the certification process, providing a top-down view of what has been accomplished, where things stand and your company's return on its automation investment. Repository metrics measure the investment in certification assets, and metrics execution show the productivity received.
Certify Features & Benefits List
|Tool independent||Supports wide variety of technologies|
|Platform independent||Supports end to end testing|
|Standard interface across platforms||Saves training and support costs|
|Native language support||Supports API, COM, SQL|
|Capture requirements inventory||Can provide import from external file|
|Link requirements to tests||Can provide export to external file|
|Manage dependencies||Prevent domino failures|
|Allocate work effort among team||Supports project coordination|
|Integrated state management||Supports navigation, error recovery|
|Unlimited user defined keywords||Customize sorting and reporting|
|No programming required||Reduced learning curve|
|Pre-written class libraries||Faster implementation|
|Supported execution engine||Reduced cost of ownership|
|Structured approach||Self-documenting, transferable|
|Linked to application map||Automated maintenance|
|Support for manual tests||Documented steps and results|
|Manual steps within automated tests||Integrate external activities|
|Read from data recordsets||Expand coverage|
|Write to data recordsets||Extract data values|
|Read values from screen||Deal with dynamic data|
|Prompt user for data values||Allow user defined options|
|Automated generation of data layout||Rapid implementation of external files|
|Integrated Excel editor||Powerful and easy data editing|
|Import support for text files||Receive data from multiple sources|
|No programming required for loop, EOF||Easy to implement|
|Multiple data levels||Can loop at any level, do nested loops|
|Recordsets stored in database||Easy to manage, maintain|
|Variable recordset names||Easy to vary data sources, values|
|Single Step||Ease of diagnosis|
|Debug||Focus on specific areas|
|Future date and time||Schedule for off-hours|
|Capture screens||Document results, diagnose errors|
|Automated context management||Load, login to application|
|Integrated context verification||Avoid runaway scripts, timing waits|
|Hotkey control||Interrupt execution, resume at will|
|Detailed watch window||Monitor execution|
|Robust step-level results log||Actual and expected results|
|Detailed error log||Actual, expected, screen capture|
|Incident reporting||System and user supplied data|
|Elapsed time||Monitor performance variances|
|Repository metrics||Measure investment in tests|
|Execution metrics||Measure productivity|
|Map coverage||Measure application coverage|
|Status analysis||Track release progress|
|Documentation format||Human-readable narratives|
|Supports multiple applications, versions||Version control|
|Automated import of application map||Read source code or learn screen|
|Automated change impact analysis||Discover changes before execution|
|Automated test maintenance||Implement changes rapidly|
Web Application Performance:
Empirix provides a broad range of Web application performance management solutions designed to ensure the Quality of Experience delivered by your Web applications. Our easy-to-use Web testing and Web monitoring products and hosted services will benefit your development, QA and operations teams, and provide a rapid return on investment.
Empirix integrated Web application testing products and hosted services support our "test early, test often" philosophy that begins with testing at the software component level and continues through system integration to deployment. In this way defects can be found early in the development cycle where the cost to fix them is lowest.
Web Testing Solutions
Web testing solutions from Empirix help organizations verify the functionality, scalability and quality of their mission-critical, Web-based applications.
Web applications are expected to have dial tone-like reliability, despite disparate underlying technologies, unpredictable user volumes, and intense time-to-market pressures. Our Web testing solutions empower organizations to meet these challenges effectively and to deploy web applications with confidence.
INTEGRATED WEB APPLICATION AND COMPONENT TESTING
e-TEST™ suite integrated, comprehensive Web application testing solution that provides all the web testing tools you need to ensure the performance of your business-critical applications. The suite includes e-Tester™ for functional/regression testing, e-Load® for load/scalability testing, and e-Manager™ Enterprise for test process management.
e-TEST suite for Web Services is the leading solution for functional and performance testing of Web services.
e-TEST suite for .NET testing is the leading solution for functional and performance testing of Web services.
LOAD / STRESS TESTING
e-Load load and stress testing gives you the fastest and most accurate way to test the scalability of your Web applications. Companies use it regularly to accurately assess and enhance the load capacity of their applications.
e-LoadExpert hosted remote load and stress testing service enables you to accurately assess the scalability of your Web site without taxing your internal resources. During a six-day e-LoadExpert consultation, we'll identify traffic bottlenecks and empower you with the information you need to resolve problems that limit your site scalability.
TEST MANAGEMENT / DEFECT TRACKING
e-Manager Enterprise test management is an easy-to-use, comprehensive solution that allows users to organize, document, and manage the entire Web testing process.
FUNCTIONAL / REGRESSION TESTING
e-Tester functional and regression testing for Web applications. Our customers use e-Tester for web testing after every new build of their Web applications to ensure that they function as intended.
Empirix Web application performance monitoring solutions track Web application performance from inside and outside the firewall. Combining measurements of user-experience and server/network activity, our Web monitoring solutions help you detect, respond to, and diagnose performance problems before they affect your customers.
Web Application Performance Management:
Web application performance management (WAPM) solutions from Empirix let you :
- Improve Web application availability and performance
- Enforce SLAs (service level agreements)
- Ensure a positive Quality of Experience for the end-user
- Reduce production costs through environment optimization and intelligent capacity planning
Empirix OneSight™ is a browser-based product that monitors online transactions and correlates them with back-end system performance to identify and fix problems before they jeopardize your customers' Quality of Experience.
Empirix FarSightSM is a hosted service that monitors your Web application in real-time from points on a global network. The service offers both URL and multi-step transaction monitoring so that you can find and fix problems quickly.
VoIP and Network Test:
Empirix, the worldwide leader in Voice over IP (VoIP) testing, provides a broad range of Hammer testing and monitoring solutions designed to ensure the quality and performance of VoIP and IP Storage applications throughout their life cycle. Our easy-to-use products will benefit your development teams, and provide rapid return on investment.
Load and Feature Test
Empirix is the leader in emulating real live call traffic over VoIP, TDM and Converged networks and assessing network performance.
Analysis and Monitoring
Complementing Empirix's call generation products are the following tools for diagnosing problems in VoIP and storage network gear before it gets deployed
How effective is your e-business?
To run a successful e-business, you need to maximize every part of your infrastructure and tune it for consistent performance. Keynote provides a continuum of testing and tuning services from content integrity to capacity planning to help you meet both your technical and business requirements.
All of Keynote's Web performance testing services are deployed in flexible ways to suit your needs. Some testing services (such as LoadPro) are offered as full-service engagements, in which case Keynote dispatches a team of professional service consultants who provide methodology, test expertise and actionable results for you. Other services such as Test Perspective® and WebIntegrity™ are self-service, giving you greater control over test scheduling and configuration. Whatever your testing needs are, we can help ensure that your site delivers a high-quality user experience and the ROI you expect.
Current offerings include:
An outsourced tuning service that combines Keynote's load testing infrastructure with detailed back-end monitoring to isolate and repair performance bottlenecks.
LoadPro is an award-winning outsourced load testing service that allows you to accurately and dynamically test your Web applications and quantify the revenue impact of performance problems.
Rent Keynote's worldwide load-generating infrastructure to run realistic load and stress tests against your Web applications and understand how much traffic they can handle.
A load-testing offering that combines both our LoadPro and Test Perspective services to help you meet all your ongoing Web application testing needs.
Streamline Web site maintenance and enforce site design, privacy, and accessibility standards with this hosted service from Keynote.
You're Only as Strong as Your Weakest Line of Code When developing mission, life or business critical applications, you need to be sure that they work right the first time, every time. McCabe IQ (Integrated Quality) tools identify, objectively measure and report on the complexity and quality of your code at the application and enterprise level. McCabe's comprehensive and easy to understand tools show you what's been tested, how thoroughly and if the code has been tested once changes have been made.
Once you have that knowledge at your fingertips you can intelligently target your development and Software Testing and QA resources more efficiently saving you time, money and business reputation. Millions of dollars are invested in the development and testing of these applications, so it is crucial that your money is targeted in the right areas.
The McCabe IQ Suite is composed of 3 core products and optional add-ons which provide the necessary components to help you improve your ability to deliver better, more reliable software to your end-users.