Software Testing

Black box testing:

Black box testing also known as “Functional Testing”. A type of software testing where internal workings of item being tested is unknown to the tester. Say for example, in Black box testing when applied to software engineering the tester has only the knowlegde of “legal” inputs not the expected output and how program actually arrives to those outputs.

Black box testing is what most of us practice and is used most widely. Black Box testing which is very close to customer experience.

Black box testing methods include equivalence partitioning, boundary value analysis, all-pairs testing, fuzz testing, model-based testing, traceability matrix, exploratory testing and specification-based testing.

For Black Box testing, test groups are often used, “Test groups are aka “professional idiots”…people those who are good at designing incorrect data. Also, do to the nature of black box testing; the test planning can begin as soon as the specifications are written.

As a test engineer if you are performing black box test cases, one thing that you need to make sure is that you do not make any assumptions about the system based on your knowledge. Assumption created in our mind because of the system knowledge could harm testing effort and increase the chances of missing critical test cases.

The only input can be for test engineer in the Black Box testing is the requirement document and functionality of system which can be get by working with the system.

Purpose of black box testing is to

1. Make sure that system is working in accordance with the system requirement.

2. Make sure that system is meeting user expectation.

For example, suppose we are testing a function that uses the quadratic formula to determine the two roots of a second-degree polynomial ax²+bx+c. For simplicity, assume that we are going to work only with real numbers, and print an error message if it turns out that the two roots are complex numbers (numbers involving the square root of a negative number).

We can come up with test data for each of the four cases, based on values of the polynomial’s discriminant (b²-4ac):

Easy data (discriminant is a perfect square):

Typical data (discriminant is positive):

Boundary / extreme data (discriminant is zero):

Bogus data (discriminant is negative, or a is zero):

As with glass-box testing, you should test your code with each set of test data. If the answers match, then your code passes the black-box test.

Process of making sure that older programming still works with the new changes is called Regression testing.

Any time you modify an implementation within a program, Regression software testing should be performed. This can be done by rerunning the existing test cases against the modified code to check whether the code change breaks the existing functionality. Regression testing is a normal part of the program development process and, in larger companies, is done by code testing specialists. Test department coders develop code test scenarios and exercises that will test new units of code after they have been written. One should try to spend as little time as possible doing regression testing without reducing the probability that you will detect new failures in old, already tested code.

Some strategies and factors to consider during this Regression Testing process include the following:

• Make changes (small and large) to data and find any resulting corruption.
• Identify tests that the program consistently passes and archive them.
• Watch for side effects of fixes. The bug itself might be fixed but the fix might create other bugs.
• Test fixed bugs promptly. The programmer might have handled the symptoms but not have gotten to the underlying cause.
• Focus on functional issues, not those related to design.
• Trace the effects of the changes on program memory.
• Write a regression test for each bug fixed.
• If two or more tests are similar, determine which is less effective and get rid of it.

Process of performing end to end test cases under anticipated production load is called Load Testing.

Objective here to determine the response of various time critical transactions and business processes and to ensure that they are with in documented expectations (or Service Level Agreements – SLAs). Load testing can also be done in the field to obtain a qualitative idea of how well a system functions in the “real world.”

Load testing is a part of a more general process known as performance testing. Examples of load testing include:

• Subjecting a server to a large amount of e-mail traffic.
• Downloading a series of large files from the Internet.
• Assigning many jobs to a printer in a queue.
• Running multiple applications on a computer or server simultaneously.
• Writing and reading data to and from a hard disk continuously.

Purpose of Load testing:

The purpose of any load test should be clearly understood and documented. A load test usually fits into one of the following categories:

• Quantification of risk- Determine, through formal testing, the likelihood that system performance will meet the formal stated performance expectations of stakeholders, such as response time requirements under given levels of load. This is a traditional Quality Assurance (QA) type test. Note that load testing does not mitigate risk directly, but through identification and quantification of risk, presents tuning opportunities and an impetus for remediation that will mitigate risk.
• Determination of minimum configuration- Determine, through formal testing, the minimum configuration that will allow the system to meet the formal stated performance expectations of stakeholders – so that extraneous hardware, software testing and the associated cost of ownership can be minimized. This is a Business Technology Optimization (BTO) type test.

Logical extension of unit testing is called Integration Testing.

Objective here is to make sure that the interaction of two or more components produces results that satisfy functional requirement. In its simplest form, two units that have already been tested are combined into a component and the interface between them is tested. In Integration Testing the test cases are developed with the purpose of exercising the interface between the components. Integration Testing can also be treated as testing assumption of fellow programmer. Integration testing identifies problems that occur when units are combined.

By using a test plan that requires you to test each unit and ensure the viability of each before combining units, you know that any errors discovered when combining units are likely related to the interface between units. This method reduces the number of possibilities to a far simpler level of analysis.

Integration Testing can be performed in various ways but here are the following three strategies:

• Top-down approach: Here Integration Testing requires the highest-level modules be test and integrated first. This allows high-level logic and data flow to be tested early in the process and it tends to minimize the need for drivers. However, the need for stubs complicates test management and low-level utilities are tested relatively late in the development cycle. Another disadvantage of top-down integration testing is its poor support for early release of limited functionality.
• Bottom-up approach: Here Integration Testing requires the lowest-level units be tested and integrated first. These units are frequently referred to as utility modules. By using this approach, utility modules are tested early in the development process and the need for stubs is minimized. The downside, however, is that the need for drivers complicates test management and high-level logic and data flow are tested late. Like the top-down approach, the bottom-up approach also provides poor support for early release of limited functionality.
• Third approach: Sometimes referred to as the umbrella approach, requires testing along functional data and control-flow paths. First, the inputs for functions are integrated in the bottom-up pattern discussed above. The outputs for each function are then integrated in the top-down manner. The primary advantage of this approach is the degree of support for early release of limited functionality. Integration Testing also helps minimize the need for stubs and drivers. The potential weaknesses of this approach are significant, however, in that it can be less systematic than the other two approaches, leading to the need for more regression testing.

Here is the above picture first the lowest level modules “B” and “C” are tested with drives then next higher module “A” is testing with a drive integrated with module “B” and “C”. Finally the integration will be continued till the top-level module is tested. The total number of tests required by this technique is the sum of the cyclomatic complexities of all modules being integrated.

Performance testing: Process of determining the effectiveness or speed of computer, network, and software program is called Performance Testing. In simple words, Performance testing is testing related to speed, scalability and stability. Quantitative tests such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions can be involved in Performance testing. Performance testing is often done in conjunction with stress testing.

Effective Performance Testing can involve following steps:

• Identify the Test Environment: Physical, production, tools and resources available to the team should be identified first. Physical environment includes hardware, software, and network communications. Thorough understanding of the entire test environment enables you the more efficient test cases and planning helps to identify testing challenges in the early stages of the project. Process must be revisited periodically throughout the project life cycle.
• Identify Performance Acceptance Criteria: Response time, throughput, and resource utilization goals and constraints should be identified as they are user concern, business concern, and system concern respectively. Additionally, identify project success criteria that may not be captured by those goals and constraints; for example, using performance tests to evaluate what combination of configuration settings will result in the most desirable performance characteristics.
• Plan and Design Tests cases: Identify key scenarios, define test data, and establish metrics to be collected. Consolidate this information into one or more models of system usage to be implemented, executed, and analyzed.
• Configure the Test Environment: Prepare the test environment, tools, and resources necessary to execute each strategy as features and components become available for test. Ensure that the test environment is instrumented for resource monitoring as necessary.
• Execute the Test: Run and monitor your tests. Validate the tests, test data, and results collection. Execute validated tests for analysis while monitoring the test and the test environment.
• Analyze Results, Report, and Retest: Consolidate and share results data. Cross verify it with other functional teams. Reprioritize the remaining tests and re-execute them as needed.

Incremental integration testing

Incremental integration testing is continuous testing of an application as new functionality is recommended. This may require that various aspects of an application’s functionality are independent enough to work separately, before all parts of the program are completed, or that test drivers are developed as needed. Incremental integration testing may be performed by programmers, software engineers, or test engineers.

Sanity testing

A sanity test is a narrow regression test that focuses on one or a few areas of functionality. Sanity testing is usually narrow and deep. A sanity test is usually unscripted. A Sanity test is used to determine a small section of the application is still working after a minor change. Sanity testing is a cursory testing; it is performed whenever a cursory testing is sufficient to prove the application is functioning according to specifications. This level of testing is a subset of regression testing. Sanity testing is to verify whether requirements are met or not, checking all features breadth-first.

Sanity testing is also similar to Smoke testing, but has some minor differences. Sanity testing is done when the application is deployed into testing for the very first time and in smoke testing only positive scenarios are validated but in sanity testing both the positive and negative scenarios are validated.

Alpha testing

Alpha testing is conducted by real customer i.e. conducted at one or more customer sites by end user of software. Alpha Testing involves few developers, few testers, and some of customer site people; this test is suitable for applications.

Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing, before the software goes to beta testing.

According to the classic definition of Myers: “Software Testing is the process of executing a program or system with the intent of finding errors.”

According to the definition given by Hetzel: ”Software Testing involves any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results”

In simple language, process of finding deviation from the requirements and user expectations is called software testing. Here tester will be the first person to look at the product from the user point of view and will be the last person who can talk about its quality prior to its release.

Regardless of the limitations, testing is an part in software development. It is broadly deployed in every phase in the software development cycle. Typically, more than 50% percent of the development time is spent in testing. Software Testing is usually performed to ensure quality, correctness, and validation & verification.

If we see software testing is just to ensure the above points but it has various methods to perform it. Lets discuss these in details…

Types of Software testing

May 6th, 2009

Before moving to the types of software testing lets give a thought that why we need to categorize software testing into different types?
whats the purpose behind it?
Basic purpose of software tesitng is to identify defects in the software, and defects can vary from requirements to user expectations.

 Defects can be categorize as follows:

If system is not functioning properly, its a functional defect.
If system is not performing well, its a performance defect.
If system is not usable, its a usability defect.
If system is not secure, its a security defect.

So we saw that defects can be of various types so is software testing. Lets go and discuss one by one.