I run a testing application that, using Selenium, runs testing "scenarios" each of which visits a bunch of pages on a website. Each page is modeled by a Java file that contains locators, each of which are used to locate certain elements on the page maybe to verify that they exist or, beyond that, that they work as intended. Some elements may always show up, and some may only show up depending on the scenario you are running. Let's say the page we would like to test in question is the "login" page, and we are talking about all the scenarios that visit that page.

Bear in mind, some scenarios may visit the login page, but only a small subset of those scenarios are actually intended to test the login page.

Let's say each page is modeled by a Java source file that contains a bunch of locators intended to locate elements on that page.

Each scenario is intended to test a different overall functionality on the website, but as far as the login portion of each scenario is concerned, multiple scenarios might be totally redundant (only one of these redundant scenarios need to be run to achieve the exact same level of interaction). Let's say one set of redundant scenarios only minimally enters a username/password and then clicks the log in button. So our Java page always has the exact same sets of locators locating the same elements and not locating any elements for those redundant scenarios.

Let's say 100 of your scenarios visit the login page. Let's also say you ran all 100 scenarios to fully interact with that page, but only 10 of them needed to be run to achieve the same thing, that is, only 10 needed to be run to fully test the page.

Let's say each locator in the Java file modelling the login page is marked with the scenarios where it successfully located an element on the page. Each locator could have anywhere from 1 to 100 scenarios tagged to it.

Having already run the 100 scenarios, how do you go about knowing what those 10 scenarios are for future benefit? What sort of algorithm would achieve this? Thanks!

  • $\begingroup$ I don't really understand your question. Can you abstract your problem using only "scenario" and "locator", use as abstract terms? $\endgroup$ – Yuval Filmus Dec 16 '18 at 0:21
  • 1
    $\begingroup$ It looks like you are dealing with covering set problem. You may want to search for some approximation algorithms. $\endgroup$ – John L. Dec 16 '18 at 3:48
  • $\begingroup$ @Apass.Jack you may be totally right, but I am having difficulty reading that article, not being a math major and all :P. Bear in mind I'm a non-math guy trying to solve a mathematical problem for a real world application. $\endgroup$ – Wes Dec 16 '18 at 4:08
  • $\begingroup$ @Apass.Jack I will check out other articles that talk about the same thing, but in maybe more understandable terms, thanks! $\endgroup$ – Wes Dec 16 '18 at 4:32

I would approach this a little differently.

Suppose you want to test a particular piece of functionality, e.g. logging in.

You first need to navigate to the login page, which may succeed or fail. Failure doesn't mean your test is failing; it means a precondition of your test, namely, being able to navigate to the login page, is failing. That's not what you're testing in this test.

Once you've arrived on the login page, you can attempt to log in, which may succeed or fail. This is the test you want to perform. If your test is correct, the results will correctly indicate whether logging in worked or not.

This is true in general: tests may generally have preconditions that aren't part of the test, but that can be tested themselves.

I think what you're trying to say is that many different paths may cause the precondition (in this case: the login page being reached) to be fulfilled, and testing those paths really should be separated from running the test that that precondition is a precondition for. We want to test those paths separately, and once we think they are reliable, use any of them to fulfill the precondition for the present test.

This can be formulated differently by saying: we'll assume that using the application will put us into certain consecutive states, and what we want to do is test each transition between states. Certain sequences of actions will (if they go well) bring us into a certain state, suitable as a starting point for performing the test we want to perform. The test will then perform another action, that is supposed to bring us into a certain state (the transition), and from the test we will be able to determine whether that state has actually been reached (whether the transition happened that is supposed to happen).

We can express this by modeling the behavior of the application as a state machine that contains the states the application can be in and the state transitions we want to test.

However, we also need to take input and output into account. Sometimes, state transitions depend on user input. For instance, supplying a correct combination of username and password produces the state of being logged in; supplying an incorrect combination produces the state of not being logged in. Furthermore, the output we see (the webpage being produced) may not correspond to a specific state, it may also depend on which transition was taken to arrive into that state. For instance, when a login attempt fails, we will see an error message, but the resulting state is one of not being logged in, not one of being shown an error message.

Therefore, it's a good idea to introduce input and output into the model, which turns our model into an I/O automaton.

So the idea is to specify the application's behavior using an I/O automaton, then to write tests to test each of its transitions. Given those tests, you can automatically generate a complete test suite to test the whole system, using various strategies. I even know a piece of software designed to do this: JTorX.

We'll need to add parametrization. For instance, when logged in, we're logged in as a specific user. The user we're logged in as is a parameter to the state of being logged in. We may need to check whether we're logged in by checking whether certain attributes of that user appear in certain positions on the page. So being able to parametrize the model by user may be important.

Furthermore, I'd factor the state machine into a Petri net or similar. For instance, being logged in is not really a state; it is an attribute of a state. Typically, being logged in we can do everything we can do when not logged in (except logging in), plus a couple of things we need to be logged in for. This can be modeled using Petri nets or some derivative (some sort of colored Petri net).

This approach will provide structure and meaning to your endeavor. Your 100 test scenarios will be paths in your I/O automaton transition graph. Your 10 paths are those of the 100 that lead from the initial state to the state in which you need to be in order to start the test at hand. The whole thing will be incomparably more understandable and concise than just specifying a zillion test scenarios and trying to group them afterwards.

Too bad Selenium, TestNG, or anything else I've worked with don't support this approach directly. If I were still working in test automation, I might have written something that does by now. The JTorX software is dead, as far as I know, but it may still be usable, or alternatives may be out there that use the same approach.


I think I may have the answer. I have tested it, but still not 100% confident it completely holds water in all situations, but so far I can't find any leaks and it seems to run decently fast. If it works, is it optimal?

  1. Get the union of all the scenario IDs into a collection of unique ID strings (e.g. HashSet). Let's call this listing "union".
  2. Create a copy of "union" called "result".
  3. Cycle through each locator's listing of scenarios that successfully located something with that locator. Let's call each listing "myDetectedScenarios".
  4. Try to remove, one at a time, each of the scenario IDs from "result", and try to remove that same ID as well as from all of the "myDetectedScenarios" listings. If removing it from any one of these listings causes it to be empty, then DON'T remove from anywhere. Otherwise, remove it from "result" and any listings using it.
  5. The remaining scenario IDs in "result" will be the scenario IDs that can be minimally run to fully test the page.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.