[Cucumber] Continuation of Cucumber Step even after assertion failure

Post by Eajaz Ali
Hi Team,
I am using cucumber-jvm framework for my automation tests. I want to
implement continuation of my step even after there is an assertion failure.
@Given("^I perform validation$")
public I_perform_validation(){
//assertion1 -> fail
//assertion2 -> pass
//assertion3 -> pass
}
If assertion1 fails, I still want to continue to assertion2 and assertion3.

What is supposed to be the outcome in that case? What are you trying to achieve?

I cannot think of a simple way to achieve what you want. Cucumber does
not define assertions itself but relies on external frameworks like
JUnit or TestNG. Cucumber only checks if the step has thrown an
exception or not.

Post by Eajaz Ali
Can someone please help. This will be really helpful, as I don't want to go
Regards
Eajaz

Eajaz Ali

2015-05-13 06:24:08 UTC

My outcome s this step needs to completely executed, irrespective of
assertion failures. I am trying to implement asserts from junit library.

Regards
Eajaz

Post by Eajaz Ali
Hi Team,
I am using cucumber-jvm framework for my automation tests. I want to
implement continuation of my step even after there is an assertion

failure.

Post by Eajaz Ali
@Given("^I perform validation$")
public I_perform_validation(){
//assertion1 -> fail
//assertion2 -> pass
//assertion3 -> pass
}
If assertion1 fails, I still want to continue to assertion2 and

assertion3.
What is supposed to be the outcome in that case? What are you trying to achieve?
I cannot think of a simple way to achieve what you want. Cucumber does
not define assertions itself but relies on external frameworks like
JUnit or TestNG. Cucumber only checks if the step has thrown an
exception or not.

Post by Eajaz Ali
Can someone please help. This will be really helpful, as I don't want to

Post by Eajaz Ali
Regards
Eajaz

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to a topic in the
Google Groups "Cukes" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/cukes/OTUjEupNjYk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
For more options, visit https://groups.google.com/d/optout.

Aslak Hellesøy

2015-05-13 07:04:00 UTC

My outcome s this step needs to completely executed, irrespective of assertion failures.

Why do you need to run the steps after the failing one?

I am trying to implement asserts from junit library.

Can you explain what you mean by this? JUnit provides static assertion methods that you can call directly.

Aslak

Regards
Eajaz

Post by Paolo Ambrosio

What is supposed to be the outcome in that case? What are you trying to achieve?
I cannot think of a simple way to achieve what you want. Cucumber does
not define assertions itself but relies on external frameworks like
JUnit or TestNG. Cucumber only checks if the step has thrown an
exception or not.

Post by Eajaz Ali
Can someone please help. This will be really helpful, as I don't want to go
Regards
Eajaz

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/OTUjEupNjYk/unsubscribe.
For more options, visit https://groups.google.com/d/optout.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
For more options, visit https://groups.google.com/d/optout.

Mucahit Kurt

2015-05-13 06:52:36 UTC

Hi,

You can catch the assertion exceptions like below but in this case you can
not see the failure assertions/steps via cucumber.

â

Post by Eajaz Ali
@Given("^I perform validation$")
public I_perform_validation(){

try{

Post by Eajaz Ali
//assertion1 -> fail

} cathc(AssertionException e) {
//log something
}

Post by Eajaz Ali
//assertion2 -> pass
//assertion3 -> pass
}

MÃŒcahit Kurt

Post by Eajaz Ali
My outcome s this step needs to completely executed, irrespective of
assertion failures. I am trying to implement asserts from junit library.
Regards
Eajaz

Post by Eajaz Ali
Hi Team,
I am using cucumber-jvm framework for my automation tests. I want to
implement continuation of my step even after there is an assertion

failure.
ââ

Post by Eajaz Ali
Can someone please help. This will be really helpful, as I don't want

to go

Post by Eajaz Ali
Regards
Eajaz

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

aslak hellesoy

2015-05-13 07:18:52 UTC

Post by Mucahit Kurt
Hi,
You can catch the assertion exceptions like below but in this case you can
not see the failure assertions/steps via cucumber.
â

Post by Eajaz Ali
@Given("^I perform validation$")
public I_perform_validation(){

try{

Post by Eajaz Ali
//assertion1 -> fail

} cathc(AssertionException e) {
//log something
}

Post by Eajaz Ali
//assertion2 -> pass
//assertion3 -> pass
}

This is bad advice.

Tests are warning lights that are supposed to start flashing when something
is broken.
What you're suggesting here is to unplug the warning lights.

Never catch exceptions in step definitions. Let them bubble through and
fail the scenario when an exception is thrown.

Aslak

Post by Mucahit Kurt
MÃŒcahit Kurt

Post by Eajaz Ali
My outcome s this step needs to completely executed, irrespective of
assertion failures. I am trying to implement asserts from junit library.
Regards
Eajaz

Post by Eajaz Ali
Hi Team,
I am using cucumber-jvm framework for my automation tests. I want to
implement continuation of my step even after there is an assertion

failure.
ââ

Post by Eajaz Ali
Can someone please help. This will be really helpful, as I don't want

to go

Post by Eajaz Ali
Regards
Eajaz

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

Paul Rayner

2015-05-13 06:16:45 UTC

'Given' is for setting up context, the state of the system under test, not
for making assertions. That is what 'Then' is for.

The 'Then' (plus any subsequent 'And' statements) is where the assertion is
made. Hence, "Then".

It's generally considered very poor practice to continue when an assertion
fails, since the whole point of making such an assertion in BDD is to
verify that some piece of important functionality is currently missing and
needs to be implemented. Why would you want to continue?

Paolo Ambrosio

2015-05-13 08:04:41 UTC

'Given' is for setting up context, the state of the system under test, not
for making assertions. That is what 'Then' is for.

I disagree with this statement. I find it very useful to have
assertions in Given steps. The purpose is to check that the data setup
has been done correctly. Sometimes I even use assertions in When steps
for sanity checks (e.g. to verify that the server didn't return an
internal server error). In both cases of course the assertions do not
verify business rules, as that must happen in Then steps.

Post by Paul Rayner
The 'Then' (plus any subsequent 'And' statements) is where the assertion is
made. Hence, "Then".
It's generally considered very poor practice to continue when an assertion
fails, since the whole point of making such an assertion in BDD is to verify
that some piece of important functionality is currently missing and needs to
be implemented. Why would you want to continue?
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

aslak hellesoy

2015-05-13 08:47:26 UTC

This post might be inappropriate. Click to display it.

Eajaz Ali

2015-05-13 15:58:06 UTC

Hi All,

Thank you very much for your great advices on using Assertions. I want to
perform field level validation on a particular field with different kinds
of test data like blank data, invalid data, special characters.

If I want to pass them from features file, then for each test data, the
selenium driver is launched in @Before method. That would eat lot of time
in execution only, but with manual testing if would hardly take seconds. To
accommodate this, I want to include all possible assertions w.r.t UI level
field level validation in my one step , for eg in @Then. At the same time.
I record all my failures and successful assertions in one single run.

Post by aslak hellesoy

Post by Eajaz Ali
Hi Team,
I am using cucumber-jvm framework for my automation tests. I want to
implement continuation of my step even after there is an assertion

failure.

Post by Paul Rayner
'Given' is for setting up context, the state of the system under test,

not

Post by Paul Rayner
for making assertions. That is what 'Then' is for.

I disagree with this statement. I find it very useful to have
assertions in Given steps. The purpose is to check that the data setup
has been done correctly.

When I explain Given-When-Then I often compare to a theatrical act.
Before the act starts, the stage workers work behind the curtains to set
up the *stage*. They place furniture in the right locations, adjust the
lighting to give the impression of dusk. The actors walk onto the stage.
Mrs Alving is in the dining room. Oswald is in the living room. The
curtains unfold, and the stage is set.
This is the Given. There is nothing to verify, just setting up the scene.
We're only describing what's happened in an imaginary recent past.
Now the act unfolds. The actors talk and move. At one point something
important *happens*.
This is the When.
Later, the act has finished. The curtains are drawn again. The spectators
are sitting in their seats, thinking about everything that unraveled before
their eyes. What happened? What was essential? What was the *outcome*?
This is the Then.
----
Now, back to Cucumber. Sometimes we're dealing with a system where test
automation is an after-thought. We can't easily set up the stage the way we
want, because we're not in control over the venue, the decorations or even
the actors. We have to deal with a system that is in a state we can't
control. So we have to improvise. We can go on with the act if certain
elements are present. Are we indoors? Check. Is there a table? Check. Do we
have a woman and a man? Check.
I think this is what you're describing Paolo. I agree that in such cases,
Given steps can make assertions. It's important to emphasise that this is a
different kind of game though. You only do that when things are
unpredictable. If we can, we want to make things predictable. We want to
control (or stub out) external dependencies. When we can do that, we don't
need assertions in our Given steps.
The same goes for When steps. You *can* use assertions here, but you
shouldn't have to.
The only place where you absolutely *should* use assertions is Then steps.
A final word of advice: Never trust a test you haven't seen fail. You want
to make sure your steps will fail with an error when the system doesn't
behave as expected. If you catch exceptions or continue after a step has
failed, your tests are useless, because they *cannot* fail.
It's like taking the batteries out of your smoke detector.
Aslak

Post by Eajaz Ali
Sometimes I even use assertions in When steps
for sanity checks (e.g. to verify that the server didn't return an
internal server error). In both cases of course the assertions do not
verify business rules, as that must happen in Then steps.

Post by Paul Rayner
The 'Then' (plus any subsequent 'And' statements) is where the

assertion is

Post by Paul Rayner
made. Hence, "Then".
It's generally considered very poor practice to continue when an

assertion

Post by Paul Rayner
fails, since the whole point of making such an assertion in BDD is to

verify

Post by Paul Rayner
that some piece of important functionality is currently missing and

needs to

Post by Paul Rayner
be implemented. Why would you want to continue?
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google

Groups

Post by Paul Rayner
"Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send

Post by Paul Rayner
For more options, visit https://groups.google.com/d/optout.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

m***@gmail.com

2015-05-13 15:52:47 UTC

With regards to the original question about whether you can run all
assertions even if one in the series fails, JUnit supports the concept of
an "Error Collector" which does just that. The API documentation can be
found here: http://junit.org/apidocs/org/junit/rules/ErrorCollector.html.
Since this is vanilla JUnit, it should work properly with cucumber, though
I haven't personally attempted it myself.

Mila

Björn Rasmusson

2015-05-13 19:58:26 UTC

Post by m***@gmail.com

JUnit ErrorCollectors cannot be used in Cucumber-JVM step definitions in
the same way they are used in JUnit test methods (JUnit preforms some
"magic" with respect to the @Rule annotated ErrorCollector fields when
running its test methods, that Cucumber-JVM does not do when running
scenarios). But with some tweaks it is possible to use the JUnit
ErrorCollectors in step definitions to be able to perform multiple assert
(and get all failures, not just the first failure).

First the verify() method of the error collector needs to be called
explicitly in the step definition. And secondly, since the verify() method
is protected in the ErrorCollector class, it need to sub-classed to make it
public.

With a ErrorCollector subclass like:
class MyCollector extends ErrorCollector {
@Override
public void verify() throws Throwable {
super.verify();
};
}

It is possible to define step definitions like:
@Then("<good description that makes the step understandable>")
public void step_def_method_name() throws Throwable {
MyCollector collector = new MyCollector();
collector.checkThat(actual1, equalTo(expected1));
collector.checkThat(actual2, equalTo(expected2));
collector.verify();
}

BjÃ¶rn

Post by m***@gmail.com
Mila

Andrew Premdas

2015-05-13 17:15:49 UTC

Post by Eajaz Ali
Hi All,
Thank you very much for your great advices on using Assertions. I want to
perform field level validation on a particular field with different kinds
of test data like blank data, invalid data, special characters.
If I want to pass them from features file, then for each test data, the
in execution only, but with manual testing if would hardly take seconds. To
accommodate this, I want to include all possible assertions w.r.t UI level
I record all my failures and successful assertions in one single run.
Well Cucumber is not really a tool for doing this kind of stuff :).

Assuming you know that, you can still do this reasonably elegantly by
pushing all your assertion testing down from the step definition into a
helper method.

The helper method would look something like (in rubyish pseudocode)

module MultipleAssertionStepHelper
def assert_lots( ...
results = []
assertions.each do |assertion|
results << assertion.result
end
results.contains failures ? report failures : passed
end
end

and your Then would be something like

Then "blah should be good" do
expect(assert_lots assertions: blah_assertions).to be_good
end

By pushing the code down you take it out of cucumber and into a domain you
control, now you can do whatever you want.

Post by aslak hellesoy

Post by Eajaz Ali
Hi Team,
I am using cucumber-jvm framework for my automation tests. I want to
implement continuation of my step even after there is an assertion

failure.