As for testing whether a particular state is a removing state or not, you can do that in a unit test that runs 100 times faster. >
Now you only have two states in your examples, but it would be very easy to add more and if you apply that across multiple scenarios in a large application you can see why people end up with cukes that have huge run times.
And those unit tests might as well be greek to the business people. They would be back to testing all of these things by hand.
I have no concern that my unit tests or my integration tests are greek to business people. That is expected. The only thing my tests MAY have to do is communicate to business people that their intentions are being met. If business people are reading the output of Cucumber scenarios and using them to determine whether their application functions or not then they are idiots or completely misinformed. A cucumber scenario in itself going green is completely meaningless, as every scenario will go green with no code. Cucumber scenarios are just a way for business people to contribute to the drive for development. Their executions proves nothing.
No matter how I implement my scenarios they and their output will never be understood by business users in the same way that the code of my application won't be understood by business users. My implemented scenarios are code and their output is not for business users it is for the developers who will have to build open the provided functionality. This idea that you can use the output of cucumber scenarios to judge the health of an application baffles me. If your a business person and you want to judge the health of an application look at the application not a wall of green text from cucumber scenarios.
If the output of Cucumber is meaningless to the people who require the software why bother going to the effort of automating a Gherkin scenario? Why add that layer that translate the Gherkin to code? You say their value is to contribute to driving the development. Couldnât you achieve that simply by using concrete examples in conversations and then translating them to a RSpec or JUnit, etc. Why are we writing in English (or other natural language) if these things arenât meant to be read by people. If they are only meant for developers code is a less ambiguous to way to specify things.
1) Executable specifications based on concrete examples â this is how we are able to consider the business rules and discuss whether they meet our (usersâ) needs, and because we specify software before we write it they fit naturally into a TDD process
2) Automated tests â as software is delivered, executable specs form part of our regression suite. We learn quickly if new changes have effected earlier work, and we can decide if this is expected or not
3) Documentation - these executable specifications, written in (almost) natural language, can document what our software does. This is valuable to people on the team and people outside of the team
Thereâs always a balance that needs to be struck between these three values, and its down to each team to strike the right balance for themselves. I do think though that the hardest to balance are automated tests and documentation. Automated tests push us to brevity and the minimum required as we get concerned with run length. Documentation screams for concrete examples, for richness to interest a reader.
Scenarios don't have to be tied to the UI, and don't have to run slowly. Given the nature of this particular project (a portal that integrates a number of back-end systems) I suspect they will be, for the most part, but I've encouraged them to organize their javascript so that some can be tested without deploying to a server.
Scenarios will always run slowly as they are by definition abstract descriptions of the behaviour of an application described via its user interface. The fact that I can run several hundred scenarios over my application in 3 minutes in a single process on my laptop with a browser doing the javascript does not effect this. I could run 10 times the number of unit test in a few seconds IF I wrote them as well as I write my scenarios. Wasting scenario runtime to test/exercise things that have already been exercised is really detrimental to the overall effectiveness of using Cucumber to drive development.
I disagree with your definition that scenarios must run against a user interface. Scenarios are concerned with being business readable. Some of them will run end-to-end, but they donât need to.
I wrote about this before: http://tooky.co.uk/cucumber-and-full-stack-testing/ <http://tooky.co.uk/cucumber-and-full-stack-testing/> as did Seb Rose: http://claysnow.co.uk/the-testing-iceberg/ <http://claysnow.co.uk/the-testing-iceberg/>.
All best
Andrew
For a better example of taking apart a more complex scenario have a look at an ancient blog post of mine http://pages.andrew.premdas.org/2011/06/27/composable-features-and-tables.html <http://pages.andrew.premdas.org/2011/06/27/composable-features-and-tables.html>
In the original formulation of that, I notice that this feature is checking two distinct and non-interacting sets of rules. One is the rules about taxes, and one is the rules about shipping costs. I would divide the feature file into separate features for each of these sets of rules before doing anything else.
cheers,
George
All best
Nice example, Andrew. I have found that when using tables for
password rules I needed a <reason> column to explain which rule was
being tested. That's a sure sign to me that a different scenario is
beneficial.
I was at a client recently and they were delighted that they could
use Scenario Outlines rather than repeating the scenario. The issue
had to do with different legal requirements in different states. I
don't remember the details, so these made up examples will have to
suffice.
Scenario Outline: State requirements for Spouse as SNI
Given the Primary Named Insured resides in <state>
And the Secondary Named Insured has a relationship of "Spouse"
When the Agent views the policy
Then there is no option to remove the Secondary Named Insured
| state |
| AL |
| AK |
Scenario Outline: No state requirements for Spouse as SNI
Given the Primary Named Insured resides in <state>
And the Secondary Named Insured has a relationship of "Spouse"
When the Agent views the policy
Then the Agent can remove the Secondary Named Insured
| state |
| SC |
| SD |
As I recall, there were other state-oriented scenarios with value
limits, but I can't recall them well enough to generate examples.
I can certainly see how the above could be written as individual
scenarios, but it was both more convenient and less error prone to
implement it with tables.
- George
On 14 September 2017 at 16:57, George Dinwiddie
Andrew,
I'm not exactly sure what you mean by this. Can you give an example
of extracting the spirit/meaning of the examples and naming them?
- George
Sure, lets take something everyone knows about registrations
(excuse bad syntax I don't use outlines examples or tables anymore).
Scenario Outline: Bad registrations
When I register with <account> <password>
Then I should not be register
Account | Password
bad_account | good_password
good_account | 1234
good_account | sdfdsflsdfsdf3313
good_account | wieikjkeiiow
OK so our business peeps have come up with some examples of
account|password combinations that should fail. Now we have to
take each pair and try and extract the spirit/meaning. I'll
start with the easy one
good_account | 1234
This is probably to do with password length. (note all examples
are by their nature ambiguous, this is why scenario outlines and
examples are bad in my book, they don't have information to be
clear about their intention)
So I'd extract a scenario
Scenario: Short password
When I register with a short password
Then I should see my password is to short
The next one is a bit more confusing. Some possibilities are
Need a capital letter
Need a special character
So here I might have to check with the business, and then I'd
rinse and repeat
From this process we end up extracting and making concrete
several important concepts
- short passwords
- passwords with special chars
- passwords without capitals
etc.
Now it might seem a stretch but you can apply this concept to
every scenario outline, and to every example group or table of
examples. And if you do, not only will you get simpler features
that are easier to implement and work with, you'll also get most
of the 'names' you need to write the code to implement the
particular behaviour you are specifiying (yay!)
HTH
Andrew
This ties in quite nicely with my idea/opinion that any Cucumber
scenario that has examples in it, particularly tables is in some
way incomplete.
You complete the scenario by extracting the
spirit/meaning of
the examples and in the process name things. When you have done
this all the tables of examples just disappear and you are left
with named concepts specific to your context.
All best
Andrew
On 14 September 2017 at 10:32, aslak hellesoy
Thanks for the clarification Gabriel.
In the past few years, after helping dozens of
organisations and
hundreds of people adopt BDD I have come to the conclusion
that it
is counter-productive to have 3 amigos (BAs, testers and
programmers) write executable specifications (Gherkin)
together.
What I encourage instead is that they create *examples*
together.
Examples are not executable specifications - they are simply
conversation topics, often jotted down on a card as a
sketch or a
few words. I recommend Example Mapping
(https://cucumber.io/blog/2015/12/08/example-mapping-introduction <https://cucumber.io/blog/2015/12/08/example-mapping-introduction> <https://cucumber.io/blog/2015/12/08/example-mapping-introduction <https://cucumber.io/blog/2015/12/08/example-mapping-introduction>>
<https://cucumber.io/blog/2015/12/08/example-mapping-introduction <https://cucumber.io/blog/2015/12/08/example-mapping-introduction> <https://cucumber.io/blog/2015/12/08/example-mapping-introduction <https://cucumber.io/blog/2015/12/08/example-mapping-introduction>>>
<https://cucumber.io/blog/2015/12/08/example-mapping-introduction <https://cucumber.io/blog/2015/12/08/example-mapping-introduction> <https://cucumber.io/blog/2015/12/08/example-mapping-introduction <https://cucumber.io/blog/2015/12/08/example-mapping-introduction>>
<https://cucumber.io/blog/2015/12/08/example-mapping-introduction <https://cucumber.io/blog/2015/12/08/example-mapping-introduction> <https://cucumber.io/blog/2015/12/08/example-mapping-introduction <https://cucumber.io/blog/2015/12/08/example-mapping-introduction>>>>)
as a technique to create examples together.
When conversations happen around lo-fi examples
like this in an
example mapping session, stakeholders tend to
uncover a lot of
misunderstandings and gain new insight. In my experience,
this does
not happen when the conversations happen around executable
specifications (Gherkin). Instead, conversations tend to
revolve
around grammar and formulation, diverting the attention
from the
problem domain and the examples themselves. Writing
specifications
together is also slow and tedious, and people
don't enjoy these
sessions as much. When it's not enjoyable, people
stop doing it
pretty quickly. Example Mapping on the other hand is
energising and fun.
BDD is a variant of TDD, where programmers write
code guided by
failing tests. In my experience, this workflow does not
work well
when the specifications were written
collaboratively. When the
programmers starts the red-green-refactor cycle with
collaboratively
written scenarios, they quickly realise they need to
reformulate
those scenarios, for a number of reasons. They're incomplete,
they're ambiguous etc. When scenarios start passing, the
scenarios
are often very different from what the diverse group created
together. This often causes frustration: "You've completely
rewritten it - this is not what we wrote together - I don't
understand this".
So instead I recommend programmers write the executable
specifications, perhaps with assistance from a tester. They
use the
example map as input when they do this, and do their best
to capture
the spirit of the examples and the conversations. When
they're done
(or better - while they're doing it), they show the
scenarios to the
BAs/POs and ask for feedback: Do you understand this? Does it
reflect what we discussed during example mapping? Then they
tweak
Specification, Test and Documentation.
BDD isn't a process where you can write the specification
up-front -
even when it happens collaboratively. They take shape as
you go, in
parallel with development. The only thing you can do up-front
(before coding) is to reduce the
misunderstandings, but
executable
specifications isn't a good medium for that I think.
Cheers,
Aslak
On Thu, Sep 14, 2017 at 7:18 AM, Gabriel Oliveira
Hi Aslak,
Thanks for pointing this out - this was a mistake on
our part.
The intent was to state that fact that bad requirements
are one
of the many causes of project failures.
Already fixed on bit.ly/bdd_quality <http://bit.ly/bdd_quality>
<http://bit.ly/bdd_quality <http://bit.ly/bdd_quality>>
<http://bit.ly/bdd_quality <http://bit.ly/bdd_quality>> <http://bit.ly/bdd_quality <http://bit.ly/bdd_quality>>
On Wed, Sep 13, 2017 at 11:03 PM, Aslak HellesÞy
Hi Gabriel,
"Since it is well known that executable solution
requirements are one of many potential causes of a
project
failure, we come to ask some questions about this
subject."
1) Are you saying that project failure can be
partly caused
by the fact that specifications are executable?
2) After 12 years in this field I've never heard
this - on
what basis are you claiming this is well known?
Cheers,
Aslak
On Thursday, September 7, 2017 at 2:17:45 PM UTC+1,
Gabriel
Hi folks,
Have you been practicing BDD
(Behavior-Driven
Development) lately and believe scenarios'
quality matter?
I'm writing my graduate thesis on that subject
and would
like to invite you all to answer a
(really) small 2
https://goo.gl/forms/dheA8XK0asiNUVze2 <https://goo.gl/forms/dheA8XK0asiNUVze2>
<https://goo.gl/forms/dheA8XK0asiNUVze2 <https://goo.gl/forms/dheA8XK0asiNUVze2>>
<https://goo.gl/forms/dheA8XK0asiNUVze2 <https://goo.gl/forms/dheA8XK0asiNUVze2>
<https://goo.gl/forms/dheA8XK0asiNUVze2 <https://goo.gl/forms/dheA8XK0asiNUVze2>>>
<https://goo.gl/forms/dheA8XK0asiNUVze2 <https://goo.gl/forms/dheA8XK0asiNUVze2>
<https://goo.gl/forms/dheA8XK0asiNUVze2 <https://goo.gl/forms/dheA8XK0asiNUVze2>>
<https://goo.gl/forms/dheA8XK0asiNUVze2 <https://goo.gl/forms/dheA8XK0asiNUVze2>
<https://goo.gl/forms/dheA8XK0asiNUVze2 <https://goo.gl/forms/dheA8XK0asiNUVze2>>>>
If you're willing to spend a little more time
to talk to
me about that topic, please remember to fill
out your
contact information there - or comment on this
thread
here - and I will promptly get in touch :) !
Best regards,
--
*Gabriel P.A. de Oliveira*
/https://gpaoliveirablog.wordpress.com <https://gpaoliveirablog.wordpress.com/>
<https://gpaoliveirablog.wordpress.com <https://gpaoliveirablog.wordpress.com/>>
<https://gpaoliveirablog.wordpress.com <https://gpaoliveirablog.wordpress.com/>
<https://gpaoliveirablog.wordpress.com <https://gpaoliveirablog.wordpress.com/>>>
<https://gpaoliveirablog.wordpress.com <https://gpaoliveirablog.wordpress.com/>
<https://gpaoliveirablog.wordpress.com <https://gpaoliveirablog.wordpress.com/>>
<https://gpaoliveirablog.wordpress.com <https://gpaoliveirablog.wordpress.com/>
<https://gpaoliveirablog.wordpress.com <https://gpaoliveirablog.wordpress.com/>>>>/
-- ----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com <http://blog.gdinwiddie.com/>
Software Development http://www.idiacomputing.com <http://www.idiacomputing.com/>
Consultant and Coach http://www.agilemaryland.org <http://www.agilemaryland.org/>
----------------------------------------------------------------------
-- Posting rules: http://cukes.info/posting-rules.html <http://cukes.info/posting-rules.html>
<http://cukes.info/posting-rules.html <http://cukes.info/posting-rules.html>>
--- You received this message because you are subscribed to the
Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it,
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>
<https://groups.google.com/d/optout <https://groups.google.com/d/optout>>.
--
------------------------
Andrew Premdas
blog.andrew.premdas.org <http://blog.andrew.premdas.org/> <http://blog.andrew.premdas.org <http://blog.andrew.premdas.org/>>
--
Posting rules: http://cukes.info/posting-rules.html <http://cukes.info/posting-rules.html>
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com <http://blog.gdinwiddie.com/>
Software Development http://www.idiacomputing.com <http://www.idiacomputing.com/>
Consultant and Coach http://www.agilemaryland.org <http://www.agilemaryland.org/>
----------------------------------------------------------------------
--
Posting rules: http://cukes.info/posting-rules.html <http://cukes.info/posting-rules.html>
--- You received this message because you are subscribed to the Google Groups "Cukes" group.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
------------------------
Andrew Premdas
blog.andrew.premdas.org <http://blog.andrew.premdas.org/>
--
Posting rules: http://cukes.info/posting-rules.html <http://cukes.info/posting-rules.html>
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.