[Cucumber] Re: Official Cucumber Specification (was: Cucumber3 go implementation)

[aslak hellesoy]

Hi Scott,

Thanks for your kind words! (I'm copying in the Cucumber mailing list since
this is of public interest)

There are many official Cucumber implementations now [1], but I suppose you
are talking about the ruby one.

I'm aware of several unofficial Cucumber implementations in Go (and also
other languages), but I'm not deeply familiar with any of them.
With the growing popularity of Cucumber, and new unofficial implementations
appearing, what we need is a specification for Cucumber.

With this I hope that the various implementations will be more uniform, and
that some of the unofficial ones can be included in the official family.
Or better yet - that someone writes one from scratch under the Cucumber
organisation.

The first step is to get the Cucumber specification to a level of quality
where it can be used as a guide to write new Cucumber implementations.
Is this something you'd be interested in helping out with? I hope to
assemble a team of 5-10 people to contribute to this spec and make it
complete enough in a couple of months time.

Cheers,
Aslak

[1] https://github.com/cucumber/cucumber#official-cucumber-implementations

Hello,
Thanks for all the wonderful work on cucumber. We have built up a huge
test suite with cucumber, almost 5000 scenarios, and it's starting to run
really slow on the windows machines some of our developers are using.
It takes about 4 minutes to run all the tests on my mac, so I'm assuming
it's a ruby/windows/fileio/forking type issue that I'd rather not try and
solve.
We use the wire protocol to talk to cucumber-lua which is the environment
that all the code is actually in.
All of that context to say, I've noticed some activity on the cucumber3-go
implementation, and I was wondering if you knew of any work to run the
tests through the wire protocol in go.
That way I could get a nice statically compiled cucumber.exe that I would
expect should be much faster then the ruby/windows combination.
I've done some work in go, and could try to hack something together if I
was pointed in the right direction to get started.
Thanks for your time
Scott

[aslak hellesoy]

To give an idea of what I want the spec to cover - here is a quick brain
dump. I'm sure there is more, so please fill in.

* Standard formats for results (JSON, Tap, other)
* Internal architecture of Cucumber
* Plugin architecture (filters, reporters etc)
* Internal data structures (AST, compiled test cases)
* Architectural recommendation for parallel execution
* Standardise configuration/command-line options
* Tag expression syntax and grammar
* Categorise all functionality (mandatory, optional,
implementation-specific)
* Wire protocol (we could use a new one that doesn't rely on external libs
like JSON)
* Specification for suites (a la Behat:
http://docs.behat.org/en/v3.0/guides/5.suites.html)
* Specification for a new, simpler alternative to regular expressions - see
https://github.com/cucumber/cucumber/issues/1

I'd like to use simple Markdown (.md) files for the various sections in the
spec, and they should live in the https://github.com/cucumber/cucumber
repo. This is better than a wiki, because we can use the regular
pull-request workflow to ensure quality and consistency.

For diagrams I want to use ASCII (as in
https://github.com/cucumber/gherkin3/blob/master/README.md#architecture) -
we'll use MonoDraw for this. Cucumber will sponsor a license to significant
contributors.

In theory it would be nice to have a common set of Cucumber scenarios to
verify an implementation. We made a previous attempt at that with
https://github.com/cucumber/cucumber-tck, but it turned out to be more of a
hassle than a benefit. Maybe we should revisit it.

Another goal I have with this specification is to come up with an
architecture that makes it possible to reuse reporters/formatters across
different language implementations. I think Cucumber itself should only
report results to the console (progress or pretty), and only support JSON
(and maybe Tap). No HTML!

What I want instead is to have a separate program (written in any language)
that would consume feature files and JSON/Tap results and produce pretty
HTML. Generating nice HTML reports is a lot of effort, and duplicating this
effort in every implementation is a waste of effort. You've seen how bad
the built-in HTML formats are, which is why there are several 3rd-party
reporters out there. I want an official one that gains traction across all
implementations. For that we need a standardised JSON format for results.
It should only contain file, line and result. There is not need to embed
info from the gherkin itself in the JSON - that can be read from the
feature file (using Gherkin3).

I'm super excited about all of this, so reply to this thread if you want to
help getting it all down in a spec!

Aslak

Post by aslak hellesoy
Hi Scott,
Thanks for your kind words! (I'm copying in the Cucumber mailing list
since this is of public interest)
There are many official Cucumber implementations now [1], but I suppose
you are talking about the ruby one.
I'm aware of several unofficial Cucumber implementations in Go (and also
other languages), but I'm not deeply familiar with any of them.
With the growing popularity of Cucumber, and new unofficial
implementations appearing, what we need is a specification for Cucumber.
With this I hope that the various implementations will be more uniform,
and that some of the unofficial ones can be included in the official family.
Or better yet - that someone writes one from scratch under the Cucumber
organisation.
The first step is to get the Cucumber specification to a level of quality
where it can be used as a guide to write new Cucumber implementations.
Is this something you'd be interested in helping out with? I hope to
assemble a team of 5-10 people to contribute to this spec and make it
complete enough in a couple of months time.
Cheers,
Aslak
[1] https://github.com/cucumber/cucumber#official-cucumber-implementations

Hello,
Thanks for all the wonderful work on cucumber. We have built up a huge
test suite with cucumber, almost 5000 scenarios, and it's starting to run
really slow on the windows machines some of our developers are using.
It takes about 4 minutes to run all the tests on my mac, so I'm assuming
it's a ruby/windows/fileio/forking type issue that I'd rather not try and
solve.
We use the wire protocol to talk to cucumber-lua which is the environment
that all the code is actually in.
All of that context to say, I've noticed some activity on the
cucumber3-go implementation, and I was wondering if you knew of any work to
run the tests through the wire protocol in go.
That way I could get a nice statically compiled cucumber.exe that I would
expect should be much faster then the ruby/windows combination.
I've done some work in go, and could try to hack something together if I
was pointed in the right direction to get started.
Thanks for your time
Scott

[Mark Winspear]

Post by aslak hellesoy
Another goal I have with this specification is to come up with an
architecture that makes it possible to reuse reporters/formatters across
different language implementations. I think Cucumber itself should only
report results to the console (progress or pretty), and only support JSON
(and maybe Tap). No HTML!
What I want instead is to have a separate program (written in any
language) that would consume feature files and JSON/Tap results and produce
pretty HTML. Generating nice HTML reports is a lot of effort, and
duplicating this effort in every implementation is a waste of effort.
You've seen how bad the built-in HTML formats are, which is why there are
several 3rd-party reporters out there. I want an official one that gains
traction across all implementations. For that we need a standardised JSON
format for results. It should only contain file, line and result. There is
not need to embed info from the gherkin itself in the JSON - that can be
read from the feature file (using Gherkin3).

Have you taken a look at the reports and logs that robot framework
generates [1]? They are clean, easy to read and can be filtered on tags

[1]
http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#report-file

[aslak hellesoy]

Post by Mark Winspear

Post by aslak hellesoy
Another goal I have with this specification is to come up with an
architecture that makes it possible to reuse reporters/formatters across
different language implementations. I think Cucumber itself should only
report results to the console (progress or pretty), and only support JSON
(and maybe Tap). No HTML!
What I want instead is to have a separate program (written in any
language) that would consume feature files and JSON/Tap results and produce
pretty HTML. Generating nice HTML reports is a lot of effort, and
duplicating this effort in every implementation is a waste of effort.
You've seen how bad the built-in HTML formats are, which is why there are
several 3rd-party reporters out there. I want an official one that gains
traction across all implementations. For that we need a standardised JSON
format for results. It should only contain file, line and result. There is
not need to embed info from the gherkin itself in the JSON - that can be
read from the feature file (using Gherkin3).

Have you taken a look at the reports and logs that robot framework
generates [1]? They are clean, easy to read and can be filtered on tags

The Robot reports don't separate content from representation. This is a
severe limitation because:

a) the representation can't be improved independently of the tool
(cucumber/robot)
b) the implementation will have to be implemented in all cucumber
implementation

So going forward, Cucumber will rely on an external tool to generate pretty
reports in HTML, taking gherkin and a simple results doc as input.

Post by Mark Winspear
[1]
http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#report-file
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Roberto Lo Giacco]

Il giorno venerdì 4 settembre 2015 11:07:29 UTC+2, Aslak HellesÞy ha

Post by aslak hellesoy
To give an idea of what I want the spec to cover - here is a quick brain
dump. I'm sure there is more, so please fill in.
* Specification for a new, simpler alternative to regular expressions -
see https://github.com/cucumber/cucumber/issues/1

I wouldn't go down that route, regexp are a well known standard used in so
many contexts

Post by aslak hellesoy
For diagrams I want to use ASCII (as in
https://github.com/cucumber/gherkin3/blob/master/README.md#architecture)
- we'll use MonoDraw for this. Cucumber will sponsor a license to
significant contributors.

That MonoDraw seems available on Mac OS only...

Post by aslak hellesoy
Another goal I have with this specification is to come up with an
architecture that makes it possible to reuse reporters/formatters across
different language implementations. I think Cucumber itself should only
report results to the console (progress or pretty), and only support JSON
(and maybe Tap). No HTML!
What I want instead is to have a separate program (written in any
language) that would consume feature files and JSON/Tap results and produce
pretty HTML. Generating nice HTML reports is a lot of effort, and
duplicating this effort in every implementation is a waste of effort.
You've seen how bad the built-in HTML formats are, which is why there are
several 3rd-party reporters out there. I want an official one that gains
traction across all implementations. For that we need a standardised JSON
format for results. It should only contain file, line and result. There is
not need to embed info from the gherkin itself in the JSON - that can be
read from the feature file (using Gherkin3).

As James already commented, that's not advisable: cross referencing the
gherkin files not only makes the report producer a lot more complex, but
requires two artifacts to be in sync to generate the report. I would advice
to either bring the untouched gherkin inside the JSON output or associate
the relevant gherkin to each report item as it is now. The only downside I
see is the dimension of the report file which I would advice to be
optionally splittable so that huge projects will not require tons of RAM to
process the output.

Regards,
Roberto

[aslak hellesoy]

Post by Roberto Lo Giacco
Il giorno venerdì 4 settembre 2015 11:07:29 UTC+2, Aslak HellesÞy ha

Post by aslak hellesoy
To give an idea of what I want the spec to cover - here is a quick brain
dump. I'm sure there is more, so please fill in.
* Specification for a new, simpler alternative to regular expressions -
see https://github.com/cucumber/cucumber/issues/1

I wouldn't go down that route, regexp are a well known standard used in so
many contexts

Did you get the impression it was a replacement? It's not. It's an
alternative. Regexps will still be supported.

Post by Roberto Lo Giacco

Post by aslak hellesoy
For diagrams I want to use ASCII (as in
https://github.com/cucumber/gherkin3/blob/master/README.md#architecture)
- we'll use MonoDraw for this. Cucumber will sponsor a license to
significant contributors.

That MonoDraw seems available on Mac OS only...

Oh crap. What do you suggest instead?

Post by Roberto Lo Giacco

Post by aslak hellesoy
Another goal I have with this specification is to come up with an
architecture that makes it possible to reuse reporters/formatters across
different language implementations. I think Cucumber itself should only
report results to the console (progress or pretty), and only support JSON
(and maybe Tap). No HTML!
What I want instead is to have a separate program (written in any
language) that would consume feature files and JSON/Tap results and produce
pretty HTML. Generating nice HTML reports is a lot of effort, and
duplicating this effort in every implementation is a waste of effort.
You've seen how bad the built-in HTML formats are, which is why there are
several 3rd-party reporters out there. I want an official one that gains
traction across all implementations. For that we need a standardised JSON
format for results. It should only contain file, line and result. There is
not need to embed info from the gherkin itself in the JSON - that can be
read from the feature file (using Gherkin3).

As James already commented, that's not advisable: cross referencing the
gherkin files not only makes the report producer a lot more complex, but
requires two artifacts to be in sync to generate the report. I would advice
to either bring the untouched gherkin inside the JSON output or associate
the relevant gherkin to each report item as it is now. The only downside I
see is the dimension of the report file which I would advice to be
optionally splittable so that huge projects will not require tons of RAM to
process the output.

I appreciate your feedback on this. I still think separation is the right
way to go, but I realise I'll have to explain why and how in more detail.
Hope to find time to do that soon.

Aslak

Post by Roberto Lo Giacco
Regards,
Roberto
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Roberto Lo Giacco]

Post by aslak hellesoy

Post by Roberto Lo Giacco
Il giorno venerdì 4 settembre 2015 11:07:29 UTC+2, Aslak HellesÞy ha

Post by aslak hellesoy
To give an idea of what I want the spec to cover - here is a quick brain
dump. I'm sure there is more, so please fill in.
* Specification for a new, simpler alternative to regular expressions -
see https://github.com/cucumber/cucumber/issues/1

I wouldn't go down that route, regexp are a well known standard used in
so many contexts

Did you get the impression it was a replacement? It's not. It's an
alternative. Regexps will still be supported.

Apologies, I misunderstood.

Post by aslak hellesoy

Post by Roberto Lo Giacco

Post by aslak hellesoy
For diagrams I want to use ASCII (as in
https://github.com/cucumber/gherkin3/blob/master/README.md#architecture)
- we'll use MonoDraw for this. Cucumber will sponsor a license to
significant contributors.

That MonoDraw seems available on Mac OS only...

Oh crap. What do you suggest instead?

I just googled " ASCII diagrams" and found something I didn't even know and
I'm certainly going to use in the future
http://asciiflow.com/

Post by aslak hellesoy

Post by Roberto Lo Giacco

Post by aslak hellesoy
Another goal I have with this specification is to come up with an
architecture that makes it possible to reuse reporters/formatters across
different language implementations. I think Cucumber itself should only
report results to the console (progress or pretty), and only support JSON
(and maybe Tap). No HTML!
What I want instead is to have a separate program (written in any
language) that would consume feature files and JSON/Tap results and produce
pretty HTML. Generating nice HTML reports is a lot of effort, and
duplicating this effort in every implementation is a waste of effort.
You've seen how bad the built-in HTML formats are, which is why there are
several 3rd-party reporters out there. I want an official one that gains
traction across all implementations. For that we need a standardised JSON
format for results. It should only contain file, line and result. There is
not need to embed info from the gherkin itself in the JSON - that can be
read from the feature file (using Gherkin3).

As James already commented, that's not advisable: cross referencing the
gherkin files not only makes the report producer a lot more complex, but
requires two artifacts to be in sync to generate the report. I would advice
to either bring the untouched gherkin inside the JSON output or associate
the relevant gherkin to each report item as it is now. The only downside I
see is the dimension of the report file which I would advice to be
optionally splittable so that huge projects will not require tons of RAM to
process the output.

I appreciate your feedback on this. I still think separation is the right
way to go, but I realise I'll have to explain why and how in more detail.
Hope to find time to do that soon.

I'll keep an eye on this, I have the opposito opinion, but let's try to
convince each other, shall we?

Roberto

[Anthony Browness]

I'm a bit late to the party, I could contribute to this spec.

I'm rather good at parallel execution and I'm quickly coming up to speed on
the internal architecture of Cucumber. Let me know if you want me on board.

Post by aslak hellesoy
To give an idea of what I want the spec to cover - here is a quick brain
dump. I'm sure there is more, so please fill in.
* Standard formats for results (JSON, Tap, other)
* Internal architecture of Cucumber
* Plugin architecture (filters, reporters etc)
* Internal data structures (AST, compiled test cases)
* Architectural recommendation for parallel execution
* Standardise configuration/command-line options
* Tag expression syntax and grammar
* Categorise all functionality (mandatory, optional,
implementation-specific)
* Wire protocol (we could use a new one that doesn't rely on external libs
like JSON)
http://docs.behat.org/en/v3.0/guides/5.suites.html)
* Specification for a new, simpler alternative to regular expressions -
see https://github.com/cucumber/cucumber/issues/1
I'd like to use simple Markdown (.md) files for the various sections in
the spec, and they should live in the https://github.com/cucumber/cucumber
repo. This is better than a wiki, because we can use the regular
pull-request workflow to ensure quality and consistency.
For diagrams I want to use ASCII (as in
https://github.com/cucumber/gherkin3/blob/master/README.md#architecture)
- we'll use MonoDraw for this. Cucumber will sponsor a license to
significant contributors.
In theory it would be nice to have a common set of Cucumber scenarios to
verify an implementation. We made a previous attempt at that with
https://github.com/cucumber/cucumber-tck, but it turned out to be more of
a hassle than a benefit. Maybe we should revisit it.
Another goal I have with this specification is to come up with an
architecture that makes it possible to reuse reporters/formatters across
different language implementations. I think Cucumber itself should only
report results to the console (progress or pretty), and only support JSON
(and maybe Tap). No HTML!
What I want instead is to have a separate program (written in any
language) that would consume feature files and JSON/Tap results and produce
pretty HTML. Generating nice HTML reports is a lot of effort, and
duplicating this effort in every implementation is a waste of effort.
You've seen how bad the built-in HTML formats are, which is why there are
several 3rd-party reporters out there. I want an official one that gains
traction across all implementations. For that we need a standardised JSON
format for results. It should only contain file, line and result. There is
not need to embed info from the gherkin itself in the JSON - that can be
read from the feature file (using Gherkin3).
I'm super excited about all of this, so reply to this thread if you want
to help getting it all down in a spec!
Aslak

Post by aslak hellesoy
Hi Scott,
Thanks for your kind words! (I'm copying in the Cucumber mailing list
since this is of public interest)
There are many official Cucumber implementations now [1], but I suppose
you are talking about the ruby one.
I'm aware of several unofficial Cucumber implementations in Go (and also
other languages), but I'm not deeply familiar with any of them.
With the growing popularity of Cucumber, and new unofficial
implementations appearing, what we need is a specification for Cucumber.
With this I hope that the various implementations will be more uniform,
and that some of the unofficial ones can be included in the official family.
Or better yet - that someone writes one from scratch under the Cucumber
organisation.
The first step is to get the Cucumber specification to a level of quality
where it can be used as a guide to write new Cucumber implementations.
Is this something you'd be interested in helping out with? I hope to
assemble a team of 5-10 people to contribute to this spec and make it
complete enough in a couple of months time.
Cheers,
Aslak
[1]
https://github.com/cucumber/cucumber#official-cucumber-implementations

Hello,
Thanks for all the wonderful work on cucumber. We have built up a huge
test suite with cucumber, almost 5000 scenarios, and it's starting to run
really slow on the windows machines some of our developers are using.
It takes about 4 minutes to run all the tests on my mac, so I'm assuming
it's a ruby/windows/fileio/forking type issue that I'd rather not try and
solve.
We use the wire protocol to talk to cucumber-lua which is the
environment that all the code is actually in.
All of that context to say, I've noticed some activity on the
cucumber3-go implementation, and I was wondering if you knew of any work to
run the tests through the wire protocol in go.
That way I could get a nice statically compiled cucumber.exe that I
would expect should be much faster then the ruby/windows combination.
I've done some work in go, and could try to hack something together if I
was pointed in the right direction to get started.
Thanks for your time
Scott

[Paolo Ambrosio]

Post by aslak hellesoy
Hi Scott,
Thanks for your kind words! (I'm copying in the Cucumber mailing list since
this is of public interest)
There are many official Cucumber implementations now [1], but I suppose you
are talking about the ruby one.
I'm aware of several unofficial Cucumber implementations in Go (and also
other languages), but I'm not deeply familiar with any of them.
With the growing popularity of Cucumber, and new unofficial implementations
appearing, what we need is a specification for Cucumber.
With this I hope that the various implementations will be more uniform, and
that some of the unofficial ones can be included in the official family.
Or better yet - that someone writes one from scratch under the Cucumber
organisation.
The first step is to get the Cucumber specification to a level of quality
where it can be used as a guide to write new Cucumber implementations.
Is this something you'd be interested in helping out with? I hope to
assemble a team of 5-10 people to contribute to this spec and make it
complete enough in a couple of months time.
Cheers,
Aslak
[1] https://github.com/cucumber/cucumber#official-cucumber-implementations

Hello,
Thanks for all the wonderful work on cucumber. We have built up a huge
test suite with cucumber, almost 5000 scenarios, and it's starting to run
really slow on the windows machines some of our developers are using.
It takes about 4 minutes to run all the tests on my mac, so I'm assuming
it's a ruby/windows/fileio/forking type issue that I'd rather not try and
solve.
We use the wire protocol to talk to cucumber-lua which is the environment
that all the code is actually in.

If the 5000 scenarios are running slow on Windows using the wire
protocol, it could be something to do with the Nagle algorithm. We hit
the same problem on Cucumber-CPP:

https://groups.google.com/forum/#!topic/cukes/4sJOm7yQgS8

Post by aslak hellesoy

All of that context to say, I've noticed some activity on the cucumber3-go
implementation, and I was wondering if you knew of any work to run the tests
through the wire protocol in go.
That way I could get a nice statically compiled cucumber.exe that I would
expect should be much faster then the ruby/windows combination.
I've done some work in go, and could try to hack something together if I
was pointed in the right direction to get started.
Thanks for your time
Scott

[James Nord]

"There is not need to embed info from the gherkin itself in the JSON - that can be read from the feature file (using Gherkin3)."

As a tool author that currently reads json i would be very anti this.
The feature files may not be easily accessible (embedded in jars etc) or be very hard to tie to the actual result. Please do not assume that there will only be one run of cucumber to produce a result. The only thing i can gaurantee access to is the json. If i need to support retreiving feature files from various sources (cucumber embedded in swc, and exe, jar, database...) then you have failed as the toolong that you want to beable to use from any language now needs to support all languages.