[Cucumber:1582] Mocking out third party site/redirects

[Mike Doel]

If this is better asked on a Fakeweb or Webrat board, redirect me
there...

In our app, we're trying to mock out interaction with a third-party
payment processing service (Braintree in our case). The way
Braintree's transparent redirect model works is that the user form
posts credit card and other billing data to Braintree which redirects
the user back to our app to display the result.

We want our Feature/Scenarios to run without requiring a network
connection, so we have:

FakeWeb.allow_net_connect=false

set. I've tried to write a scenario like this...

Scenario: Submit valid payment data using default address
Given FakeWeb will mimic a braintree success
And I am ready to transact
When I select "VISA" from "Card Type"
And I fill in "Card Number" with "4111111111111111"
And I press "Continue"
Then I should see "Woo - Billing!"

with a step like:

Given /^FakeWeb will mimic a braintree success$/ do
FakeWeb.register_uri :post,'https://
secure.braintreepaymentgateway.com/api/transact.php',
:status => ["301", "Moved Permanently"],
:location => transaction_success_url
(Transaction.last)
end

The URL in the above step of course matches the first parameter to the
form_tag in the view. However, when I try executing this, I get the
error:

No route matches "/api/transact.php" with {:method=>:post}
(ActionController::RoutingError)
(eval):2:in `click_button'

Can anyone suggest what the right way to mock out the network would be
for a case like this? I had thought the above FakeWeb call would
result in the "I press 'Continue'" step getting redirected as
mentioned above, but the error looks like it never even gets to FakeWeb.

Any help/suggestions would be appreciated.

Mike


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Stephen Eley]

Post by Mike Doel
Can anyone suggest what the right way to mock out the network would be
for a case like this?  I had thought the above FakeWeb call would
result in the "I press 'Continue'" step getting redirected as
mentioned above, but the error looks like it never even gets to FakeWeb.

Unfortunately, Webrat doesn't play well with redirects outside your
own site. I discovered this when trying to write a feature for
Twitter's OAuth authorization. It doesn't really hook itself in at
the HTTP level like that.

I eventually handled it by requiring Mocha in my Cucumber setup and
stubbing out the various OAuth calls. I don't feel like it's a very
_good_ solution -- having a mocking framework in my integration tests
feels a bit like cheating -- but apart from using a test account and
calling out to Twitter itself every time, or building a whole fake
Twitter (both of which I also considered) I couldn't think of another
smooth answer. In your case, maybe you could just use Braintree's
sandbox? It may not be "clean testing," it may have performance
implications, etc. etc. -- but it's a whole lot faster and easier than
stubbing.
--
Have Fun,
Steve Eley (sfeley-***@public.gmane.org)
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+***@googlegroups.com
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Ashley Moran]

Post by Mike Doel
In our app, we're trying to mock out interaction with a third-party
payment processing service (Braintree in our case). The way
Braintree's transparent redirect model works is that the user form
posts credit card and other billing data to Braintree which redirects
the user back to our app to display the result.

Hi Mike

FWIW, when faced with this situation myself, I usually knock out a
simple webapp (in a smaller, low-dependency framework like Ramaze) to
act as a mock implementation. Sometimes it's as simple as stubbing it
to return XML, other times it needs a bit of logic. (The biggest I've
written was a mock Twitter, which stored tweets and returned timelines.)

There's more investment in this, and it doesn't protect you agains API
changes. But, as you haven't stubbed any internal code, your
confidence that your app will integrate in production is equivalent to
your confidence that you simulated the remote service correctly. The
problems I've run up against have largely been assumptions on how the
service behaves. ( This is a good way to find out exactly how the web
service *does* behave :) )

Hope this is a useful alternative perspective.

Ashley
--
http://www.patchspace.co.uk/
http://www.linkedin.com/in/ashleymoran
http://aviewfromafar.net/








--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Mike Doel]

Post by Ashley Moran
FWIW, when faced with this situation myself, I usually knock out a
simple webapp (in a smaller, low-dependency framework like Ramaze) to
act as a mock implementation

Thanks for the feedback from folks on this. We're going to use a
similar approach to this I think, except that we'll just embed the
simulator inside the webapp itself as a controller that only gets used
in the test environment. As you said, we're vulnerable to API
changes, but that's largely unavoidable if you don't actually hit the
real web service. And to help reduce the risk of that, we'll have one
or two scenarios that lift the FakeWeb network restriction and hit the
real test gateway at Braintree. Those scenarios may be tagged in a
way that they won't get run as part of our default cuke profile, but
we'll be able to run them anyway with a different profile.

Thanks again.

Mike


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Ashley Moran]

Post by Mike Doel
We're going to use a
similar approach to this I think, except that we'll just embed the
simulator inside the webapp itself as a controller that only gets used
in the test environment.

Hey, clever way of doing this if you want to avoid creating a new
service! Just as long as there's no cheating going on in that
controller :)

Post by Mike Doel
As you said, we're vulnerable to API
changes, but that's largely unavoidable if you don't actually hit the
real web service. And to help reduce the risk of that, we'll have one
or two scenarios that lift the FakeWeb network restriction and hit the
real test gateway at Braintree. Those scenarios may be tagged in a
way that they won't get run as part of our default cuke profile, but
we'll be able to run them anyway with a different profile.

You could go further - how about tagging all the scenarios that access
it as @braintree, and have an overnight run against the Braintree
sandbox environment? Might need alternative step implementations but
would give high confidence in the integration without slowing down
development in the day.

Ashley
--
http://www.patchspace.co.uk/
http://www.linkedin.com/in/ashleymoran
http://aviewfromafar.net/








--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Stephen Eley]

On Thu, Oct 1, 2009 at 3:55 PM, Ashley Moran

Post by Ashley Moran

Post by Mike Doel
We're going to use a
similar approach to this I think, except that we'll just embed the
simulator inside the webapp itself as a controller that only gets used
in the test environment.

I started to do something like this for my own mock-Twitter
implementation (before I gave up on the work involved). If you keep
on this road, I'll give you one tip: Rails Engines. They're
first-class citizens in Rails 2.3, and it means you can bundle those
test controllers, models, etc. into a plugin. Just exclude that
plugin in every environment except environments/test.rb.

Post by Ashley Moran
You could go further - how about tagging all the scenarios that access
sandbox environment?  Might need alternative step implementations but
would give high confidence in the integration without slowing down
development in the day.

I'm going to be heretical here and ask the obvious question: if you're
writing steps that could run against the real service, _why not_ save
all the mocking work and test against the real service (or its
official test sandbox) _all the time?_

Seriously. What's the typical risk here? Performance? Cucumber step
runs are already slow. Is it worth the hours it takes to fully stub
out a service to get an incremental speed improvement? If you're
doing continuous integration, does speed matter all that much? And if
the service is _that_ painfully slow, should you even be using it?

The service might be down? Maybe -- but if that's what makes your
tests fail, at least you'll know it. And if it happens _often_,
wouldn't that be an issue you'd like to know about before depending on
it in production?

I've lately started working more and more against real services,
because FakeWeb is such a hassle for anything with more than a few
operations and it has so many limitations. (Can't examine POST data,
any variation on query parameters won't match, etc.) For the Twitter
project I'm doing, I eventually just created a number of real accounts
on Twitter that act as test fixtures. They've only friended each
other, I've created known statuses that I can test on, etc. It was a
bit tricky to set them up in the first place -- but having done it, I
can be absolutely sure at least that my OAuth logins work. >8->

In real work, the only major problem I've identified is repeatability.
You usually want your test runs to start with a clean slate, and the
ability to enter the same transactions over and over without former
runs screwing up your current results. In some cases that's a
showstopper, or forces you to get way too clever or contrived in your
test cases; but most services will give you operations to delete data.
So I just write a cleanup script that runs as an "After" block. As a
positive side effect, deleting data becomes an operation that's tested
early.

I haven't gone too far in this direction _yet_ -- I was indoctrinated
into the "stub everything" way of life too, and it's only been a short
while since I dared to venture out of it experimentally. If there are
bigger pitfalls, I haven't hit them yet. What I _have_ experienced is
a tremendous time savings and stress relief, as I start to let go of
the nagging feeling that I'm spending more time on my test doubles
than I am on my actual application.
--
Have Fun,
Steve Eley (sfeley-***@public.gmane.org)
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+***@googlegroups.com
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Ashley Moran]

Post by Stephen Eley
I'm going to be heretical here and ask the obvious question: if you're
writing steps that could run against the real service, _why not_ save
all the mocking work and test against the real service (or its
official test sandbox) _all the time?_

I've come against two issues that make this harder than it sounds.

First, it may not be possible to get the sandbox environment to
respond as if in all the significant situations it can find itself.
For example, I dealt with a payment gateway sandbox that could not
return all of the failure states defined in the API, and some were
significant from our app's point of view.

Second, the service sandbox may not be resettable. If, for example,
submitting invalid card details blocks the card for a day, you have to
make your features submit different details. Individually these
issues don't seem significant, but every workaround compounds the
problem and makes features harder to understand for a new developer.

Of course, it's possible that neither of these are true in Mike's case.

Post by Stephen Eley
Seriously. What's the typical risk here? Performance? Cucumber step
runs are already slow. Is it worth the hours it takes to fully stub
out a service to get an incremental speed improvement?

Having recently spent a weekend speeding up our features (from over an
hour to about 20 mins), I've been reminded that the value of frequent
feature runs outweighs the cost of the optimisation (payback on this 2
days' work is only 20 days of development). Lower transaction cost
encourages more feature runs, which gives a shorter feedback loop,
which reduces thrashing (and frayed tempers), which reduces
development time/cost. Now I'm not against having regular runs
against a live service, but I feel that allowing this to creep in may
be the thin end of the wedge, and allow further slowdowns to creep in
unnoticed. This is just my cautious/paranoid attitude though, YMMV.

Post by Stephen Eley
If you're
doing continuous integration, does speed matter all that much?

I'm 100% with you there. BUT, if the CI is running against the live
service, it should probably be set to check for integration errors and
not report the build as failed. Especially if the team has a build-
breaker's dunce/clown hat :) I wouldn't wanna mediate that one...

Post by Stephen Eley
And if
the service is _that_ painfully slow, should you even be using it?

There may be higher business value/needs that justify the cost of
developing against a really painful service. That said, as someone
who spends most of their time coding, I like to gently guide people
towards the non-sucky technology :)


Ashley
--
http://www.patchspace.co.uk/
http://www.linkedin.com/in/ashleymoran
http://aviewfromafar.net/








--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Stephen Eley]

On Fri, Oct 2, 2009 at 3:21 AM, Ashley Moran

Post by Ashley Moran
First, it may not be possible to get the sandbox environment to
respond as if in all the significant situations it can find itself.
For example, I dealt with a payment gateway sandbox that could not
return all of the failure states defined in the API, and some were
significant from our app's point of view.

Sure. If you need to verify your behavior in responding to "edge
case" failures, you can always mock out those specific failures
(generate the complete HTTP response or whatever) and feed them
directly to your app. It can still be easier than mocking the
service, because most failures probably don't need to return data.

One thing to keep in mind, though: if you're testing against failures
you can't create on the other service, you're really just _guessing_
about what that service will do. You're trusting its documentation
and building to a hypothetical. Sometimes you might need to do that,
but it should be a caveat kept in mind any time you say "We've
verified our app's behavior." My own feeling is that the fewer
guesses you have to make, the better.

Post by Ashley Moran
Second, the service sandbox may not be resettable.  If, for example,
submitting invalid card details blocks the card for a day, you have to
make your features submit different details.  Individually these
issues don't seem significant, but every workaround compounds the
problem and makes features harder to understand for a new developer.

True. That can be tricky. But a service behavior like that is
probably something you'd want to test for, because someday a customer
will try it. Testing against the service makes it more likely you'll
discover that behavior sooner. If you only ran a test suite once a
night you might never know about the limitations, or only discover it
accidentally the one time you try your test run twice and get very
confused because you couldn't reproduce the issue the next day.

If you did mock it out, and you knew about the blocking, you'd want to
make sure your mock had at least some cases where it "remembered"
prior submitted card details so that it could return the same errors.
You're in for some extra work either way, unless you choose to ignore
the problem.

Post by Ashley Moran
Having recently spent a weekend speeding up our features (from over an
hour to about 20 mins), I've been reminded that the value of frequent
feature runs outweighs the cost of the optimisation (payback on this 2
days' work is only 20 days of development).

I definitely agree on the value of frequent runs. I set
'AUTOFEATURE=true' and run autospec all the time, so my Cucumber
features are running every time my RSpec tests pass. (I find the
Cucumber autotest integration has its own weird annoyances, but that's
a different thread.)

The thing is, though: they're running in the _background._ I don't
stop typing and wait for them to finish so that I can see what
happened. I click into that window when I'm ready, and then I respond
to whatever I saw in the last traces. It's not an interrupt. It's
asynchronous.

And so far, going against live services hasn't slowed down Cucumber
that much for me. Maybe I haven't put in enough scenarios yet. >8->
--
Have Fun,
Steve Eley (sfeley-***@public.gmane.org)
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+***@googlegroups.com
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Paul Cortens]

Hi,

Sorry to resurrect this old(ish) post, but I am running up against
this issue right now. I've got a couple of additional issues that make
writing features against the staging environment more difficult.

1. Only a limited number of API calls are allowed against the staging
environment. I'm not dealing with a huge site like Twitter. They will
notice if I am running my cucumber features with autotest. What makes
things worse is that (for what I need to do) the API isn't always
efficient. Some things which could be theoretically completed with a
single API call will actually take many API calls. (It's sort-of an N
+1 issue.)
2. I only get 1 staging account, and my auth-tokens get reset each
time I use the app in a different environment. The site is using
OAuth, and each time I request access in a different environment, I
get a different access key/secret. I can live with having to manually
type in my username/password on the 3rd party's staging site each time
a move from my development environment to my staging environment, but
this won't work with automated tests. Maybe I could change my
implementation, or hard-code something, but I don't see a work-around
at the moment.
3. The API is incomplete. Much of the stuff I would put in my "Given"
clauses can't be done with the API. It can only be done with their Web
UI. Even if it could be done with their API, my main application has
no business using this part of the API so I am very hesitant to put in
all the work involved in implementing that part of the API just so I
can use it in my "Given" clauses.

Stephen, are these reasonable objections to your method?

If so, I'm left with faking the API. My question is, at what layer do
I mock?

My gut tells me to mock the XML. I have a config file that points to
the API server. It points to their staging server for staging (and
development) and their production server for production. For testing,
I have been pointing it to my own Sinatra app which returns (mostly)
hard-coded XML. This seems nice on the surface, but then we get into
Webrat's issues with hitting other servers. (OAuth uses redirects.)
I've sort-of got it working with Selenium, but that's painfully slow.
The 2nd problem is that I don't have a good way to set the "state" of
the API server. I'd like to have "Given" clauses such as "Given the
external web service has an Person named Joe." What kind of step
matcher do I use with that? I guess I could implement an API to my
Sinatra app that allows me to POST xml to "create" new Person objects.
That seems like over kill. An alternative would be to forget using
Sinatra app and bring all the mock code into a Controller that is only
loaded in the test environment (like Mike said). That would let me
just use something simple like a global variable for storing my mock
data. My previous "Given" clause might have a step matcher like
$third_party_api_mocks[:people] << {:name => 'Joe'}.

A 2nd option would be to create a pure-Ruby wrapper for the API. Then
I'd just use RSpec's mock/stub framework (or something else, like
mocha) to catch all the API calls. My main objection to this is that I
haven't quite pinned down what this pure-Ruby wrapper should look
like. I think I'd like to end up with something that is ActiveRecourd-
like. Do I stub at this level? That makes me nervous, because the
wrapper needs to be tested. Or do I do the thinnest Ruby wrapper
possible, then stub that? It seems like the only advantage of this is
that I don't have to write quite so much XML. However, I feel like I'm
introducing a fairly significant hole in my integration tests by going
with this method. Is that fear justified? I expect to see a LOT of
refactoring in this application, so I definitely don't want
unnecessary gaps in my tests.


I really hope this discussion continues. I love BDD, but dealing with
the limitation of working with external APIs really hurts sometimes.

Thanks,
Paul

Post by Ashley Moran

Post by Stephen Eley
I'm going to be heretical here and ask the obvious question: if you're
writing steps that could run against the real service, _why not_ save
all the mocking work and test against the real service (or its
official test sandbox) _all the time?_

I've come against two issues that make this harder than it sounds.

[Matt Wynne]

Post by Paul Cortens
Hi,
Sorry to resurrect this old(ish) post, but I am running up against
this issue right now. I've got a couple of additional issues that make
writing features against the staging environment more difficult.
1. Only a limited number of API calls are allowed against the staging
environment. I'm not dealing with a huge site like Twitter. They will
notice if I am running my cucumber features with autotest. What makes
things worse is that (for what I need to do) the API isn't always
efficient. Some things which could be theoretically completed with a
single API call will actually take many API calls. (It's sort-of an N
+1 issue.)
2. I only get 1 staging account, and my auth-tokens get reset each
time I use the app in a different environment. The site is using
OAuth, and each time I request access in a different environment, I
get a different access key/secret. I can live with having to manually
type in my username/password on the 3rd party's staging site each time
a move from my development environment to my staging environment, but
this won't work with automated tests. Maybe I could change my
implementation, or hard-code something, but I don't see a work-around
at the moment.
3. The API is incomplete. Much of the stuff I would put in my "Given"
clauses can't be done with the API. It can only be done with their Web
UI. Even if it could be done with their API, my main application has
no business using this part of the API so I am very hesitant to put in
all the work involved in implementing that part of the API just so I
can use it in my "Given" clauses.
Stephen, are these reasonable objections to your method?
If so, I'm left with faking the API. My question is, at what layer do
I mock?
My gut tells me to mock the XML. I have a config file that points to
the API server. It points to their staging server for staging (and
development) and their production server for production. For testing,
I have been pointing it to my own Sinatra app which returns (mostly)
hard-coded XML. This seems nice on the surface, but then we get into
Webrat's issues with hitting other servers. (OAuth uses redirects.)
I've sort-of got it working with Selenium, but that's painfully slow.
The 2nd problem is that I don't have a good way to set the "state" of
the API server. I'd like to have "Given" clauses such as "Given the
external web service has an Person named Joe." What kind of step
matcher do I use with that? I guess I could implement an API to my
Sinatra app that allows me to POST xml to "create" new Person objects.
That seems like over kill. An alternative would be to forget using
Sinatra app and bring all the mock code into a Controller that is only
loaded in the test environment (like Mike said). That would let me
just use something simple like a global variable for storing my mock
data. My previous "Given" clause might have a step matcher like
$third_party_api_mocks[:people] << {:name => 'Joe'}.
A 2nd option would be to create a pure-Ruby wrapper for the API. Then
I'd just use RSpec's mock/stub framework (or something else, like
mocha) to catch all the API calls. My main objection to this is that I
haven't quite pinned down what this pure-Ruby wrapper should look
like. I think I'd like to end up with something that is ActiveRecourd-
like. Do I stub at this level? That makes me nervous, because the
wrapper needs to be tested. Or do I do the thinnest Ruby wrapper
possible, then stub that? It seems like the only advantage of this is
that I don't have to write quite so much XML. However, I feel like I'm
introducing a fairly significant hole in my integration tests by going
with this method. Is that fear justified? I expect to see a LOT of
refactoring in this application, so I definitely don't want
unnecessary gaps in my tests.

You're getting warm with the Sinatra app idea, I think.

The thing people forget when talking about mocks and stubs is that the
mock objects we get from our mocking libraries are special cases of a
more general class of tool called a 'test double' - something which
behaves enough like the real thing that it gives your code a thorough
work-out, but is simple enough that you can guarantee it behaves in a
deterministic way during the test runs. These test doubles don't need
to be fancy mock objects - they can just as easily be hand-crafted
Ruby classes (or even Sinatra web services), and these are often much
more suitable.

The first thing you need to do is build a wrapper around the third-
party API. This is not only a good idea for the testability of your
app, it also protects you from ripple effects if they release a new
version with a changed API, or you even decide you want to switch to
using another service. Working outside-in, think about the things
*your code* needs from this service, then build an API for yourself
that gives you that. Now you sit down and write an adapter[1] which
implements that API you've designed, and calls the third-party
service. You won't be able to test this code (unless you set something
up with this staging service they offer) so make sure it's as thin as
possible - mostly just forwarding calls on to their own library. Next
you write another adapter which also implements that interface, and
adds some more methods which your Given steps can call to prepare the
behaviour of the fake for the scenario, and maybe some more for the
Then steps to use to sense what has happened.

If you are confident in how the third party API will behave, you can
confidently write a fake that mimics its behaviour. What you may find
is that because you're unfamiliar with the service you can't mimic
it's behaviour in certain scenarios because you don't know how the
real thing will behave. In this case I recommend writing a test (use
RSpec or Test::Unit) against the 3rd party API to nail down exactly
what it does, then run that test on both the real and fake adapters,
to make sure your fake is doing it's job.

Make sense?

These ideas aren't mine: they're taken from people like Steve Freeman:
http://www.mockobjects.com/book/third-party.html


[1]http://en.wikipedia.org/wiki/Adapter_pattern

Post by Paul Cortens
I really hope this discussion continues. I love BDD, but dealing with
the limitation of working with external APIs really hurts sometimes.
Thanks,
Paul

Post by Ashley Moran

Post by Stephen Eley
I'm going to be heretical here and ask the obvious question: if you're
writing steps that could run against the real service, _why not_ save
all the mocking work and test against the real service (or its
official test sandbox) _all the time?_

I've come against two issues that make this harder than it sounds.

cheers,
Matt

http://mattwynne.net
[phone]

[Paul Cortens]

Thank you very much for your reply. You have forced me to keep re-
thinking things, and every time I do that it gets a little more clear.

I still feel like I am left with two options: use a fake at the XML
level or use a fake at the adapter level.

The XML level (sinatra or similar app) provides the thinnest level of
abstraction. I can go through all the API calls and scenarios that I
ever expect to occur and copy/paste the XML. For the 3rd parties with
good documentation (yes there are actually a few APIs out there with
good, accurate docs!) I can even avoid most of the manual work
involved. This approach will work even when I can't use the API to set
up the state I need for my tests. However, any automated tests that I
can write will be all about the XML. Something like
Given there are no People
When I GET /people.xml
Then I should see:
<people count="0">
</people>

The adapter level (i.e. making two adapters, one real and one fake)
would let me write test statements like "ThirdParty::People.all.should
be_empty"
However, for the situations where I can't use the API to set up the
appropriate state I am a bit stuck. I guess I could manually set up
the state for each test, then run the test. It would be a lot of work,
but I'd only have to do this when the 3rd-party API actually changed.
However, it sounds a LOT harder than just copying/pasting the XML.


I'm really stuck between the two. This has been the first time TDD/BDD
has slowed me down so much. (I'm not blaming TDD/BDD! I just need to
learn how to do it in this scenario.)

I actually have two similar projects with different 3rd-party APIs. I
am very tempted to try one of them by faking at the XML level and one
of them by faking at the adapter level. Then I can experience the pros
and cons of each approach. Maybe that's the only really good way to
decide.

Thanks,
Paul

[Matt Wynne]

Post by Paul Cortens
Thank you very much for your reply. You have forced me to keep re-
thinking things, and every time I do that it gets a little more clear.
I still feel like I am left with two options: use a fake at the XML
level or use a fake at the adapter level.
The XML level (sinatra or similar app) provides the thinnest level of
abstraction. I can go through all the API calls and scenarios that I
ever expect to occur and copy/paste the XML. For the 3rd parties with
good documentation (yes there are actually a few APIs out there with
good, accurate docs!) I can even avoid most of the manual work
involved. This approach will work even when I can't use the API to set
up the state I need for my tests. However, any automated tests that I
can write will be all about the XML. Something like
Given there are no People
When I GET /people.xml
<people count="0">
</people>
The adapter level (i.e. making two adapters, one real and one fake)
would let me write test statements like "ThirdParty::People.all.should
be_empty"
However, for the situations where I can't use the API to set up the
appropriate state I am a bit stuck. I guess I could manually set up
the state for each test, then run the test. It would be a lot of work,
but I'd only have to do this when the 3rd-party API actually changed.
However, it sounds a LOT harder than just copying/pasting the XML.
I'm really stuck between the two. This has been the first time TDD/BDD
has slowed me down so much. (I'm not blaming TDD/BDD! I just need to
learn how to do it in this scenario.)
I actually have two similar projects with different 3rd-party APIs. I
am very tempted to try one of them by faking at the XML level and one
of them by faking at the adapter level. Then I can experience the pros
and cons of each approach. Maybe that's the only really good way to
decide.

Where is the boundary between your code and the 3rd party code? Is it
at the XML? Do you do the parsing of the XML yourself? If so, I'd
almost certainly put my 'seam' (as Michael Feathers calls them) at the
point where XML comes back from the remote web service, using FakeWeb
or similar. Otherwise I'd always try to put some abstraction layer
around the call to the 3rd party API and depend on that.

I think you might be right that at this stage you're best off doing
some discovery for yourself - don't get stuck in analysis paralysis! :)


cheers,
Matt

[phone]
matt-***@public.gmane.org
http://mattwynne.net

[Paul Cortens]

Where is the boundary between your code and the 3rd party code? Is it  
at the XML? Do you do the parsing of the XML yourself? If so, I'd  
almost certainly put my 'seam' (as Michael Feathers calls them) at the  
point where XML comes back from the remote web service, using FakeWeb  
or similar. Otherwise I'd always try to put some abstraction layer  
around the call to the 3rd party API and depend on that.

It is at the XML. I make a GET/POST/PUT/DELETE request (which may or
may not include XML in the body) to the 3rd party and they reply with
XML.

Thanks,
Paul

[Ashley Moran]

Post by Paul Cortens
I'm really stuck between the two. This has been the first time TDD/BDD
has slowed me down so much. (I'm not blaming TDD/BDD! I just need to
learn how to do it in this scenario.)

This is like a form of the management delusion "writing tests takes
too long". It's very easy to see only the amount of work you have to
do to protect yourself against the external service, because that's
what you make a decision to invest in. But you never see the time you
save later, due to either not testing your design against the
production service (eg because you can't), or because of a bug in that
service that would have been highlighted by investigating it closely
yourself.

In effect, you've got what I can only describe as "inherited" or
"imposed" technical debt. That sucks. It means someone else's poor
work is incurring expense on every company that uses it. Whether to
use it remains a business decisions, though.

At the extreme end, if you want your mock implementation of their
service to deliver maximum value, you'd have to write both the XML API
and the web interface you use for hoop-jumping. That's the only way
you could do a nightly run against the staging service using the same
steps you use locally. And that, to my fanatical, slightly neurotic
brain, is the only way to believe your app will remain running for any
length of time. This is based mainly on my experience that every time
I've left assumptions in my (step/spec) code, I've paid more in the
long run fixing that I would have done with upfront work.

Post by Paul Cortens
I guess I could implement an API to my
Sinatra app that allows me to POST xml to "create" new Person objects.
That seems like over kill.

I don't believe this is overkill, it's just being thorough.


If you choose to do less than a full simulation of the staging
service, be aware that your exposure to false assumptions is much
higher. The benefit is that you may be able to produce an accurate
mock service faster. In this situation, I'd suggest making a log of
integration bugs that occur in the wild, and record the reason they
crept in. This would give you a useful metric to determine whether
it's worth investing more time in tightening up your features.

Post by Paul Cortens
I really hope this discussion continues. I love BDD, but dealing with
the limitation of working with external APIs really hurts sometimes.

It really, really does. And since most corporate APIs are written by
developers who probably haven't even heard of BDD, I think this will
continue. Looks like it's up to us to do something about it.

Ashley

[Jim Meyer]

Post by Matt Wynne
The first thing you need to do is build a wrapper around the third-
party API. This is not only a good idea for the testability of your
app, it also protects you from ripple effects if they release a new
version with a changed API, or you even decide you want to switch to
using another service. Working outside-in, think about the things
*your code* needs from this service, then build an API for yourself
that gives you that. Now you sit down and write an adapter[1] which
implements that API you've designed, and calls the third-party
service. You won't be able to test this code (unless you set something
up with this staging service they offer) so make sure it's as thin as
possible - mostly just forwarding calls on to their own library. Next
you write another adapter which also implements that interface, and
adds some more methods which your Given steps can call to prepare the
behaviour of the fake for the scenario, and maybe some more for the
Then steps to use to sense what has happened.

I keep thinking that there ought to be a way to record and reply
request/response data interacting with an external API. Think of it as a
response cache for a known set of requests against a given API.

Ideally, you'd use the response data to fuel your mocks and stubs; as a
bonus, you'd be able to reply your requests against the API and validate
that it still responds as you expect (and thus that your tests are still
valid.

Obviously, there are going to be edge cases and impossibilities; for
example, you probably don't want to replay that high-dollar credit card
transaction repeatedly against the production server. Still, it feels
like something like this would be net useful.

WDYT, folks?

--j

[Mike Doel]

Post by Jim Meyer
I keep thinking that there ought to be a way to record and reply
request/response data interacting with an external API. Think of it as a
response cache for a known set of requests against a given API.

This is pretty much what we've done for mocking out sending the user
through the Braintree billing system. In non-test mode, the user
starts on a page where they enter credit card data. When they submit,
the form is actually posting to Braintree which then redirects the
user back to our APP along with a transaction id and some basic info
that we use to pull more substantial info from them via an API.

We've mocked this out by creating:

1. a BraintreeSimulationController that gets used as a stand-in for
the third-party site and
2. a BraintreeTransaction that wraps the concept of hitting the API
after we get the redirect back.

A payment.yml file is used to control the target of the billing form
(going to a braintree dev gateway in development mode and the
BraintreeSimulationController in test mode). Then, we use FakeWeb
inside our step definitions like this:

When /^I use a credit card that will approve$/ do
url = %r|https://secure.braintreepaymentgateway.com/api/query.php|
stub_get url, 'braintree/approved_query_response_contact_info.txt'
end

and in a module included into Cucumber...

def stub_get(url, filename)
FakeWeb.register_uri(:get, url, :response => fixture_file
(filename))
end

The fixture files were captured by using curl to hit the actual
Braintree test gateway.

I don't know if this is terribly clear, but if not, I would be happy
to create a more thorough blog post showing more context of our
solution. We're still not fully done with it yet (e.g. I still want a
few scenarios that actually use the real Braintree Payment gateway as
a way to improve confidence in our simulator), but it's been useful to
us and is possibly valuable to others.

Mike

[Matt Southerden]

Post by Ashley Moran
You could go further - how about tagging all the scenarios that access
sandbox environment? Might need alternative step implementations but
would give high confidence in the integration without slowing down
development in the day.
Ashley

I'm currently writing some payment gateway code, and this is the
approach I'm taking wrt tests which require the remote service.

For access to the fake gateway, I'm writing a simple facade to the
gateway which I can swap out with a fake in testing. This way, I can
have the fake return canned responses but at an internal object level
rather than trying to mock out built-in ruby classes such as tcp
socke, net http, etc, or running my own http or socket server.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Matt Wynne]

Post by Ashley Moran

Post by Mike Doel
In our app, we're trying to mock out interaction with a third-party
payment processing service (Braintree in our case). The way
Braintree's transparent redirect model works is that the user form
posts credit card and other billing data to Braintree which redirects
the user back to our app to display the result.

Hi Mike
FWIW, when faced with this situation myself, I usually knock out a
simple webapp (in a smaller, low-dependency framework like Ramaze) to
act as a mock implementation. Sometimes it's as simple as stubbing it
to return XML, other times it needs a bit of logic. (The biggest I've
written was a mock Twitter, which stored tweets and returned
timelines.)

A tip for those who go down this route which I got from Dan North, is
to write tests for the fake service as you build it. These tests
should then also run against the real service too, which gives you a
clear idea of whether you've succeeded in simulating it's behaviour
(or at least the subset of its behaviour that's interesting to you).
You don't need to run those tests on every check-in, but you can use
them as a tool to keep you on track.


cheers,
Matt

[phone]
matt-***@public.gmane.org
http://mattwynne.net


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---

[Ashley Moran]

Post by Matt Wynne
A tip for those who go down this route which I got from Dan North, is
to write tests for the fake service as you build it. These tests
should then also run against the real service too, which gives you a
clear idea of whether you've succeeded in simulating it's behaviour
(or at least the subset of its behaviour that's interesting to you).
You don't need to run those tests on every check-in, but you can use
them as a tool to keep you on track.

Very true. I haven't done this in the past, assuming that the service
would remain very simple, but they can grow over time. The situation
you get into is that the features for your app verify the mock
service, which makes a dangerous assumption that the mock service is
actually working correctly in the first place.

Likewise with feature support code. We recently lost a few hours due
to a parsing error in some feature code that we would have caught if
we'd developed that code BDD the same way as the app itself.

Ashley
--
http://www.patchspace.co.uk/
http://www.linkedin.com/in/ashleymoran
http://aviewfromafar.net/


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Cukes" group.
To post to this group, send email to cukes-/***@public.gmane.org
To unsubscribe from this group, send email to cukes+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/cukes?hl=en
-~----------~----~----~----~------~----~------~--~---